The answer is that Nginx since version 0.8.44 does not cache backend responses if they have a “Set-Cookie” header. This makes perfect sense because you don’t want a response which sets a cookie to be cached for subsequent requests, but I was stupid because I had totally forgotten that my Django app was using a POST form for all responses for non-authenticated clients (due to how Django’s CSRF middleware does its stuff).

The solution was to change the app so that it uses the GET method on the form in question, which in this case is fine from a security point-of-view.

The moral of this story is I should pay attention to my HTTP response headers and that I am badly short-sighted both figuratively and literally. With that fixed the site has gone from 15 requests per second to ~2000 requests per second!

(If you were mad keen on Django’s shift to class-based views you might reckon life wouldn’t be complete with a Bottle-based application until you employ classes for views. However Bottle’s use of decorators for tying URLs to views means it is less a natural fit than the same thing in Django.)

The problem is that you can’t just decorate the method in your class using bottle.route because if you use that decorator on a method in a class you are telling Bottle to use the method before it has been bound to an instance of that class.

So although I wish it did, the following example will not work:

import bottle

class ViewClass(object):
    @bottle.route("/")
    def home_view(self):
        return "My home page."

obj = ViewClass()
bottle.run()

Running that will lead to errors about not enough arguments passed to the view method of your ViewClass instance.

Instead you need to register the route right after the object is created. This can be done in the class’s __new__ method:

import bottle

class ViewClass(object):
    def __new__(cls, *args, **kwargs):
        obj = super(ViewClass, cls).__new__(cls, *args, **kwargs)
        bottle.route("/")(obj.home_view)
        return obj

    def home_view(self):
        return "My home page."

obj = ViewClass()
bottle.run()

It works. It isn’t that pretty. You could achieve exactly the same thing by explicitly passing the obj.home_view method to bottle.route after the instance is created. The advantage to doing this in the __new__ method is it will happen automatically whenever ViewClass is instantiated.

And if you go down this path then you should be aware of threads. Hey! Nice threads! Also I have a cold.

Use MacPorts: relatively easy to install and the best thing is everything is contained in a directory that you can be confident won’t eff up Apple’s stuff and won’t be effed up by Apple’s stuff.

Install Xcode

You need the compiler and bits that are installed with Xcode. If you can’t find your Mac install discs (Xcode is included with every new Mac but not installed) you can download it from Apple’s developer website. Registration is required but is free.

The current version of Xcode is easy to find, while older versions are available in the downloads section under “Developer Tools”. Xcode version 3.1.4 is the last version that will work for Mac OS X 10.5 systems.

Install MacPorts

MacPorts have a nice pkg installer. You can also build it from source.

curl -O http://distfiles.macports.org/MacPorts/MacPorts-1.9.1-10.5-Leopard.dmg
hdiutil attach MacPorts-1.9.1-10.5-Leopard.dmg
sudo installer -pkg /Volumes/MacPorts-1.9.1/MacPorts-1.9.1.pkg -target /
hdiutil detach /Volumes/MacPorts-1.9.1

If for some reason MacPorts cannot fetch updates you may need to pull updates by hand.

Check your $PATH after installing ports to make sure /opt/local/bin is in there. If it isn’t your can do export PATH=/opt/local/bin:/opt/local/sbin:${PATH} to fix things, and even add taht line to ~/.profile so that bash picks it up every time (assuming you haven’t switched your shell).

Install software

The port command is used to manipulate the MacPorts installation. Use it to build and install the various bits we need. This takes a while, especially on old PowerPC machines. Make it more exciting by adding the --verbose flag. Exciting!

sudo port install python26
sudo port install apache2
sudo port install mysql5-server
sudo port install mod_python26
sudo port install py26-mysql
sudo port install py26-django
sudo port install py26-south

And if you want to hook Django into a network directory then you almost certainly want to use LDAP.

sudo port install py26-ldap

Cool kids these days say use mod_wsgi instead of mod_python for hosting Python applications with Apache, but I am not cool (and on 20 September 2010 I couldn’t persuade mod_wsgi to build from MacPorts on a clean installation).

Configuring and starting MySQL

UPDATED: commenter matea pointed to Jason Rowland’s MySQL on Mac posting that includes steps to secure a default installation, so I’ve updated this section with the appropriate steps.

I always seem to be the only person who cares about non-English visitors… anyway, so I want to have MySQL use UTF8 for everything. Edit the configuration so it does. As root, create a configuration at /opt/local/var/db/mysql5/my.cnf with these lines:

[mysqld]
init-connect = 'SET NAMES utf8'
character-set-server = utf8
collation-server = utf8_general_ci
skip-networking

[mysql]
default-character-set = utf8

One thing about the line skip-networking in the configuration file is that it means MySQL will not listen to any network clients, including connections to 127.0.0.1. Instead clients should connect to localhost or they should specify the path to the socket that MySQL uses for communication. If your MySQL “client” is a Django instance running on the same host then that should not be a problem.

Now initialize the database and start the server. (The use of -w in the second line tells launchctl to have the database daemon start at boot. If you don’t want to have MySQL running at boot use -F to force start just this one time instead of every time.)

sudo -u mysql /opt/local/bin/mysql_install_db5
sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist

And let’s check that the server is up and configured right.

/opt/local/bin/mysql5 -e "SHOW variables LIKE '%char%'"

You should see a table showing that the character set for the client and server is set to utf8.

Now run the secure installation script for MySQL. This will ask you to set a password for MySQL’s root account (the administrator) and ask whether to remove the test database and anonymous user access (you should do both):

/opt/local/bin/mysql_secure_installation5

Thaz better.

Configuring Postgresql instead of MySQL

If you want to use Postgres instead of MySQL then you need a couple different packages out of ports.

sudo port install postgresql84-server
sudo port install py26-psycopg2

Did you know Apple’s management tools use Postgres? Is true.

Configuring Apache to serve a Django project

Let’s suppose your Django project lives under /Library/WebServer/example.com/myproj, and the project’s settings file is /Library/WebServer/example.com/myproj/settings.py. Here’s how to configure Apache with mod_python to serve your project.

Create a separate site configuration for Apache in /Library/WebServer/example.com/site.conf.

<Location "/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE myproj.settings
    PythonOption django.root /
    PythonDebug On
    PythonPath "['/Library/WebServer/example.com'] + sys.path"
</Location>

<Directory /Library/WebServer/example.com>
    Order deny,allow
    Allow from all
</Directory>

Of course once everything is hunky dory you will go back and edit the site configuration so that PythonDebug Off.

And finally tell Apache to use mod_python and read the site configuration. Edit /opt/local/apache2/conf/httpd.conf and add a line at the end of the modules like:

LoadModule python_module modules/mod_python.so

And then a line like:

Include /Library/WebServer/example.com/site.conf

Now fire up Apache:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist

MacPorts has a convenient shortcut for this:

sudo port load apache2

You also want to save Apache a little grief by pre-compiling the Python source files for the project:

/opt/local/bin/python2.6 -m compileall /Library/WebServer/example.com

Hope this helps.

Updated: I have re-written this post and the example to make it simpler following Marcel Hellkamp’s comments (Marcel is the primary author of Bottle). My original example was needlessly complicated.

It is possible to have a Django-style urlpatterns stanza with a Bottle app. Here’s how it can work:

from bottle import route

# Assuming your *_page view functions are defined above somewhere
urlpatterns = (
    # (path, func, name)
    ('/', home_page, 'home'),
    ('/about', about_page, 'about'),
    ('/contact', contact_page, 'contact'),
)

for path, func, name in urlpatterns:
    route(path, name=name)(func)

Here we run through a list where each item is a triple of URL path, view function and a name for the route. For each we simply call the route method and then invoke it with the function object. Not as flexible as using the decorator on a function (because the @route decorator can take additional keyword arguments) but at least you can have all the routes in one place at the end of the module.

Then again if you have so many routes that you need to keep them in a pretty list you probably aren’t writing the simple application that Bottle was intended for.

(This was tested with Bottle’s 0.8 and 0.9-dev branches.)

The newly-decorated function can then do things like checking the called arguments before invoking the original un-decorated function.

Django provides decorators for authentication so that you can wrap a view function with a check for client credentials before deciding whether to return the original response or a deny access.

In this manner Django’s authentication decorators encourage orthogonal code: the logic for displaying a view is separated from the logic for deciding whether you should be permitted to see the view’s output. By keeping them separate, it becomes simpler to re-use the authentication logic and apply it to other views.

Suppose you have a view that accepts a Django request object and checks whether the user is signed in:

def administration_page(request):
    if request.user.is_authenticated():
        return HttpResponse("Welcome, dear user.")
    else:
        return HttpResponseRedirect("/signin/")

With a decorator you can simplify and clarify things:

@login_required
def administration_page(request):
    return HttpResponse("Welcome, dear user.")

For older versions of Python (pre 2.4) which don’t understand the @ operator one must explicitly decorate the view function like so:

def administration_page(request):
    return HttpResponse("Welcome, dear administrator.")

administration_page = login_required(administration_page)

Note in the example that the original administration_page function is passed to the decorator. The @ syntax in the first example makes that implicit but the two are equivalent.

The implementation of a decorator is interesting. It takes the function itself as an argument and returns a new function which does the actual checking. Here is how the decorator used above might do its stuff:

def login_required(view_function):
    def decorated_function(request):
        if request.user.is_authenticated():
            return view_function(request)
        else:
            return HttpResponseRedirect("/signin/")

    return decorated_function

The actual implementation of Django’s login_required decorator is considerably less idiotic. Python’s functools module has helpers for writing well-behaved decorators.

Because functions in Python are themselves objects the decorator can accept a function reference, construct a new function that checks for authentication and then return a reference to that new function.

Simples!

(Simples gets less simples when you want to write a decorator that accepts configuration arguments because you then need either another layer of nested function definitions or a class whose instances can be called directly, but I’m going to ignore you for a bit and wow is that Concorde…?)

Django uses an instance of ModelAdmin (defined in django.contrib.admin.options) to handle the request for a model object add / change view in the admin site. ModelAdmin.add_view and ModelAdmin.change_view are responsible for populating the template context when rendering the add object and change object pages respectively.

Here are the keys common to add and change views:

• title, ‘Add ‘ or ‘Change ‘ + your model class’ _meta.verbose_name
• adminform is an instance of AdminForm
• is_popup, a boolean which is true when _popup is passed as a request parameter
• media is an instance of django.forms.Media
• inline_admin_formsets is a list of InlineAdminFormSet objects
• errors is an instance of AdminErrorList
• root_path is the root_path attribute of the AdminSite object
• app_label is your model class’ _meta.app_label attribute

The way that Django renders a form in the admin view is to iterate over the adminform instance and then iterate over each FieldSet which in turn yield AdminField instances. All I want to do is layout the form fields, ignoring the fieldset groupings which may or may not be defined in the model’s ModelAdmin.fieldset attribute.

This turns out to be easy once you know how. The regular form is an attribute of the adminform object. So if your model has a field named “king_of_pop” you can refer to the form field in your template like so:

{{ adminform.form.king_of_pop.label_tag }}: {{ adminform.form.king_of_pop }}

Or if you want to save your finger tips you can use the with template tag:

{% with adminform.form as f %}
{{ f.king_of_pop.label_tag }}: {{ f.king_of_pop }}
{% endwith %}

Delving through the Django source while I tried to understand all of this I was struck by how Python defines hook functions for iteration and accessing attributes. Half of Python’s attraction is in how easy it is from the program author’s point of view to treat objects as built-in types like lists, dicts, etc.; the other half is the responsibility of the author of a Python module to encourage that same ease of use by implementing the related iteration protocols. It is harder to write a good Python module than it is to write a good Python program that uses a good module.

A suitable ModelForm for your Django model will consume each row and do the conversion of each field to an appropriate Python type. Much simpler than explicitly converting each value yourself before creating a new model instance.

Suppose you have a model for an address book entry and its associated ModelForm (this works for Django 1.1):

# myapp/models.py
from django.db import models
from django import forms

class Contact(models.Model):
    first_name = models.CharField(max_length=100)
    second_name = models.CharField(max_length=100)
    telephone = models.CharField(max_length=50, blank=True)
    email = models.EmailField(blank=True)

class ContactForm(forms.ModelForm):
    class Meta:
        model = Contact

Here’s a script to run through a comma-separated list of contacts where each line looks something like “Smits, Jimmy, jimmy@example.com, 555-1234″:

from myapp.models import ContactForm

# Map columns to fields, adjusting the order as necessary
column_map = (
    'second_name',
    'first_name',
    'email',
    'telephone',
)

for line in open('tab-separated-data.txt'):
    row = dict(zip(column_map, (field.strip() for field in line.split(','))))
    form_obj = ContactForm(row)
    try:
        form_obj.save()
    except ValueError:
        for k, v in form_obj.errors.items():
            print k, row[k], ', '.join(map(unicode, v))

If a line doesn’t validate the script prints the validation errors and moves to the next line. If your data has columns you want to ignore then just name them in the column_map – the form class will ignore extra keys in the dictionary.

However what bit me was I wanted to use a customised set of admin media and had configured a view for the ADMIN_MEDIA_URL path and was going batty trying to work out why Django was ignoring it. It used to be that as long as you had DEBUG = False in settings.py then the development server did not try to help serve the admin media automatically.

Changeset 6075 added a switch to the runserver command for over-riding the admin media directory.

python manage.py runserver --adminmedia /path/to/custom/media

That change was made more than two years ago. It is right there in the documentation. A little bit of magic that wasted fifteen minutes of my frantic schedule (except for the fact I do not have a frantic schedule).

At some point in the process of re-writing each database for use with Django I have needed to convert all the records from Filemaker to Django. There exist good Python libraries for talking to Filemaker but they rely on the XML Web interface, meaning that you need Filemaker running and set to publish the database on the Web while you are running an import.

In my experience Filemaker’s built-in XML publishing interface is too slow when you want to migrate tens of thousands of records. During development of a Django-based application I find I frequently need to re-import the records as the new database schema evolves – doing this by communicating with Filemaker is tedious when you want to re-import the data several times a day.

So my approach has been to export the data from Filemaker as XML using Filemaker’s FMPXMLRESULT format. The Filemaker databases at work are old (Filemaker 5.5) and perhaps things have improved in more recent versions but Filemaker 5/6 is a very poor XML citizen. When using the FMPDSORESULT format (which has been dropped from more recent versions) it will happily generate invalid XML all over the shop. The FMPXMLRESULT format is better but even then it will emit invalid XML if the original data happens to contain funky characters.

So here is filemaker.py, a Python module for parsing an XML file produced by exporting to FMPXMLRESULT format from Filemaker.

To use it you create a sub-class of the FMPImporter class and over-ride the FMPImporter.import_node method. This method is called for each row of data in the XML file and is passed an XML node instance for the row. You can convert that node to a more useful dictionary where keys are column names and values are the column values. You would then convert the data to your Django model object and save it.

A trivial example:

import filemaker

class MyImporter(filemaker.FMPImporter):
    def import_node(self, node):
        node_dict = self.format_node(node)
        print node['RECORDID'], node_dict

importer = MyImporter(datefmt='%d/%m/%Y')
filemaker.importfile('/path/to/data.xml', importer=importer)

The FMPImporter.format_node method converts values to an appropriate Python type according to the Filemaker column type. Filemaker’s DATE and TIME types are converted to Python datetime.date and datetime.time instances respectively. NUMBER types are converted to Python float instances. Everything else is left as strings, but you can customize the conversion by over-riding the appropriate methods in your sub-class (see the source for the appropriate method names).

In the case of Filemaker DATE values you can pass the datefmt argument to your sub-class to specify the date format string. See Python’s time.strptime documentation for the complete list of the format specifiers.

The code uses Python’s built-in SAX parser so that it is efficent when importing huge XML files (the process uses a constant 15 megabytes for any size of data on my Mac running Python 2.5).

Fortunately I haven’t had to deal with Filemaker’s repeating fields so I have no idea how the code works on repeating fields. Please let me know if it works for you. Or not.

Download filemaker.py. This code is released under a 2-clause BSD license.

Django’s documentation includes an example of how to export data in comma-separated values (CSV) format. CSV has the significant advantage of being a standard Python module as well as being a relatively simple and non-vendor specific format. However there are some disadvantages to using CSV:

1. Values can only be stored as strings or numbers.
2. Unicode text must be explicitly encoded as UTF-8.
3. Users are often unfamiliar with the .csv file name extension – “What the hell do I do with this damn you?”

It would be unfriendly of me to expect a user to open a CSV file and then format a column of date strings as proper date values (especially when the user is almost certainly using Excel already). So I choose Excel format over CSV format.

Dates in Excel documents (97/2004 format) are actually stored as numbers. In order to have them appear as dates one must apply a date formatting. You do this by using xlwt.easyxf to create a suitable style instance and then pass that when writing the cell data.

A word of advice: do not instantiate style objects more than once! My initial approach created a new style whenever writing a date/time value. Only once I was testing with more than a few dozen rows did I discover that Excel will grow grumpy and complain about too many fonts being open when trying to display the spreadsheet. The correct approach is to have one instance for each different style and then re-use that instance for the appropriate type of value.

Here is an example that writes all objects of one class to a spreadsheet and sends that file to the client’s browser. You could stuff this in a Django view method.

from datetime import datetime, date
from django.http import HttpResponse
from myproject.myapp.models import MyModel
import xlwt


book = xlwt.Workbook(encoding='utf8')
sheet = book.add_sheet('untitled')

default_style = xlwt.Style.default_style
datetime_style = xlwt.easyxf(num_format_str='dd/mm/yyyy hh:mm')
date_style = xlwt.easyxf(num_format_str='dd/mm/yyyy')

values_list = MyModel.objects.all().values_list()

for row, rowdata in enumerate(values_list):
    for col, val in enumerate(rowdata):
        if isinstance(val, datetime):
            style = datetime_style
        elif isinstance(val, date):
            style = date_style
        else:
            style = default_style

        sheet.write(row, col, val, style=style)

response = HttpResponse(mimetype='application/vnd.ms-excel')
response['Content-Disposition'] = 'attachment; filename=example.xls'
book.save(response)
return response

That code works a peach with a 30,000 row / 25 column database, taking about a minute to generate a 13 megabyte file on my lowly iMac G5.

You want to buy me a new Intel iMac, don’t you? Yes, you do.

I posted a module for handling time zone-aware datetime objects, but I left out all the hassle of dealing with form input. Here is a more complete python package for Django that includes a form field sub-class that can handle a small set of datetime string formats that include a time zone offset.

Timezones 0.1

This code is released under Django’s BSD license.

You can use grep to remove any lines that do not match your pattern:

>>> s = 'The quick brown fox'
>>> grep(s, 'quick')
u'The quick brown fox'

And its converse grepv to remove any lines that do match your pattern:

>>> s = 'The quick brown fox'
>>> grepv(s, 'quick')
''
>>> s2 = s + '\nJumps over the lazy dog'
>>> grepv(s2, 'quick')
u'Jumps over the lazy dog'

Stick it in a module in your Django application (documentation), then load it up at the top of a template.

from django import template
from django.template.defaultfilters import stringfilter
import re


register = template.Library()


@register.filter
@stringfilter
def grep(value, arg):
    """Lines that do not match the regular expression are removed."""
    pattern = re.compile(arg)
    lines = [line for line in re.split(r'[\r\n]', value) if pattern.search(line)]
    return '\n'.join(lines)


@register.filter
@stringfilter
def grepv(value, arg):
    """Lines that match the regular expression are removed."""
    pattern = re.compile(arg)
    lines = [line for line in re.split(r'[\r\n]', value) if not pattern.search(line)]
    return '\n'.join(lines)

This is understandable given the database-agnostic nature of the Django ORM: although PostgreSQL 8.3 supports a datetime type which is time zone-aware, MySQL 5.1 does not (I have no idea what SQLite does about time zones). By ignoring time zones, Django works with the lowest common denominator.

Given time zone support in Postgres, there is a chunk of work to write a variation of models.DateTimeField which can handle time zone-wise datetimes. Python 2.5 does not help things – Python’s native datetime module is similarly agnostic about time zones, the standard library does not include a module for handling wise datetimes.

(If regular datetime instances are naive then datetime instances that honour time zones are wise.)

Django does make it pretty easy to write a custom field class, which means it shouldn’t be too difficult to write a custom datetime field class that is time zone-wise. As ever it is the Django project’s regard for documentation that transforms that which is possible into that which is practical.

Given your backend database has a time zone-wise datetime type (i.e. PostgreSQL), what input values does one need to handle in a time zone-wise custom field class?

• value set to None
• value set to a naive datetime instance
• value set to a wise datetime instance
• value set to a naive datetime string
• value set to a wise datetime string

Now the essence of a custom field in Django is two methods: to_python and get_db_prep_value. If the custom field defines

__metaclass__ = models.SubfieldBase

then the to_python method will be called any time a value is assigned to the field, and we can make sure that a suitable type is returned before the model object is saved. Because Postgres supports time zone-wise datetimes and if we take care to return a wise datetime instance we can ignore get_db_prep_value.

When Django reads a record from the database it strips the time zone information, effectively giving your custom field a naive datetime string that belongs to the same time zone as the database connection object. (At least this seems to be true for Postgres and the psycopg2 adaptor.) And since the database connection sets the time zone to be the same as set by settings.TIME_ZONE your custom class needs to treat any naive datetime strings as belonging to the time zone set with settings.TIME_ZONE.

So this leads to the important behaviour for a time zone-wise DateTimeField sub-class: always convert naive datetimes to the time zone set in settings.TIME_ZONE.

For convenience my custom field class, the TZDateTimeField, returns a sub-class of Python’s datetime which has an extra method that converts the datetime to the zone defined by the project’s time zone. Therefore whether the field value has been set from a naive or wise datetime instance, or a naive or wise date string you will end up with a time zone-wise value and you can get the value converted to the project’s time zone. This extra method is intended for use in a Django template.

What I was hoping was that the backend would store the datetime as a datetime in an arbitrary zone, potentially a different time zone from one record to the next for the same field. That behaviour would allow one to infer that one datetime value was created in this time zone while another datetime value was created in that time zone. Instead all datetime values are effectively normalized to your Django project’s time zone.

So here is an example of a model class that uses my time zone-aware datetime field. It ought to work just like a regular DateTimeField but always stores a time zone-aware datetime instance:

from django.db import models
from timezones.fields import TZDateTimeField
from datetime import datetime


class Article(models.Model):
    pub_date = TZDateTimeField(default=datetime.now)

And below is my custom field definition, which has a dependency on the pytz module to handle all the difficult stuff. You can grab the complete module over here, including tests in doctest format. The tests are intended to be run by Django’s manage.py test management command, and so one needs to add the module to the list of installed apps.

"""A time zone-aware DateTime field.

When saving, naive datetime objects are assumed to belong to the local time
zone and are converted to UTC. When loading from the database the naive datetime
objects are converted to UTC.

These field types require database support. MySQL 5 will not work.
"""
from datetime import datetime, tzinfo, timedelta
from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import models
import pytz
import re


# 2009-06-04 12:00:00+01:00 or 2009-06-04 12:00:00 +0100
TZ_OFFSET = re.compile(r'^(.*?)\s?([-\+])(\d\d):?(\d\d)$')


class TZDatetime(datetime):
    def aslocaltimezone(self):
        """Returns the datetime in the local time zone."""
        tz = pytz.timezone(settings.TIME_ZONE)
        return self.astimezone(tz)


class TZDateTimeField(models.DateTimeField):
    """A DateTimeField that treats naive datetimes as local time zone."""
    __metaclass__ = models.SubfieldBase

    def to_python(self, value):
        """Returns a time zone-aware datetime object.

        A naive datetime is assigned the time zone from settings.TIME_ZONE.
        This should be the same as the database session time zone.
        A wise datetime is left as-is. A string with a time zone offset is
        assigned to UTC.
        """
        try:
            value = super(TZDateTimeField, self).to_python(value)
        except ValidationError:
            match = TZ_OFFSET.search(value)
            if match:
                value, op, hours, minutes = match.groups()
                value = super(TZDateTimeField, self).to_python(value)
                value = value - timedelta(hours=int(op + hours), minutes=int(op + minutes))
                value = value.replace(tzinfo=pytz.utc)
            else:
                raise

        if value is None:
            return value

        # Only force zone if the datetime has no tzinfo
        if (value.tzinfo is None) or (value.tzinfo.utcoffset(value) is None):
            value = force_tz(value, settings.TIME_ZONE)
        return TZDatetime(value.year, value.month, value.day, value.hour,
            value.minute, value.second, value.microsecond, tzinfo=value.tzinfo)


def force_tz(obj, tz):
    """Converts a datetime to the given timezone.

    The tz argument can be an instance of tzinfo or a string such as
    'Europe/London' that will be passed to pytz.timezone. Naive datetimes are
    forced to the timezone. Wise datetimes are converted.
    """
    if not isinstance(tz, tzinfo):
        tz = pytz.timezone(tz)

    if (obj.tzinfo is None) or (obj.tzinfo.utcoffset(obj) is None):
        return tz.localize(obj)
    else:
        return obj.astimezone(tz)

The simplest route is to customize the project settings.py as part of the site deployment, but that will drive you insane when you deploy the wrong custom-settings to a site.

Another approach is similar to that used by many for switching between settings when moving between testing / staging / live environments: your settings.py has a few lines something like

try:
    from sitesettings import *
except ImportError:
    pass

so you can over-ride any setting by putting them in a sitesettings.py file, and then make sure your deployment never overwrites that site-specific file.

In my case I want to make it easy for the site administrator to customize the settings, but I am worried that it is too easy for someone who does not know Python syntax to inadvertently break things by writing a sitesettings.py that throws a SyntaxError exception. Given the significance of white-space in Python I feel this would be easy to get wrong.

So I’ve gone for storing the custom settings in Mac OS X’s property list format. Bless Python for it has the plistlib module that reads and writes the simple XML format of property lists.

Here’s my module that imports all properties from a plist straight into the module’s namespace. This then makes it easy to over-ride Django’s settings by doing

from plistsettings import *

A couple bits made my lips move during the writing. The contents of __all__ are updated dynamically because I wanted to use this with from plistingsettings import * without worrying that my module’s imports would get clobbered by imports used in the plistsettings module. And working out how to bind keys and values to the module itself is not obvious to me - it feels like one ought to be able to use self within the scope of the module to refer to the module itself. Except you can’t. No biggie.

# plistsettings.py
import os.path
import plistlib
import sys
from xml.parsers.expat import ExpatError


__all__ = []


PLIST_PATH = '/Library/Preferences/com.example.plist'


def read_prefs(plist_path):
    """Import settings from preference file into this module's global namespace.

    Returns a dictionary as returned by plistlib.readPlist().
    """
    try:
        if os.path.exists(plist_path):
            prefs = plistlib.readPlist(plist_path)
        else:
            return
    except ExpatError:
        return

    mod = sys.modules[__name__]
    global __all__

    for key, value in prefs.items():
        setattr(mod, key, value)
        __all__.append(key)
    return prefs

read_prefs(PLIST_PATH)

Now if you are the kind of Mac guy who enjoys using defaults you can write out your site-specific settings from the command-line.

defaults write /Library/Preferences/com.example EMAIL_HOST smtp.example.com
plutil -convert xml1 /Library/Preferences/com.example.plist

N.B. Mac OS X 10.5 defaults uses the binary format by default, so you need plutil to convert it back to XML because plistlib does not handle the binary format.

For a simple site search feature I needed to search for a term across several fields in a model. Suppose the model looks like this:

class BlogPost(models.Model):
    title = models.CharField(max_length=100)
    body = models.TextField()
    summary = models.TextField()

And you have a view method that accepts a parameter q for searching across the title, body and summary fields. I want to find objects that contain the q phrase in any of those fields. I need to build a QuerySet with a filter that is the equivalent of

queryset = BlogPost.objects.filter(
    Q(title__icontains=q) | Q(body__icontains=q) | Q(summary__icontains=q)
)

That’s not too much of a hassle for this simple example, but in cases where the fields you are searching are chosen dynamically, or where you just have an awful lot of fields to search against, I think it is nicer to do it like so:

import operator

search_fields = ('title', 'body', 'summary')
q_objects = [Q(**{field + '__icontains':q}) for field in search_fields]
queryset = BlogPost.objects.filter(reduce(operator.or_, q_objects))

Nice one! The list comprehension gives me a list of Q objects generated from the names in search_fields, so it is easy to change the fields to be searched. And using reduce and operator.or_ gives me the required OR filter in one line.

I see for Python 3 reduce has been moved to the functools module.

This stuff never used to be that obvious to me. It kind of isn’t even now.

P.S. I promise I am not writing a blog engine at this time, it was just for the example.

It all started so innocently… I had wanted a set of urls for my application like this:

• http://example.com/a/ # List view of arrivals
• http://example.com/d/ # List view of departures
• http://example.com/a/feed/ # Syndication feed for arrivals
• http://example.com/d/feed/ # Syndication feed for departures

So I put the following in the application’s urls.py:

# myapp/urls.py
from django.conf.urls.defaults import *
from views import arrivals_list, departures_list
from feeds import LatestArrivals, LatestDepartures


feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}


urlpatterns = patterns('',
    (r'^a/$', arrivals_list, {}, 'arrivals'),
    (r'^d/$', departures_list, {}, 'departures'),
    (r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)

That covers my URL wishes, and because I have named the URL patterns I can use that name in templates with the {% url %} template tag and in Python code using the reverse helper.

So naturally the feed classes in feeds.py look like this:

# myapp/feeds.py
from django.contrib.syndication.feeds import Feed
from django.core.urlresolvers import reverse
from django.utils.feedgenerator import Atom1Feed
from models import Tx


class LatestArrivals(Feed):
    """Produces an Atom feed of recent arrival tickets."""
    feed_type = Atom1Feed
    title = 'Arrivals'
    link = reverse('arrivals')
    subtitle = 'Most recent arrivals'

    def items(self):
        return Tx.objects.arrivals()[:10]


class LatestDepartures(Feed):
    """Produces an Atom feed of recent departure tickets."""
    feed_type = Atom1Feed
    title = 'Departures'
    link = reverse('departures')
    subtitle = 'Most recent departures'

    def items(self):
        return Tx.objects.departures()[:10]

Note I used reverse on the link attribute of each class so that I can define the URL in one place, the urls.py module, and a change there will be reflected in the feed’s link too.

But this doesn’t work! When Django imports my urls.py module, it imports LatestDepartures and LatestArrivals, and they in turn use reverse to find the named URL patterns – except those names aren’t defined until after urlpatterns has been defined in urls.py so Django throws an exception and never imports my urls.py module.

You could work around this either by defining your syndication feeds in an entirely different urls.py module. But you can also split up urlpatterns within the same module and import the feed classes after their named URL patterns have been defined.

Here’s the working urls.py module:

from django.conf.urls.defaults import *
from views import arrivals_list, departures_list


urlpatterns = patterns('',
    (r'^a/$', arrivals_list, {}, 'arrivals'),
    (r'^d/$', departures_list, {}, 'departures'),
)


from feeds import LatestArrivals, LatestDepartures
feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}


urlpatterns += patterns('',
    (r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)

from django import forms


class LetterChoices(object):
    """Return a random list of max_choices letters of the alphabet."""
    def __init__(self, max_choices=3):
        self.max_choices = max_choices

    def __iter__(self):
        import string, random

        return iter((l, l) for l in random.sample(string.ascii_uppercase, self.max_choices))


class LetterForm(forms.Form):
    """Pick a letter from a small, random set."""
    letter = forms.ChoiceField(choices=LetterChoices())

I don’t know if I prefer that style to having a simple function – having to instantiate the class seems wrong to me, I’d much rather use any callable as the choices argument.

from contextlib import contextmanager


@contextmanager
def test_db_connection():
    """A context manager for Django's test runner.

    For Python 2.5 you will need
        from __future__ import with_statement
    """

    from django.conf import settings
    from django.test.utils import setup_test_environment, teardown_test_environment
    from django.db import connection

    setup_test_environment()

    settings.DEBUG = False    
    verbosity = 0
    interactive = False

    old_name = settings.DATABASE_NAME
    connection.creation.create_test_db(verbosity, autoclobber=not interactive)

    yield connection

    connection.creation.destroy_test_db(old_name, verbosity)
    teardown_test_environment()

All of this requires Python 2.5 or later.

So with that snippet you could write a test something like so:

import unittest


class MyTestCase(unittest.TestCase):
    def test_myModelTest(self):
        with test_db_connection():
            from myproject.myapp.models import MyModel

            obj = MyModel()
            obj.save()
            self.assert_(obj.pk)

… and just as with Django’s manage.py test command the objects would be created within the test database then destroyed when the with test_db_connection() block is finished.

Everything’s going to be hunky dory.

As of Django 1.0.2 the default behaviour for the test runner is the run_tests function in django.test.simple. Here is the bones of that function with the required setup and teardown calls.

from django.conf import settings
from django.test.utils import setup_test_environment, teardown_test_environment


verbosity = 1
interactive = True

setup_test_environment()
settings.DEBUG = False    
old_name = settings.DATABASE_NAME

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)

# Here you run tests using the test database and with mock SMTP objects

connection.creation.destroy_test_db(old_name, verbosity)
teardown_test_environment()

Hmmm… Wouldn’t this be a good candidate to be wrapped up for use with Python 2.5′s with statement?

Fortunately this was covered long ago by James Bennett when Django’s newforms module was introduced. See Getting dynamic model choices in newforms on Django Snippets.

The following form example has a field for picking a letter of the alphabet (works for Django 1.0). The choices are limited to 3 letters only, picked at random and different for each form instance:

from django import forms


def letter_choices(max_choices=3):
    """Return a random list of max_choices letters of the alphabet."""
    import string, random

    for l in random.sample(string.ascii_uppercase, max_choices):
        yield (l, l)


class LetterForm(forms.Form):
    """Pick a letter from a small, random set."""
    letter = forms.ChoiceField(choices=letter_choices())

    def __init__(self, *args, **kwargs):
        super(LetterForm, self).__init__(*args, **kwargs)
        self.fields['letter'].choices = letter_choices()

So that works. The LetterForm class uses the helper function letter_choices to provide the random choices, which actually returns a generator object rather than a list or tuple of choice pairs. I am relying on Django’s base ChoiceField class calling list() on the choices when each form is instantiated, so having letter_choices return a generator is hunky dory.

>>> f1 = LetterForm()
>>> f1['letter']
>>> print f1['letter']
<select name="letter" id="id_letter">
<option value="R">R</option>
<option value="N">N</option>
<option value="U">U</option>
</select>
>>> f2 = LetterForm()
>>> print f2['letter']
<select name="letter" id="id_letter">
<option value="O">O</option>
<option value="N">N</option>
<option value="T">T</option>
</select>
>>> 

Now the only thing is… this example is not practical. Using genuinely random choices means that the valid choices on the form submitted by the user will be different to the valid choices on the form used to validate the user input on the next request, and this will likely raise a ValidationError.

Django feature suggestion: allow choices to be any iterable or callable, calling it as appropriate when instantiating the field. If it is callable you could pass a function which returns an iterable at that point, which would save one having to write an __init__ method for the form sub-class.