Updated: I have re-written this post and the example to make it simpler following Marcel Hellkamp’s comments (Marcel is the primary author of Bottle). My original example was needlessly complicated.

It is possible to have a Django-style urlpatterns stanza with a Bottle app. Here’s how it can work:

from bottle import route

# Assuming your *_page view functions are defined above somewhere
urlpatterns = (
    # (path, func, name)
    ('/', home_page, 'home'),
    ('/about', about_page, 'about'),
    ('/contact', contact_page, 'contact'),
)

for path, func, name in urlpatterns:
    route(path, name=name)(func)

Here we run through a list where each item is a triple of URL path, view function and a name for the route. For each we simply call the route method and then invoke it with the function object. Not as flexible as using the decorator on a function (because the @route decorator can take additional keyword arguments) but at least you can have all the routes in one place at the end of the module.

Then again if you have so many routes that you need to keep them in a pretty list you probably aren’t writing the simple application that Bottle was intended for.

(This was tested with Bottle’s 0.8 and 0.9-dev branches.)

The newly-decorated function can then do things like checking the called arguments before invoking the original un-decorated function.

Django provides decorators for authentication so that you can wrap a view function with a check for client credentials before deciding whether to return the original response or a deny access.

In this manner Django’s authentication decorators encourage orthogonal code: the logic for displaying a view is separated from the logic for deciding whether you should be permitted to see the view’s output. By keeping them separate, it becomes simpler to re-use the authentication logic and apply it to other views.

Suppose you have a view that accepts a Django request object and checks whether the user is signed in:

def administration_page(request):
    if request.user.is_authenticated():
        return HttpResponse("Welcome, dear user.")
    else:
        return HttpResponseRedirect("/signin/")

With a decorator you can simplify and clarify things:

@login_required
def administration_page(request):
    return HttpResponse("Welcome, dear user.")

For older versions of Python (pre 2.4) which don’t understand the @ operator one must explicitly decorate the view function like so:

def administration_page(request):
    return HttpResponse("Welcome, dear administrator.")

administration_page = login_required(administration_page)

Note in the example that the original administration_page function is passed to the decorator. The @ syntax in the first example makes that implicit but the two are equivalent.

The implementation of a decorator is interesting. It takes the function itself as an argument and returns a new function which does the actual checking. Here is how the decorator used above might do its stuff:

def login_required(view_function):
    def decorated_function(request):
        if request.user.is_authenticated():
            return view_function(request)
        else:
            return HttpResponseRedirect("/signin/")

    return decorated_function

The actual implementation of Django’s login_required decorator is considerably less idiotic. Python’s functools module has helpers for writing well-behaved decorators.

Because functions in Python are themselves objects the decorator can accept a function reference, construct a new function that checks for authentication and then return a reference to that new function.

Simples!

(Simples gets less simples when you want to write a decorator that accepts configuration arguments because you then need either another layer of nested function definitions or a class whose instances can be called directly, but I’m going to ignore you for a bit and wow is that Concorde…?)

Django uses an instance of ModelAdmin (defined in django.contrib.admin.options) to handle the request for a model object add / change view in the admin site. ModelAdmin.add_view and ModelAdmin.change_view are responsible for populating the template context when rendering the add object and change object pages respectively.

Here are the keys common to add and change views:

• title, ‘Add ‘ or ‘Change ‘ + your model class’ _meta.verbose_name
• adminform is an instance of AdminForm
• is_popup, a boolean which is true when _popup is passed as a request parameter
• media is an instance of django.forms.Media
• inline_admin_formsets is a list of InlineAdminFormSet objects
• errors is an instance of AdminErrorList
• root_path is the root_path attribute of the AdminSite object
• app_label is your model class’ _meta.app_label attribute

The way that Django renders a form in the admin view is to iterate over the adminform instance and then iterate over each FieldSet which in turn yield AdminField instances. All I want to do is layout the form fields, ignoring the fieldset groupings which may or may not be defined in the model’s ModelAdmin.fieldset attribute.

This turns out to be easy once you know how. The regular form is an attribute of the adminform object. So if your model has a field named “king_of_pop” you can refer to the form field in your template like so:

{{ adminform.form.king_of_pop.label_tag }}: {{ adminform.form.king_of_pop }}

Or if you want to save your finger tips you can use the with template tag:

{% with adminform.form as f %}
{{ f.king_of_pop.label_tag }}: {{ f.king_of_pop }}
{% endwith %}

Delving through the Django source while I tried to understand all of this I was struck by how Python defines hook functions for iteration and accessing attributes. Half of Python’s attraction is in how easy it is from the program author’s point of view to treat objects as built-in types like lists, dicts, etc.; the other half is the responsibility of the author of a Python module to encourage that same ease of use by implementing the related iteration protocols. It is harder to write a good Python module than it is to write a good Python program that uses a good module.

A suitable ModelForm for your Django model will consume each row and do the conversion of each field to an appropriate Python type. Much simpler than explicitly converting each value yourself before creating a new model instance.

Suppose you have a model for an address book entry and its associated ModelForm (this works for Django 1.1):

# myapp/models.py
from django.db import models
from django import forms

class Contact(models.Model):
    first_name = models.CharField(max_length=100)
    second_name = models.CharField(max_length=100)
    telephone = models.CharField(max_length=50, blank=True)
    email = models.EmailField(blank=True)

class ContactForm(forms.ModelForm):
    class Meta:
        model = Contact

Here’s a script to run through a comma-separated list of contacts where each line looks something like “Smits, Jimmy, jimmy@example.com, 555-1234″:

from myapp.models import ContactForm

# Map columns to fields, adjusting the order as necessary
column_map = (
    'second_name',
    'first_name',
    'email',
    'telephone',
)

for line in open('tab-separated-data.txt'):
    row = dict(zip(column_map, (field.strip() for field in line.split(','))))
    form_obj = ContactForm(row)
    try:
        form_obj.save()
    except ValueError:
        for k, v in form_obj.errors.items():
            print k, row[k], ', '.join(map(unicode, v))

If a line doesn’t validate the script prints the validation errors and moves to the next line. If your data has columns you want to ignore then just name them in the column_map – the form class will ignore extra keys in the dictionary.

However what bit me was I wanted to use a customised set of admin media and had configured a view for the ADMIN_MEDIA_URL path and was going batty trying to work out why Django was ignoring it. It used to be that as long as you had DEBUG = False in settings.py then the development server did not try to help serve the admin media automatically.

Changeset 6075 added a switch to the runserver command for over-riding the admin media directory.

python manage.py runserver --adminmedia /path/to/custom/media

That change was made more than two years ago. It is right there in the documentation. A little bit of magic that wasted fifteen minutes of my frantic schedule (except for the fact I do not have a frantic schedule).

At some point in the process of re-writing each database for use with Django I have needed to convert all the records from Filemaker to Django. There exist good Python libraries for talking to Filemaker but they rely on the XML Web interface, meaning that you need Filemaker running and set to publish the database on the Web while you are running an import.

In my experience Filemaker’s built-in XML publishing interface is too slow when you want to migrate tens of thousands of records. During development of a Django-based application I find I frequently need to re-import the records as the new database schema evolves – doing this by communicating with Filemaker is tedious when you want to re-import the data several times a day.

So my approach has been to export the data from Filemaker as XML using Filemaker’s FMPXMLRESULT format. The Filemaker databases at work are old (Filemaker 5.5) and perhaps things have improved in more recent versions but Filemaker 5/6 is a very poor XML citizen. When using the FMPDSORESULT format (which has been dropped from more recent versions) it will happily generate invalid XML all over the shop. The FMPXMLRESULT format is better but even then it will emit invalid XML if the original data happens to contain funky characters.

So here is filemaker.py, a Python module for parsing an XML file produced by exporting to FMPXMLRESULT format from Filemaker.

To use it you create a sub-class of the FMPImporter class and over-ride the FMPImporter.import_node method. This method is called for each row of data in the XML file and is passed an XML node instance for the row. You can convert that node to a more useful dictionary where keys are column names and values are the column values. You would then convert the data to your Django model object and save it.

A trivial example:

import filemaker

class MyImporter(filemaker.FMPImporter):
    def import_node(self, node):
        node_dict = self.format_node(node)
        print node['RECORDID'], node_dict

importer = MyImporter(datefmt='%d/%m/%Y')
filemaker.importfile('/path/to/data.xml', importer=importer)

The FMPImporter.format_node method converts values to an appropriate Python type according to the Filemaker column type. Filemaker’s DATE and TIME types are converted to Python datetime.date and datetime.time instances respectively. NUMBER types are converted to Python float instances. Everything else is left as strings, but you can customize the conversion by over-riding the appropriate methods in your sub-class (see the source for the appropriate method names).

In the case of Filemaker DATE values you can pass the datefmt argument to your sub-class to specify the date format string. See Python’s time.strptime documentation for the complete list of the format specifiers.

The code uses Python’s built-in SAX parser so that it is efficent when importing huge XML files (the process uses a constant 15 megabytes for any size of data on my Mac running Python 2.5).

Fortunately I haven’t had to deal with Filemaker’s repeating fields so I have no idea how the code works on repeating fields. Please let me know if it works for you. Or not.

Download filemaker.py. This code is released under a 2-clause BSD license.

Django’s documentation includes an example of how to export data in comma-separated values (CSV) format. CSV has the significant advantage of being a standard Python module as well as being a relatively simple and non-vendor specific format. However there are some disadvantages to using CSV:

1. Values can only be stored as strings or numbers.
2. Unicode text must be explicitly encoded as UTF-8.
3. Users are often unfamiliar with the .csv file name extension – “What the hell do I do with this damn you?”

It would be unfriendly of me to expect a user to open a CSV file and then format a column of date strings as proper date values (especially when the user is almost certainly using Excel already). So I choose Excel format over CSV format.

Dates in Excel documents (97/2004 format) are actually stored as numbers. In order to have them appear as dates one must apply a date formatting. You do this by using xlwt.easyxf to create a suitable style instance and then pass that when writing the cell data.

A word of advice: do not instantiate style objects more than once! My initial approach created a new style whenever writing a date/time value. Only once I was testing with more than a few dozen rows did I discover that Excel will grow grumpy and complain about too many fonts being open when trying to display the spreadsheet. The correct approach is to have one instance for each different style and then re-use that instance for the appropriate type of value.

Here is an example that writes all objects of one class to a spreadsheet and sends that file to the client’s browser. You could stuff this in a Django view method.

from datetime import datetime, date
from django.http import HttpResponse
from myproject.myapp.models import MyModel
import xlwt


book = xlwt.Workbook(encoding='utf8')
sheet = book.add_sheet('untitled')

default_style = xlwt.Style.default_style
datetime_style = xlwt.easyxf(num_format_str='dd/mm/yyyy hh:mm')
date_style = xlwt.easyxf(num_format_str='dd/mm/yyyy')

values_list = MyModel.objects.all().values_list()

for row, rowdata in enumerate(values_list):
    for col, val in enumerate(rowdata):
        if isinstance(val, datetime):
            style = datetime_style
        elif isinstance(val, date):
            style = date_style
        else:
            style = default_style

        sheet.write(row, col, val, style=style)

response = HttpResponse(mimetype='application/vnd.ms-excel')
response['Content-Disposition'] = 'attachment; filename=example.xls'
book.save(response)
return response

That code works a peach with a 30,000 row / 25 column database, taking about a minute to generate a 13 megabyte file on my lowly iMac G5.

You want to buy me a new Intel iMac, don’t you? Yes, you do.

I posted a module for handling time zone-aware datetime objects, but I left out all the hassle of dealing with form input. Here is a more complete python package for Django that includes a form field sub-class that can handle a small set of datetime string formats that include a time zone offset.

Timezones 0.1

This code is released under Django’s BSD license.

You can use grep to remove any lines that do not match your pattern:

>>> s = 'The quick brown fox'
>>> grep(s, 'quick')
u'The quick brown fox'

And its converse grepv to remove any lines that do match your pattern:

>>> s = 'The quick brown fox'
>>> grepv(s, 'quick')
''
>>> s2 = s + '\nJumps over the lazy dog'
>>> grepv(s2, 'quick')
u'Jumps over the lazy dog'

Stick it in a module in your Django application (documentation), then load it up at the top of a template.

from django import template
from django.template.defaultfilters import stringfilter
import re


register = template.Library()


@register.filter
@stringfilter
def grep(value, arg):
    """Lines that do not match the regular expression are removed."""
    pattern = re.compile(arg)
    lines = [line for line in re.split(r'[\r\n]', value) if pattern.search(line)]
    return '\n'.join(lines)


@register.filter
@stringfilter
def grepv(value, arg):
    """Lines that match the regular expression are removed."""
    pattern = re.compile(arg)
    lines = [line for line in re.split(r'[\r\n]', value) if not pattern.search(line)]
    return '\n'.join(lines)

This is understandable given the database-agnostic nature of the Django ORM: although PostgreSQL 8.3 supports a datetime type which is time zone-aware, MySQL 5.1 does not (I have no idea what SQLite does about time zones). By ignoring time zones, Django works with the lowest common denominator.

Given time zone support in Postgres, there is a chunk of work to write a variation of models.DateTimeField which can handle time zone-wise datetimes. Python 2.5 does not help things – Python’s native datetime module is similarly agnostic about time zones, the standard library does not include a module for handling wise datetimes.

(If regular datetime instances are naive then datetime instances that honour time zones are wise.)

Django does make it pretty easy to write a custom field class, which means it shouldn’t be too difficult to write a custom datetime field class that is time zone-wise. As ever it is the Django project’s regard for documentation that transforms that which is possible into that which is practical.

Given your backend database has a time zone-wise datetime type (i.e. PostgreSQL), what input values does one need to handle in a time zone-wise custom field class?

• value set to None
• value set to a naive datetime instance
• value set to a wise datetime instance
• value set to a naive datetime string
• value set to a wise datetime string

Now the essence of a custom field in Django is two methods: to_python and get_db_prep_value. If the custom field defines

__metaclass__ = models.SubfieldBase

then the to_python method will be called any time a value is assigned to the field, and we can make sure that a suitable type is returned before the model object is saved. Because Postgres supports time zone-wise datetimes and if we take care to return a wise datetime instance we can ignore get_db_prep_value.

When Django reads a record from the database it strips the time zone information, effectively giving your custom field a naive datetime string that belongs to the same time zone as the database connection object. (At least this seems to be true for Postgres and the psycopg2 adaptor.) And since the database connection sets the time zone to be the same as set by settings.TIME_ZONE your custom class needs to treat any naive datetime strings as belonging to the time zone set with settings.TIME_ZONE.

So this leads to the important behaviour for a time zone-wise DateTimeField sub-class: always convert naive datetimes to the time zone set in settings.TIME_ZONE.

For convenience my custom field class, the TZDateTimeField, returns a sub-class of Python’s datetime which has an extra method that converts the datetime to the zone defined by the project’s time zone. Therefore whether the field value has been set from a naive or wise datetime instance, or a naive or wise date string you will end up with a time zone-wise value and you can get the value converted to the project’s time zone. This extra method is intended for use in a Django template.

What I was hoping was that the backend would store the datetime as a datetime in an arbitrary zone, potentially a different time zone from one record to the next for the same field. That behaviour would allow one to infer that one datetime value was created in this time zone while another datetime value was created in that time zone. Instead all datetime values are effectively normalized to your Django project’s time zone.

So here is an example of a model class that uses my time zone-aware datetime field. It ought to work just like a regular DateTimeField but always stores a time zone-aware datetime instance:

from django.db import models
from timezones.fields import TZDateTimeField
from datetime import datetime


class Article(models.Model):
    pub_date = TZDateTimeField(default=datetime.now)

And below is my custom field definition, which has a dependency on the pytz module to handle all the difficult stuff. You can grab the complete module over here, including tests in doctest format. The tests are intended to be run by Django’s manage.py test management command, and so one needs to add the module to the list of installed apps.

"""A time zone-aware DateTime field.

When saving, naive datetime objects are assumed to belong to the local time
zone and are converted to UTC. When loading from the database the naive datetime
objects are converted to UTC.

These field types require database support. MySQL 5 will not work.
"""
from datetime import datetime, tzinfo, timedelta
from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import models
import pytz
import re


# 2009-06-04 12:00:00+01:00 or 2009-06-04 12:00:00 +0100
TZ_OFFSET = re.compile(r'^(.*?)\s?([-\+])(\d\d):?(\d\d)$')


class TZDatetime(datetime):
    def aslocaltimezone(self):
        """Returns the datetime in the local time zone."""
        tz = pytz.timezone(settings.TIME_ZONE)
        return self.astimezone(tz)


class TZDateTimeField(models.DateTimeField):
    """A DateTimeField that treats naive datetimes as local time zone."""
    __metaclass__ = models.SubfieldBase

    def to_python(self, value):
        """Returns a time zone-aware datetime object.

        A naive datetime is assigned the time zone from settings.TIME_ZONE.
        This should be the same as the database session time zone.
        A wise datetime is left as-is. A string with a time zone offset is
        assigned to UTC.
        """
        try:
            value = super(TZDateTimeField, self).to_python(value)
        except ValidationError:
            match = TZ_OFFSET.search(value)
            if match:
                value, op, hours, minutes = match.groups()
                value = super(TZDateTimeField, self).to_python(value)
                value = value - timedelta(hours=int(op + hours), minutes=int(op + minutes))
                value = value.replace(tzinfo=pytz.utc)
            else:
                raise

        if value is None:
            return value

        # Only force zone if the datetime has no tzinfo
        if (value.tzinfo is None) or (value.tzinfo.utcoffset(value) is None):
            value = force_tz(value, settings.TIME_ZONE)
        return TZDatetime(value.year, value.month, value.day, value.hour,
            value.minute, value.second, value.microsecond, tzinfo=value.tzinfo)


def force_tz(obj, tz):
    """Converts a datetime to the given timezone.

    The tz argument can be an instance of tzinfo or a string such as
    'Europe/London' that will be passed to pytz.timezone. Naive datetimes are
    forced to the timezone. Wise datetimes are converted.
    """
    if not isinstance(tz, tzinfo):
        tz = pytz.timezone(tz)

    if (obj.tzinfo is None) or (obj.tzinfo.utcoffset(obj) is None):
        return tz.localize(obj)
    else:
        return obj.astimezone(tz)

The simplest route is to customize the project settings.py as part of the site deployment, but that will drive you insane when you deploy the wrong custom-settings to a site.

Another approach is similar to that used by many for switching between settings when moving between testing / staging / live environments: your settings.py has a few lines something like

try:
    from sitesettings import *
except ImportError:
    pass

so you can over-ride any setting by putting them in a sitesettings.py file, and then make sure your deployment never overwrites that site-specific file.

In my case I want to make it easy for the site administrator to customize the settings, but I am worried that it is too easy for someone who does not know Python syntax to inadvertently break things by writing a sitesettings.py that throws a SyntaxError exception. Given the significance of white-space in Python I feel this would be easy to get wrong.

So I’ve gone for storing the custom settings in Mac OS X’s property list format. Bless Python for it has the plistlib module that reads and writes the simple XML format of property lists.

Here’s my module that imports all properties from a plist straight into the module’s namespace. This then makes it easy to over-ride Django’s settings by doing

from plistsettings import *

A couple bits made my lips move during the writing. The contents of __all__ are updated dynamically because I wanted to use this with from plistingsettings import * without worrying that my module’s imports would get clobbered by imports used in the plistsettings module. And working out how to bind keys and values to the module itself is not obvious to me - it feels like one ought to be able to use self within the scope of the module to refer to the module itself. Except you can’t. No biggie.

# plistsettings.py
import os.path
import plistlib
import sys
from xml.parsers.expat import ExpatError


__all__ = []


PLIST_PATH = '/Library/Preferences/com.example.plist'


def read_prefs(plist_path):
    """Import settings from preference file into this module's global namespace.

    Returns a dictionary as returned by plistlib.readPlist().
    """
    try:
        if os.path.exists(plist_path):
            prefs = plistlib.readPlist(plist_path)
        else:
            return
    except ExpatError:
        return

    mod = sys.modules[__name__]
    global __all__

    for key, value in prefs.items():
        setattr(mod, key, value)
        __all__.append(key)
    return prefs

read_prefs(PLIST_PATH)

Now if you are the kind of Mac guy who enjoys using defaults you can write out your site-specific settings from the command-line.

defaults write /Library/Preferences/com.example EMAIL_HOST smtp.example.com
plutil -convert xml1 /Library/Preferences/com.example.plist

N.B. Mac OS X 10.5 defaults uses the binary format by default, so you need plutil to convert it back to XML because plistlib does not handle the binary format.

For a simple site search feature I needed to search for a term across several fields in a model. Suppose the model looks like this:

class BlogPost(models.Model):
    title = models.CharField(max_length=100)
    body = models.TextField()
    summary = models.TextField()

And you have a view method that accepts a parameter q for searching across the title, body and summary fields. I want to find objects that contain the q phrase in any of those fields. I need to build a QuerySet with a filter that is the equivalent of

queryset = BlogPost.objects.filter(
    Q(title__icontains=q) | Q(body__icontains=q) | Q(summary__icontains=q)
)

That’s not too much of a hassle for this simple example, but in cases where the fields you are searching are chosen dynamically, or where you just have an awful lot of fields to search against, I think it is nicer to do it like so:

import operator

search_fields = ('title', 'body', 'summary')
q_objects = [Q(**{field + '__icontains':q}) for field in search_fields]
queryset = BlogPost.objects.filter(reduce(operator.or_, q_objects))

Nice one! The list comprehension gives me a list of Q objects generated from the names in search_fields, so it is easy to change the fields to be searched. And using reduce and operator.or_ gives me the required OR filter in one line.

I see for Python 3 reduce has been moved to the functools module.

This stuff never used to be that obvious to me. It kind of isn’t even now.

P.S. I promise I am not writing a blog engine at this time, it was just for the example.

It all started so innocently… I had wanted a set of urls for my application like this:

• http://example.com/a/ # List view of arrivals
• http://example.com/d/ # List view of departures
• http://example.com/a/feed/ # Syndication feed for arrivals
• http://example.com/d/feed/ # Syndication feed for departures

So I put the following in the application’s urls.py:

# myapp/urls.py
from django.conf.urls.defaults import *
from views import arrivals_list, departures_list
from feeds import LatestArrivals, LatestDepartures


feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}


urlpatterns = patterns('',
    (r'^a/$', arrivals_list, {}, 'arrivals'),
    (r'^d/$', departures_list, {}, 'departures'),
    (r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)

That covers my URL wishes, and because I have named the URL patterns I can use that name in templates with the {% url %} template tag and in Python code using the reverse helper.

So naturally the feed classes in feeds.py look like this:

# myapp/feeds.py
from django.contrib.syndication.feeds import Feed
from django.core.urlresolvers import reverse
from django.utils.feedgenerator import Atom1Feed
from models import Tx


class LatestArrivals(Feed):
    """Produces an Atom feed of recent arrival tickets."""
    feed_type = Atom1Feed
    title = 'Arrivals'
    link = reverse('arrivals')
    subtitle = 'Most recent arrivals'

    def items(self):
        return Tx.objects.arrivals()[:10]


class LatestDepartures(Feed):
    """Produces an Atom feed of recent departure tickets."""
    feed_type = Atom1Feed
    title = 'Departures'
    link = reverse('departures')
    subtitle = 'Most recent departures'

    def items(self):
        return Tx.objects.departures()[:10]

Note I used reverse on the link attribute of each class so that I can define the URL in one place, the urls.py module, and a change there will be reflected in the feed’s link too.

But this doesn’t work! When Django imports my urls.py module, it imports LatestDepartures and LatestArrivals, and they in turn use reverse to find the named URL patterns – except those names aren’t defined until after urlpatterns has been defined in urls.py so Django throws an exception and never imports my urls.py module.

You could work around this either by defining your syndication feeds in an entirely different urls.py module. But you can also split up urlpatterns within the same module and import the feed classes after their named URL patterns have been defined.

Here’s the working urls.py module:

from django.conf.urls.defaults import *
from views import arrivals_list, departures_list


urlpatterns = patterns('',
    (r'^a/$', arrivals_list, {}, 'arrivals'),
    (r'^d/$', departures_list, {}, 'departures'),
)


from feeds import LatestArrivals, LatestDepartures
feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}


urlpatterns += patterns('',
    (r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)

from django import forms


class LetterChoices(object):
    """Return a random list of max_choices letters of the alphabet."""
    def __init__(self, max_choices=3):
        self.max_choices = max_choices

    def __iter__(self):
        import string, random

        return iter((l, l) for l in random.sample(string.ascii_uppercase, self.max_choices))


class LetterForm(forms.Form):
    """Pick a letter from a small, random set."""
    letter = forms.ChoiceField(choices=LetterChoices())

I don’t know if I prefer that style to having a simple function – having to instantiate the class seems wrong to me, I’d much rather use any callable as the choices argument.

from contextlib import contextmanager


@contextmanager
def test_db_connection():
    """A context manager for Django's test runner.

    For Python 2.5 you will need
        from __future__ import with_statement
    """

    from django.conf import settings
    from django.test.utils import setup_test_environment, teardown_test_environment
    from django.db import connection

    setup_test_environment()

    settings.DEBUG = False    
    verbosity = 0
    interactive = False

    old_name = settings.DATABASE_NAME
    connection.creation.create_test_db(verbosity, autoclobber=not interactive)

    yield connection

    connection.creation.destroy_test_db(old_name, verbosity)
    teardown_test_environment()

All of this requires Python 2.5 or later.

So with that snippet you could write a test something like so:

import unittest


class MyTestCase(unittest.TestCase):
    def test_myModelTest(self):
        with test_db_connection():
            from myproject.myapp.models import MyModel

            obj = MyModel()
            obj.save()
            self.assert_(obj.pk)

… and just as with Django’s manage.py test command the objects would be created within the test database then destroyed when the with test_db_connection() block is finished.

Everything’s going to be hunky dory.

As of Django 1.0.2 the default behaviour for the test runner is the run_tests function in django.test.simple. Here is the bones of that function with the required setup and teardown calls.

from django.conf import settings
from django.test.utils import setup_test_environment, teardown_test_environment


verbosity = 1
interactive = True

setup_test_environment()
settings.DEBUG = False    
old_name = settings.DATABASE_NAME

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)

# Here you run tests using the test database and with mock SMTP objects

connection.creation.destroy_test_db(old_name, verbosity)
teardown_test_environment()

Hmmm… Wouldn’t this be a good candidate to be wrapped up for use with Python 2.5’s with statement?

Fortunately this was covered long ago by James Bennett when Django’s newforms module was introduced. See Getting dynamic model choices in newforms on Django Snippets.

The following form example has a field for picking a letter of the alphabet (works for Django 1.0). The choices are limited to 3 letters only, picked at random and different for each form instance:

from django import forms


def letter_choices(max_choices=3):
    """Return a random list of max_choices letters of the alphabet."""
    import string, random

    for l in random.sample(string.ascii_uppercase, max_choices):
        yield (l, l)


class LetterForm(forms.Form):
    """Pick a letter from a small, random set."""
    letter = forms.ChoiceField(choices=letter_choices())

    def __init__(self, *args, **kwargs):
        super(LetterForm, self).__init__(*args, **kwargs)
        self.fields['letter'].choices = letter_choices()

So that works. The LetterForm class uses the helper function letter_choices to provide the random choices, which actually returns a generator object rather than a list or tuple of choice pairs. I am relying on Django’s base ChoiceField class calling list() on the choices when each form is instantiated, so having letter_choices return a generator is hunky dory.

>>> f1 = LetterForm()
>>> f1['letter']
>>> print f1['letter']
<select name="letter" id="id_letter">
<option value="R">R</option>
<option value="N">N</option>
<option value="U">U</option>
</select>
>>> f2 = LetterForm()
>>> print f2['letter']
<select name="letter" id="id_letter">
<option value="O">O</option>
<option value="N">N</option>
<option value="T">T</option>
</select>
>>> 

Now the only thing is… this example is not practical. Using genuinely random choices means that the valid choices on the form submitted by the user will be different to the valid choices on the form used to validate the user input on the next request, and this will likely raise a ValidationError.

Django feature suggestion: allow choices to be any iterable or callable, calling it as appropriate when instantiating the field. If it is callable you could pass a function which returns an iterable at that point, which would save one having to write an __init__ method for the form sub-class.

I have a simple application to store serial numbers and purchasing information for software. It has a License model, but I wanted to add a comment on each license. Following the instructions for converting an existing app on the South wiki, here’s what I did to convert my software license application.

First I added south to INSTALLED_APPS in settings.py and ran manage.py syncdb to install the South application tables for the first time. This was before using South to actually manage any of the evolutions.

% ./manage.py syncdb
Syncing...
Creating table south_migrationhistory

Synced:
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.admin
 > myproject.software
 > south

Not synced (use migrations):
 - 
(use ./manage.py migrate to migrate these)

Then I created an initial migration for my application. This gives me a migration that matches the existing database schema.

% ./manage.py startmigration software --initial
Creating migrations directory at '/Users/david/myproject/../myproject/software/migrations'...
Creating __init__.py in '/Users/david/myproject/../myproject/software/migrations'...
Created 0001_initial.py.

The next step was to bring this application under South’s control. You have to pretend to apply the initial migration (because it already exists in the database), which is done using South’s --fake switch. I discovered a problem with my use of _ in my 0001_initial.py migration: the easiest fix was to import it explicitly within the migration (this fix was necessary for the second migration too).

from south.db import db
from django.db import models
from myproject.software.models import *
from django.utils.translation import gettext_lazy as _

Then I was ready to bring my application under South’s control.

% ./manage.py migrate software 0001 --fake
 - Soft matched migration 0001 to 0001_initial.
Running migrations for software:
 - Migrating forwards to 0001_initial.
 > software: 0001_initial
   (faked)

Now I was ready to alter my model definitions and get South to do the tedious work of updating the database schema. I added a new Note model in the application’s models.py and added a new field to the existing License model.

class License(models.Model):
    """A license for a software title."""
    ...
    department = models.CharField(_("Department"), max_length=100, blank=True)
    ...


class Note(models.Model):
    """A note for a license."""
    license = models.ForeignKey(License, editable=False)
    author = models.CharField(_("author"), max_length=100, editable=False)
    created = models.DateTimeField(auto_now_add=True)
    note = models.TextField(_("note text"))

    def __unicode__(self):
        return self.note

    class Meta:
        ordering = ['-created']

With those changes I used the startmigration command to generate a migration for the new model and fields.

% ./manage.py startmigration software notes_department --model Note --add-field License.department
Created 0002_notes_department.py.

(Afterwards I edited the migration to import _ as noted above.)

The final step was to use South to apply this migration (and any others) to the database.

% ./manage.py migrate software 
Running migrations for software:
 - Migrating forwards to 0002_notes_department.
 > software: 0002_notes_department
   = ALTER TABLE "software_license" ADD COLUMN "department" varchar(100) NOT NULL; []
   = CREATE TABLE "software_note" ("id" serial NOT NULL PRIMARY KEY, "license_id" integer NOT NULL, "author" varchar(100) NOT NULL, "created" timestamp with time zone NOT NULL, "note" text NOT NULL); []
   = ALTER TABLE "software_note" ADD CONSTRAINT "license_id_refs_id_61c4291d" FOREIGN KEY ("license_id") REFERENCES "software_license" ("id") DEFERRABLE INITIALLY DEFERRED; []
   = CREATE INDEX "software_note_license_id" ON "software_note" ("license_id"); []
 - Sending post_syncdb signal for software: ['Note']
 - Loading initial data for software.

Sweet!

But there is no documented method to specify one form class for creating an object and a different form class for editing an object.

My solution is to override the get_form method on your ModelAdmin sub-class. When adding an object, ModelAdmin will use the add_view method, whereas when changing an existing object ModelAdmin will use the change_view method. You could override the add_view method and specify a different form class there, but you would need to handle the entire business of object creation, including checking the client has permission, validating the form instance and populating the template context. Django’s auth application takes that approach for adding a new user, providing a simple username / password form in place of the full-blown object form.

A simpler method is to override get_form. When changing an object, get_form is called with the optional argument obj set to the object instance. When adding an object the obj argument is None. This makes it easy to determine when to return a different form for adding versus changing.

Here is an example of how to sub-class ModelAdmin and override the get_form method to return a different form depending on whether the client is adding a new object or changing an existing object.

# admin.py
from django.contrib import admin
from forms import MyAddForm
from models import MyModel


class MyModelAdmin(admin.ModelAdmin):
    """A ModelAdmin that uses a different form class when adding an object."""
    def get_form(self, request, obj=None, **kwargs):
        if obj is None:
            return MyAddForm
        else:
            return super(MyModelAdmin, self).get_form(request, obj, **kwargs)


admin.site.register(MyModel, MyModelAdmin)

Note a certain amount of hand-waving in the example where I’ve assumed that the appropriate models and forms are defined elsewhere. The custom form class will need to be a sub-class of ModelForm (or at least implement a suitable save method).

Extensis Portfolio is digital asset management software that has been available on Macinctosh (and Windows) for donkeys years. The most expensive edition of Portfolio replaces the native data store with an SQL back-end (currently supported databases are MySQL, MSSQL and Oracle). Extensis also sells NetPublish, a web front-end to a Portfolio catalogue.

For various reasons I decided to write a custom web front-end for Portfolio using Django. As with Farmers Wife, the existing database schema was not well-suited to Django’s ORM. More than one table used a compound primary key which Django does not support (there is a page discussing how support would be implemented on the Django Wiki).

Defining models with SQLAlchemy

The meat of the application consists of two models:

• Items represent items in the catalogue. Each picture is an Item, and has properties for things like file type, file size, path to file on disk.
• Keywords represent keywords related to an Item, using a many-to-many relationship.

I defined two tables named item_table and keyword and a third that is used to hold the many-to-many relationship:

item_table = Table('item_table', metadata,
    Column('Record_ID', Integer(), primary_key=True, nullable=False, key="id"),
    Column(u'Filename', MSString(length=249), key="filename"),
    Column(u'Path', MSString(length=249), key="path"),
    Column(u'Created', MSDateTime(timezone=False)),
    Column(u'Last_Modified', MSDateTime(timezone=False)),
    Column(u'File_Size', MSInteger(length=11)),
    # Additional columns omitted here
)

keyword_table = Table('keyword', metadata,
    Column('Record_ID', Integer(), primary_key=True, nullable=False, key="id"),
    Column(u'Keyword', MSString(length=249), nullable=False, key="keyword"),
)

item_keyword_table = Table('Item_Keyword', metadata,
    Column(u'Item_ID', MSInteger(length=11), ForeignKey('item_table.id'), primary_key=True, nullable=False, default=PassiveDefault(u'0')),
    Column(u'Keyword_ID', MSInteger(length=11), ForeignKey('keyword.id'), primary_key=True, nullable=False, default=PassiveDefault(u'0')),
)

Of note here is how one can rename the table columns for use in the table definition (roughly equivalent to the db_column field option in Django). I prefer using the short name id for the primary key in place of the original designer’s choice of Record_ID.

Putting primary_key=True on more than one column for the item_keyword_table tells SQLAlchemy to use a compound primary key. And notice the ForeignKey relationship uses our customized column names for the related tables.

Then we need classes to associate the Item and Keyword objects with the underlying tables,

class Keyword(object): pass
class Item(object): pass

mapper(Keyword, keyword_table)
mapper(Item, item_table, properties=dict(
    keywords = relation(Keyword, secondary=item_keyword_table, backref="items", lazy=False, order_by=[keyword_table.c.keyword]),
))

The second statement above added a keywords property to an Item object to define the many-to-many relationship with a Keyword object. Specifying lazy=False causes the keywords related to an item to be fetched when the item itself is loaded – the default behaviour is to leave the related queries until the related object is accessed.

Making models more Django-friendly

The mapped classes for Keyword and Item behave in ways similar to a Django model. Any features that are missing can be easily put in place.

One can provide the pk shortcut for a model using a Python property:

class Keyword(object):
    # Extra pk property to access primary key
    def pk(self):
        return self.id
    pk = property(pk)

And get_absolute_url is just as straight-forward:

class Item(object):
    # Match a URL pattern like '^(?P<item_id>\d+)/$'
    def get_absolute_url(self):
        return reverse('catalogue_item', kwargs={'item_id':unicode(self.pk)})

These extra methods help SQLAlchemy model objects behave almost interchangeably with Django model objects in your template and view methods.

Using the Django paginator class with SQLAlchemy Query objects requires one to override a single method calculating the total number of hits:

from django.core.paginator import ObjectPaginator as DjangoPaginator

class ObjectPaginator(DjangoPaginator):
    """SQLAlchemy compatible flavour of Django's ObjectPaginator."""
    def _get_hits(self):
        if self._hits is None:
            self._hits = self.query_set.offset(0).limit(0).count()
        return self._hits

Using the models in views

Unfortunately there’s no simple means for using Django’s generic view methods (or the admin application) with SQLAlchemy’s query methods. Django’s views rely on accessing model._meta from the queryset and use an incompatible query syntax (albeit a syntax with similar intent).

But there’s nothing to stop you using SQLAlchemy query sets in custom views and in the template context.

The following view method shows the object detail page for an Item object; the primary key of the item is used in the URL and passed to the view as item_id:

# Session, Item are imported from a module with the model definitions
def item(request, item_id):
    """Detail page for a Portfolio item."""
    item_id = int(item_id)
    session = Session()
    item = session.query(Item).filter_by(id=item_id).one()
    return render_to_response("catalogue/item.html", {'object':item})

To complete the example one ought to handle the situation where there is not exactly one matching item. If SQLAlchemy throws sqlalchemy.exceptions.InvalidRequestError: Multiple rows returned for one() the view should in turn throw the corresponding Django exceptions so that the middleview machinery can operate as normal.

The future

Many parts of the API changed (or were deprected for compatibility) when SQLAlchemy progressed from 0.3 to the current 0.4 branch, and it is interesting how some of those changes seemed to be bringing SQLAlchemy’s ORM approach closer to Django’s ORM. I expect more similarities to appear on both sides in future releases.

The Django/SQLAlchemy branch has the explicit goal of combining the two without changing the Django model API, which would eliminate the need for any of the code in this post.

Until that gets merged to trunk, I hope you find this useful.