Previously on 24…
I posted a module for handling time zone-aware datetime objects, but I left out all the hassle of dealing with form input. Here is a more complete python package for Django that includes a form field sub-class that can handle a small set of datetime string formats that include a time zone offset.
This code is released under Django’s BSD license.
Nice and easy couple of Django template tags that filter lines of text using a regular expression. I had a block of text where I wanted to remove some of the lines but not others.
You can use grep to remove any lines that do not match your pattern:
>>> s = 'The quick brown fox'
>>> grep(s, 'quick')
u'The quick brown fox'
And its converse grepv to remove any lines that do match your pattern:
>>> s = 'The quick brown fox'
>>> grepv(s, 'quick')
''
>>> s2 = s + '\nJumps over the lazy dog'
>>> grepv(s2, 'quick')
u'Jumps over the lazy dog'
Stick it in a module in your Django application (documentation), then load it up at the top of a template.
from django import template
from django.template.defaultfilters import stringfilter
import re
register = template.Library()
@register.filter
@stringfilter
def grep(value, arg):
"""Lines that do not match the regular expression are removed."""
pattern = re.compile(arg)
lines = [line for line in re.split(r'[\r\n]', value) if pattern.search(line)]
return '\n'.join(lines)
@register.filter
@stringfilter
def grepv(value, arg):
"""Lines that match the regular expression are removed."""
pattern = re.compile(arg)
lines = [line for line in re.split(r'[\r\n]', value) if not pattern.search(line)]
return '\n'.join(lines)
Django makes it inordinately complicated to support time zone-aware dates and times because it has so far simply ignored the problem (so far being Django 1.0.2).
This is understandable given the database-agnostic nature of the Django ORM: although PostgreSQL 8.3 supports a datetime type which is time zone-aware, MySQL 5.1 does not (I have no idea what SQLite does about time zones). By ignoring time zones, Django works with the lowest common denominator.
Given time zone support in Postgres, there is a chunk of work to write a variation of models.DateTimeField which can handle time zone-wise datetimes. Python 2.5 does not help things – Python’s native datetime module is similarly agnostic about time zones, the standard library does not include a module for handling wise datetimes.
(If regular datetime instances are naive then datetime instances that honour time zones are wise.)
Django does make it pretty easy to write a custom field class, which means it shouldn’t be too difficult to write a custom datetime field class that is time zone-wise. As ever it is the Django project’s regard for documentation that transforms that which is possible into that which is practical.
Given your backend database has a time zone-wise datetime type (i.e. PostgreSQL), what input values does one need to handle in a time zone-wise custom field class?
Now the essence of a custom field in Django is two methods: to_python and get_db_prep_value. If the custom field defines
__metaclass__ = models.SubfieldBase
then the to_python method will be called any time a value is assigned to the field, and we can make sure that a suitable type is returned before the model object is saved. Because Postgres supports time zone-wise datetimes and if we take care to return a wise datetime instance we can ignore get_db_prep_value.
When Django reads a record from the database it strips the time zone information, effectively giving your custom field a naive datetime string that belongs to the same time zone as the database connection object. (At least this seems to be true for Postgres and the psycopg2 adaptor.) And since the database connection sets the time zone to be the same as set by settings.TIME_ZONE your custom class needs to treat any naive datetime strings as belonging to the time zone set with settings.TIME_ZONE.
So this leads to the important behaviour for a time zone-wise DateTimeField sub-class: always convert naive datetimes to the time zone set in settings.TIME_ZONE.
For convenience my custom field class, the TZDateTimeField, returns a sub-class of Python’s datetime which has an extra method that converts the datetime to the zone defined by the project’s time zone. Therefore whether the field value has been set from a naive or wise datetime instance, or a naive or wise date string you will end up with a time zone-wise value and you can get the value converted to the project’s time zone. This extra method is intended for use in a Django template.
What I was hoping was that the backend would store the datetime as a datetime in an arbitrary zone, potentially a different time zone from one record to the next for the same field. That behaviour would allow one to infer that one datetime value was created in this time zone while another datetime value was created in that time zone. Instead all datetime values are effectively normalized to your Django project’s time zone.
So here is an example of a model class that uses my time zone-aware datetime field. It ought to work just like a regular DateTimeField but always stores a time zone-aware datetime instance:
from django.db import models
from timezones.fields import TZDateTimeField
from datetime import datetime
class Article(models.Model):
pub_date = TZDateTimeField(default=datetime.now)
And below is my custom field definition, which has a dependency on the pytz module to handle all the difficult stuff. You can grab the complete module over here, including tests in doctest format. The tests are intended to be run by Django’s manage.py test management command, and so one needs to add the module to the list of installed apps.
"""A time zone-aware DateTime field.
When saving, naive datetime objects are assumed to belong to the local time
zone and are converted to UTC. When loading from the database the naive datetime
objects are converted to UTC.
These field types require database support. MySQL 5 will not work.
"""
from datetime import datetime, tzinfo, timedelta
from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import models
import pytz
import re
# 2009-06-04 12:00:00+01:00 or 2009-06-04 12:00:00 +0100
TZ_OFFSET = re.compile(r'^(.*?)\s?([-\+])(\d\d):?(\d\d)$')
class TZDatetime(datetime):
def aslocaltimezone(self):
"""Returns the datetime in the local time zone."""
tz = pytz.timezone(settings.TIME_ZONE)
return self.astimezone(tz)
class TZDateTimeField(models.DateTimeField):
"""A DateTimeField that treats naive datetimes as local time zone."""
__metaclass__ = models.SubfieldBase
def to_python(self, value):
"""Returns a time zone-aware datetime object.
A naive datetime is assigned the time zone from settings.TIME_ZONE.
This should be the same as the database session time zone.
A wise datetime is left as-is. A string with a time zone offset is
assigned to UTC.
"""
try:
value = super(TZDateTimeField, self).to_python(value)
except ValidationError:
match = TZ_OFFSET.search(value)
if match:
value, op, hours, minutes = match.groups()
value = super(TZDateTimeField, self).to_python(value)
value = value - timedelta(hours=int(op + hours), minutes=int(op + minutes))
value = value.replace(tzinfo=pytz.utc)
else:
raise
if value is None:
return value
# Only force zone if the datetime has no tzinfo
if (value.tzinfo is None) or (value.tzinfo.utcoffset(value) is None):
value = force_tz(value, settings.TIME_ZONE)
return TZDatetime(value.year, value.month, value.day, value.hour,
value.minute, value.second, value.microsecond, tzinfo=value.tzinfo)
def force_tz(obj, tz):
"""Converts a datetime to the given timezone.
The tz argument can be an instance of tzinfo or a string such as
'Europe/London' that will be passed to pytz.timezone. Naive datetimes are
forced to the timezone. Wise datetimes are converted.
"""
if not isinstance(tz, tzinfo):
tz = pytz.timezone(tz)
if (obj.tzinfo is None) or (obj.tzinfo.utcoffset(obj) is None):
return tz.localize(obj)
else:
return obj.astimezone(tz)
I have a Django project that I am going to deploy at several sites, but I need to tweak the project settings slightly for each site. Specifically I need different a EMAIL_HOST address and related settings for sending mail at each site.
The simplest route is to customize the project settings.py as part of the site deployment, but that will drive you insane when you deploy the wrong custom-settings to a site.
Another approach is similar to that used by many for switching between settings
when moving between testing / staging / live environments: your settings.py
has a few lines something like
try:
from sitesettings import *
except ImportError:
pass
so you can over-ride any setting by putting them in a sitesettings.py file,
and then make sure your deployment never overwrites that site-specific file.
In my case I want to make it easy for the site administrator to customize
the settings, but I am worried that it is too easy for someone who does not
know Python syntax to inadvertently break things by writing a sitesettings.py
that throws a SyntaxError exception. Given the significance of
white-space in Python I feel this would be easy to get wrong.
So I’ve gone for storing the custom settings in Mac OS X’s property list format. Bless Python for it has the plistlib module that reads and writes the simple XML format of property lists.
Here’s my module that imports all properties from a plist straight into the module’s namespace. This then makes it easy to over-ride Django’s settings by doing
from plistsettings import *
A couple bits made my lips move during the writing. The contents of __all__
are updated dynamically because I wanted to use this with
from plistingsettings import * without worrying that my module’s imports
would get clobbered by imports used in the plistsettings module. And working
out how to bind keys and values to the module itself is not obvious to me -
it feels like one ought to be able to use self within the scope of the
module to refer to the module itself. Except you can’t. No biggie.
# plistsettings.py
import os.path
import plistlib
import sys
from xml.parsers.expat import ExpatError
__all__ = []
PLIST_PATH = '/Library/Preferences/com.example.plist'
def read_prefs(plist_path):
"""Import settings from preference file into this module's global namespace.
Returns a dictionary as returned by plistlib.readPlist().
"""
try:
if os.path.exists(plist_path):
prefs = plistlib.readPlist(plist_path)
else:
return
except ExpatError:
return
mod = sys.modules[__name__]
global __all__
for key, value in prefs.items():
setattr(mod, key, value)
__all__.append(key)
return prefs
read_prefs(PLIST_PATH)
Now if you are the kind of Mac guy who enjoys using defaults you can write
out your site-specific settings from the command-line.
defaults write /Library/Preferences/com.example EMAIL_HOST smtp.example.com
plutil -convert xml1 /Library/Preferences/com.example.plist
N.B. Mac OS X 10.5 defaults uses the binary format by default, so you need
plutil to convert it back to XML because plistlib does not handle
the binary format.
I’ve finally settled on a nice syntax for OR-ing Django Q objects.
For a simple site search feature I needed to search for a term across several fields in a model. Suppose the model looks like this:
class BlogPost(models.Model):
title = models.CharField(max_length=100)
body = models.TextField()
summary = models.TextField()
And you have a view method that accepts a parameter q for searching across
the title, body and summary fields. I want to find objects that contain
the q phrase in any of those fields. I need to build a QuerySet
with a filter that is the equivalent of
queryset = BlogPost.objects.filter(
Q(title__icontains=q) | Q(body__icontains=q) | Q(summary__icontains=q)
)
That’s not too much of a hassle for this simple example, but in cases where the fields you are searching are chosen dynamically, or where you just have an awful lot of fields to search against, I think it is nicer to do it like so:
import operator
search_fields = ('title', 'body', 'summary')
q_objects = [Q(**{field + '__icontains':q}) for field in search_fields]
queryset = BlogPost.objects.filter(reduce(operator.or_, q_objects))
Nice one! The list comprehension gives me a list of Q objects generated from
the names in search_fields, so it is easy to change the fields to be searched.
And using reduce and operator.or_ gives me the
required OR filter in one line.
I see for Python 3 reduce has been moved to the functools module.
This stuff never used to be that obvious to me. It kind of isn’t even now.
P.S. I promise I am not writing a blog engine at this time, it was just for the example.
I wound up in a chicken and egg situation today using Django’s syndication
framework and the reverse helper. The problem was
that immediately after starting the development server, Django would throw a
NoReverseMatch exception on the first client visit, followed by AttributeError
on all subsequent visits.
It all started so innocently… I had wanted a set of urls for my application like this:
So I put the following in the application’s urls.py:
# myapp/urls.py
from django.conf.urls.defaults import *
from views import arrivals_list, departures_list
from feeds import LatestArrivals, LatestDepartures
feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}
urlpatterns = patterns('',
(r'^a/$', arrivals_list, {}, 'arrivals'),
(r'^d/$', departures_list, {}, 'departures'),
(r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)
That covers my URL wishes, and because I have named the URL patterns I
can use that name in templates with the {% url %} template tag
and in Python code using the reverse helper.
So naturally the feed classes in feeds.py look like this:
# myapp/feeds.py
from django.contrib.syndication.feeds import Feed
from django.core.urlresolvers import reverse
from django.utils.feedgenerator import Atom1Feed
from models import Tx
class LatestArrivals(Feed):
"""Produces an Atom feed of recent arrival tickets."""
feed_type = Atom1Feed
title = 'Arrivals'
link = reverse('arrivals')
subtitle = 'Most recent arrivals'
def items(self):
return Tx.objects.arrivals()[:10]
class LatestDepartures(Feed):
"""Produces an Atom feed of recent departure tickets."""
feed_type = Atom1Feed
title = 'Departures'
link = reverse('departures')
subtitle = 'Most recent departures'
def items(self):
return Tx.objects.departures()[:10]
Note I used reverse on the link attribute of each class so that I can
define the URL in one place, the urls.py module, and a change there will
be reflected in the feed’s link too.
But this doesn’t work! When Django imports my urls.py module, it imports
LatestDepartures and LatestArrivals, and they in turn use reverse to
find the named URL patterns – except those names aren’t defined until after
urlpatterns has been defined in urls.py so Django throws an exception
and never imports my urls.py module.
You could work around this either by defining your syndication feeds in an
entirely different urls.py module. But you can also split up urlpatterns
within the same module and import the feed classes after their named URL
patterns have been defined.
Here’s the working urls.py module:
from django.conf.urls.defaults import *
from views import arrivals_list, departures_list
urlpatterns = patterns('',
(r'^a/$', arrivals_list, {}, 'arrivals'),
(r'^d/$', departures_list, {}, 'departures'),
)
from feeds import LatestArrivals, LatestDepartures
feed_dict = {'a': LatestArrivals, 'd': LatestDepartures}
urlpatterns += patterns('',
(r'^(?P<url>[ad])/feed/$', 'django.contrib.syndication.views.feed', {'feed_dict':feed_dict}),
)
I had another thought about per-instance choices for forms.ChoiceField.
Instead of overriding the __init__ method of your form class, you could use
an object with an __iter__ method that returns a fresh iterable each time
it is called.
from django import forms
class LetterChoices(object):
"""Return a random list of max_choices letters of the alphabet."""
def __init__(self, max_choices=3):
self.max_choices = max_choices
def __iter__(self):
import string, random
return iter((l, l) for l in random.sample(string.ascii_uppercase, self.max_choices))
class LetterForm(forms.Form):
"""Pick a letter from a small, random set."""
letter = forms.ChoiceField(choices=LetterChoices())
I don’t know if I prefer that style to having a simple function – having to
instantiate the class seems wrong to me, I’d much rather use any callable as
the choices argument.
In my last post I mentioned it might be an idea to wrap up the Django test
database setup / teardown in a context manager for use with Python’s
with statement. Here’s my first stab, which seems to work.
from contextlib import contextmanager
@contextmanager
def test_db_connection():
"""A context manager for Django's test runner.
For Python 2.5 you will need
from __future__ import with_statement
"""
from django.conf import settings
from django.test.utils import setup_test_environment, teardown_test_environment
from django.db import connection
setup_test_environment()
settings.DEBUG = False
verbosity = 0
interactive = False
old_name = settings.DATABASE_NAME
connection.creation.create_test_db(verbosity, autoclobber=not interactive)
yield connection
connection.creation.destroy_test_db(old_name, verbosity)
teardown_test_environment()
All of this requires Python 2.5 or later.
So with that snippet you could write a test something like so:
import unittest
class MyTestCase(unittest.TestCase):
def test_myModelTest(self):
with test_db_connection():
from myproject.myapp.models import MyModel
obj = MyModel()
obj.save()
self.assert_(obj.pk)
… and just as with Django’s manage.py test command the objects would be
created within the test database then destroyed when the
with test_db_connection() block is finished.
Everything’s going to be hunky dory.
I needed to run tests involving a Django application but without using the
manage.py test management command. So I need my own test suite that
sets up the test database and drops it after, leaving my real database untouched.
As of Django 1.0.2 the default behaviour for the test runner is the run_tests
function in django.test.simple. Here is the bones of that function
with the required setup and teardown calls.
from django.conf import settings
from django.test.utils import setup_test_environment, teardown_test_environment
verbosity = 1
interactive = True
setup_test_environment()
settings.DEBUG = False
old_name = settings.DATABASE_NAME
from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)
# Here you run tests using the test database and with mock SMTP objects
connection.creation.destroy_test_db(old_name, verbosity)
teardown_test_environment()
Hmmm… Wouldn’t this be a good candidate to be wrapped up for use with
Python 2.5′s with statement?
I keep forgetting the details of how one customizes the choices in a forms.ChoiceField per instance. The forms.ChoiceField documentation says the required argument has to be an iterable but then moves straight to the next section.
Fortunately this was covered long ago by James Bennett when Django’s newforms module was introduced. See Getting dynamic model choices in newforms on Django Snippets.
The following form example has a field for picking a letter of the alphabet (works for Django 1.0). The choices are limited to 3 letters only, picked at random and different for each form instance:
from django import forms
def letter_choices(max_choices=3):
"""Return a random list of max_choices letters of the alphabet."""
import string, random
for l in random.sample(string.ascii_uppercase, max_choices):
yield (l, l)
class LetterForm(forms.Form):
"""Pick a letter from a small, random set."""
letter = forms.ChoiceField(choices=letter_choices())
def __init__(self, *args, **kwargs):
super(LetterForm, self).__init__(*args, **kwargs)
self.fields['letter'].choices = letter_choices()
So that works. The LetterForm class uses the helper function letter_choices to provide the random choices, which actually returns a generator object rather than a list or tuple of choice pairs. I am relying on Django’s base ChoiceField class calling list() on the choices when each form is instantiated, so having letter_choices return a generator is hunky dory.
>>> f1 = LetterForm()
>>> f1['letter']
>>> print f1['letter']
<select name="letter" id="id_letter">
<option value="R">R</option>
<option value="N">N</option>
<option value="U">U</option>
</select>
>>> f2 = LetterForm()
>>> print f2['letter']
<select name="letter" id="id_letter">
<option value="O">O</option>
<option value="N">N</option>
<option value="T">T</option>
</select>
>>>
Now the only thing is… this example is not practical. Using genuinely random choices means that the valid choices on the form submitted by the user will be different to the valid choices on the form used to validate the user input on the next request, and this will likely raise a ValidationError.
Django feature suggestion: allow choices to be any iterable or callable, calling it as appropriate when instantiating the field. If it is callable you could pass a function which returns an iterable at that point, which would save one having to write an __init__ method for the form sub-class.