Monthly Archives: June 2009

Casio versus Canon versus Fuji

I bought me a [Casio Exilim EX-Z29][casio] (lipgloss pink) to replace a four-year-old [Canon Digital Ixus 50][canon]. The Canon in turn replaced a [Fuji FinePix F40][fuji] or something.

That old Fuji was a fantastic little camera, given me by my brother and sister about eight years ago. Its great strength was getting good shots in all kinds of low-light party conditions, often without a flash, but it eventually limped along and would make a little scrunch noise when the lens cover retracted. Time for a new Canon!

The Canon was a good camera, way over my intended budget (it cost about £220 new) but I reckoned it would be worth it. However I was always disappointed with it because it couldn’t handle low-light situations without the power of a thousand suns from the flash, and I kind of like to take pics at parties without a flash going off to announce I am taking a pic. And often you want a picture with natural light; maybe it is indoors and there’s already a lot of ambient light or something, but the flipping Canon would always insist on using the flash as soon as it determined the sun had passed midday. If I disabled the flash I would usually end up with severely out-of-focus or blurred pics. Not a Fuji!

So eventually the lens on the Canon got scuffed in the bottom left corner, leaving my pics with an irritating blurry halo like I hadn’t quite finished applying the vaseline. Time for a new camera!

The Casio was on offer on Amazon for £105 (it was another £30 if you didn’t want lipgloss pink – I fancied the idea of a pink camera) and it seems good. Took a bunch of pics which have all come out well, including some where I deliberately disabled the flash. I am happier with the Casio for point-and-click duty than I ever was with the Canon.

The difference between a relatively expensive Canon and a cheap-ish Casio is automatic picture orientation. The Canon would tag portrait shots as revolved, the little Casio does not. Minor bummer.

The lipgloss pink is not too blatant, I am pleased I went for it.

[casio]: http://exilim.casio.com/products_exz29.shtml
[canon]: http://www.canon.co.uk/for_home/product_finder/cameras/digital_camera/ixus/Digital_IXUS_50/index.asp
[fuji]: http://www.fujifilm.co.uk/consumer/digital/

Tuples to dicts, toot sweet!

Looking through [Trac][trac]’s search internals I came across [a chunk where a list of tuples is converted to a list of dictionaries][searchmodule] for the convenience of the template engine. Each tuple has five fields: *href*, *title*, *date*, *author* and *excerpt*.

for idx, result in enumerate(results):
results[idx] = {‘href’: result[0], ‘title’: result[1],
‘date’: format_datetime(result[2]),
‘author’: result[3], ‘excerpt’: result[4]}

This allows the template author to use nice names for the fields in a row, like `${result.href}` etc. Looking at this reminded me of another approach that uses [list comprehension][list], [`zip`][zip] and [`dict`][dict].

keys = (‘href’, ‘title’, ‘date’, ‘author’, ‘excerpt’)
results = [dict(zip(keys, row)) for row in results]
for row in results:
row[‘date’] = format_datetime(row[‘date’])

The second line in this snippet is where the list of dictionaries is created, but one still has to go back and format the datetime values (the third and fourth lines). There’s no advantage in speed (the majority of the execution time is spent in `format_datetime`) but I like it a little better.

Maybe if Trac used the second approach I would like Trac a little better too.

[trac]: http://trac.edgewall.org
[searchmodule]: http://trac.edgewall.org/browser/tags/trac-0.11.4/trac/search/web_ui.py#L111
[list]: http://docs.python.org/tutorial/datastructures.html#list-comprehensions
[zip]: http://docs.python.org/library/functions.html#zip
[dict]: http://docs.python.org/library/stdtypes.html#typesmapping

Installers: correlate of hate

The degree of hatefulness for a Mac installer often correlates to the degree of hatefulness for the software it installs. [Adobe Creative Suite][adobecs] is this rule’s exemplar. And if you have a [Blackberry][blackberry] mobile phone you may have had fun trying to get the [PocketMac for Blackberry software][pocketmac] working.

The PocketMac installation experience is pretty poor. Not surprisingly it uses [Mindvision’s Installer Vise][installervise]. But the developers have really pushed the boat out and gone to the trouble of using an Apple installer package icon for the install application. Do not be deceived! It is an application, not a package.

The PocketMac installer

The PocketMac installer

And if you succeed in installing the sync application, don’t expect it to work unless you happen to have learnt that you should only connect the Blackberry to the USB cable *after* you have opened the sync application.

[adobecs]: http://www.adobe.com/products/creativesuite/
[blackberry]: http://www.blackberry.com/
[pocketmac]: http://na.blackberry.com/eng/services/desktop/mac.jsp
[installervise]: http://www.mindvision.com/macvise.asp

Fiery RIPs, old-fashioned installation hate

Whether you buy from Canon or Xerox (or Epson or Konica Minolta, etc.), if you are buying a printer for heavy-duty use in a Mac design studio then it will probably have a Fiery RIP[^1] manufactured by [Electronics For Imaging][efi].

There are lots of things to hate about Fiery RIPs.

But right now I am going to hate their backward attitude to Mac driver installation. It is the year 2009 and this lot have yet to supply drivers as anything other than [Installer Vise][vise] applications. So that means no simple command-line roll-outs with [Apple Remote Desktop][ard]. That means no way of looking at the installation manifest ahead of installation. No way of having any confidence that the installer won’t screw everything up.

To compound their sins, EFI like to give their installer applications a very pretty icon that looks just like a stylized Apple installer package. It is a very pretty icon of a box; a box full of hate.

Purple box of Installer Vise hate

Purple box of Installer Vise hate

[efi]: http://www.efi.com/
[vise]: http://www.mindvision.com/macvise.asp
[ard]: http://www.apple.com/remotedesktop/

[^1]: RIP stands for Raster Image Processor. Which leads to quite enjoyable conversations about ripping through pages.

os.walk and UnicodeDecodeError

My Python program was raising a `UnicodeDecodeError` when using [`os.walk`][oswalk] to traverse a directory containing files and folders with UTF-8 encoded names. What had me baffled was the exact same program was working perfectly on the exact same hardware just minutes earlier.

Turns out the difference was between me starting my program as root from an interactive bash shell, versus the program getting started as part of the boot sequence by init (on a [Debian Lenny][lenny] system). When started interactively, the locale was set to en_GB.UTF-8 and so names on the filesystem were assumed to be UTF-8 encoded. When started by init, the locale was set to ASCII.

The fix, as described in this article [*Python: how is sys.stdout.encoding chosen?*][codemonk], was to wrap my program in a script that set the LC_CTYPE environment variable.

#!/bin/sh
export LC_CTYPE=’en_GB.UTF-8′
/path/to/program.py

[codemonk]: http://drj11.wordpress.com/2007/05/14/python-how-is-sysstdoutencoding-chosen/
[lenny]: http://www.debian.org/
[oswalk]: http://docs.python.org/library/os.html#os.walk

Django and time zone-aware date fields

[Django][django] makes it inordinately complicated to support time zone-aware dates and times because it has so far simply ignored the problem (so far being [Django 1.0.2][django102]).

This is understandable given the database-agnostic nature of the Django ORM: although [PostgreSQL 8.3][postgres83] supports a datetime type which is time zone-aware, [MySQL 5.1 does not][mysql51] (I have no idea what [SQLite][sqlite] does about time zones). By ignoring time zones, Django works with the lowest common denominator.

Given time zone support in Postgres, there is a chunk of work to write a variation of [`models.DateTimeField`][datetimefield] which can handle time zone-wise datetimes. Python 2.5 does not help things – [Python’s native datetime module][datetime] is similarly agnostic about time zones, the standard library does not include a module for handling wise datetimes.

(If regular datetime instances are *naive* then datetime instances that honour time zones are *wise*.)

Django does make it pretty easy to [write a custom field class][customfields], which means it shouldn’t be too difficult to write a custom datetime field class that is time zone-wise. As ever it is the Django project’s regard for documentation that transforms *that which is possible* into *that which is practical*.

Given your backend database has a time zone-wise datetime type (i.e. PostgreSQL), what input values does one need to handle in a time zone-wise custom field class?

* value set to None
* value set to a naive datetime instance
* value set to a wise datetime instance
* value set to a naive datetime string
* value set to a wise datetime string

Now the essence of a custom field in Django is two methods: `to_python` and `get_db_prep_value`. If the custom field defines

__metaclass__ = models.SubfieldBase

then the `to_python` method will be called any time a value is assigned to the field, and we can make sure that a suitable type is returned before the model object is saved. Because Postgres [supports time zone-wise datetimes][postgresdt] and if we take care to return a wise datetime instance we can ignore `get_db_prep_value`.

When Django reads a record from the database it strips the time zone information, effectively giving your custom field a naive datetime string that belongs to the same time zone as the database connection object. (At least this seems to be true for Postgres and [the psycopg2 adaptor][psycopg2].) And since the database connection sets the time zone to be the same as set by [`settings.TIME_ZONE`][settingstz] your custom class needs to treat any naive datetime strings as belonging to the time zone set with `settings.TIME_ZONE`.

So this leads to the important behaviour for a time zone-wise `DateTimeField` sub-class: always convert naive datetimes to the time zone set in `settings.TIME_ZONE`.

For convenience my custom field class, the `TZDateTimeField`, returns a sub-class of Python’s `datetime` which has an extra method that converts the datetime to the zone defined by the project’s time zone. Therefore whether the field value has been set from a naive or wise datetime instance, or a naive or wise date string you will end up with a time zone-wise value and you can get the value converted to the project’s time zone. This extra method is intended for use in a Django template.

What I was hoping was that the backend would store the datetime as a datetime in an arbitrary zone, potentially a different time zone from one record to the next for the same field. That behaviour would allow one to infer that one datetime value was created in this time zone while another datetime value was created in that time zone. Instead all datetime values are effectively normalized to your Django project’s time zone.

So here is an example of a model class that uses my time zone-aware datetime field. It ought to work just like a regular `DateTimeField` but always stores a time zone-aware datetime instance:

from django.db import models
from timezones.fields import TZDateTimeField
from datetime import datetime

class Article(models.Model):
pub_date = TZDateTimeField(default=datetime.now)

And below is my custom field definition, which has a dependency on [the pytz module][pytz] to handle all the difficult stuff. [You can grab the complete module over here][timezones], including tests in [doctest format][doctest]. The tests are intended to be run by Django’s `manage.py` test management command, and so one needs to add the module to [the list of installed apps][installedapps].

“””A time zone-aware DateTime field.

When saving, naive datetime objects are assumed to belong to the local time
zone and are converted to UTC. When loading from the database the naive datetime
objects are converted to UTC.

These field types require database support. MySQL 5 will not work.
“””
from datetime import datetime, tzinfo, timedelta
from django.conf import settings
from django.core.exceptions import ValidationError
from django.db import models
import pytz
import re

# 2009-06-04 12:00:00+01:00 or 2009-06-04 12:00:00 +0100
TZ_OFFSET = re.compile(r’^(.*?)\s?([-\+])(\d\d):?(\d\d)$’)

class TZDatetime(datetime):
def aslocaltimezone(self):
“””Returns the datetime in the local time zone.”””
tz = pytz.timezone(settings.TIME_ZONE)
return self.astimezone(tz)

class TZDateTimeField(models.DateTimeField):
“””A DateTimeField that treats naive datetimes as local time zone.”””
__metaclass__ = models.SubfieldBase

def to_python(self, value):
“””Returns a time zone-aware datetime object.

A naive datetime is assigned the time zone from settings.TIME_ZONE.
This should be the same as the database session time zone.
A wise datetime is left as-is. A string with a time zone offset is
assigned to UTC.
“””
try:
value = super(TZDateTimeField, self).to_python(value)
except ValidationError:
match = TZ_OFFSET.search(value)
if match:
value, op, hours, minutes = match.groups()
value = super(TZDateTimeField, self).to_python(value)
value = value – timedelta(hours=int(op + hours), minutes=int(op + minutes))
value = value.replace(tzinfo=pytz.utc)
else:
raise

if value is None:
return value

# Only force zone if the datetime has no tzinfo
if (value.tzinfo is None) or (value.tzinfo.utcoffset(value) is None):
value = force_tz(value, settings.TIME_ZONE)
return TZDatetime(value.year, value.month, value.day, value.hour,
value.minute, value.second, value.microsecond, tzinfo=value.tzinfo)

def force_tz(obj, tz):
“””Converts a datetime to the given timezone.

The tz argument can be an instance of tzinfo or a string such as
‘Europe/London’ that will be passed to pytz.timezone. Naive datetimes are
forced to the timezone. Wise datetimes are converted.
“””
if not isinstance(tz, tzinfo):
tz = pytz.timezone(tz)

if (obj.tzinfo is None) or (obj.tzinfo.utcoffset(obj) is None):
return tz.localize(obj)
else:
return obj.astimezone(tz)

[django]: http://www.djangoproject.com/
[django102]: http://docs.djangoproject.com/en/dev/releases/1.0.2/
[postgres83]: http://www.postgresql.org/docs/8.3/
[mysql51]: http://dev.mysql.com/doc/refman/5.1/en/
[sqlite]: http://www.sqlite.org/
[datetimefield]: http://docs.djangoproject.com/en/dev/ref/models/fields/#datetimefield
[datetime]: http://docs.python.org/library/datetime.html
[customfields]: http://docs.djangoproject.com/en/dev/howto/custom-model-fields/
[postgresdt]: http://developer.postgresql.org/pgdocs/postgres/functions-datetime.html
[psycopg2]: http://initd.org/pub/software/psycopg/
[settingstz]: http://docs.djangoproject.com/en/dev/ref/settings/#time-zone
[pytz]: http://pytz.sourceforge.net/
[doctest]: http://docs.python.org/library/doctest.html
[timezones]: http://reliablybroken.com/b/wp-content/uploads/2009/06/timezones.zip
[installedapps]: http://docs.djangoproject.com/en/dev/ref/settings/#installed-apps