Let’s create a new Excel workbook and add a sheet:

>>> import xlwt
>>> book = xlwt.Workbook(encoding='utf-8')
>>> sheet = book.add_sheet('sheeeeeet')

We need to get a column in order to set its width. You do that by call col() on the sheet, passing the column’s index as the only argument (or row() for accessing rows):

>>> sheet.col(0)    # First column
<xlwt.Column.Column object at 0x10b2a6190>
>>> sheet.row(2)    # Third row
<xlwt.Row.Row object at 0x10b2a7050>

The index is zero-based. You can fetch a column even if you have not written to any cell in that column (this applies equally to rows).

Columns have a property for setting the width. The value is an integer specifying the size measured in 1/256 of the width of the character ’0′ as it appears in the sheet’s default font. xlwt creates columns with a default width of 2962, roughly equivalent to 11 characters wide.

>>> first_col = sheet.col(0)
>>> first_col.width = 256 * 20              # 20 characters wide (-ish)
>>> first_col.width
5120

For rows, the height is determined by the style applied to the row or any cell in the row. (In fact rows also have a property called height but it doesn’t do what you want.) To set the height of the row itself, create a new style with a font height:

>>> tall_style = xlwt.easyxf('font:height 720;') # 36pt
>>> first_row = sheet.row(0)
>>> first_row.set_style(tall_style)

Setting the style on the row does not change the style of the cells in that row.

There is no obvious way to set a default width and height for all columns and rows. An instance of xlwt.Worksheet.Worksheet has properties for col_default_width and row_default_height but changing those does not actually change the defaults.

The problem is that new columns are always created with an explicit width, while rows take their height from the style information.

My first attempt at setting defaults set the width on every column and the height on every row. It works, but creates 65,536 unnecessary empty row objects.

A slightly better approach is to set the width on every column and to set the font height on the default style record in the workbook:

import itertools
import xlwt

book = xlwt.Workbook(encoding='utf-8')
sheet = book.add_sheet('sheeeeeet')

col_width = 256 * 20                        # 20 characters wide

try:
    for i in itertools.count():
        sheet.col(i).width = col_width
except ValueError:
    pass

default_book_style = book.default_style
default_book_style.font.height = 20 * 36    # 36pt

book.save('example.xls')

Here I used itertools.count() and wrapped the loop in a try block so I can forget exactly how many columns are permitted. When the loop tries to access a bad index it will throw ValueError and the loop will exit.

You mustn’t replace the default style on an instance of xlwt.Workbook.Workbook, you have to update the property of the existing style (to ensure you are changing the first style record). Unfortunately there is no way to set a default column width (as of xlwt version 0.7.2) so the brute force method of setting every column will have to do – it isn’t so bad since there are only 256 columns.

Talking of widths and heights, have you heard “Widths & Heights” by Magic Arm? Is good.

A few weeks ago I received an e-mail from Timothy Williams with changes to the my module so that it preserves the fractions of a second in the conversion. How sweet is that?!?!! Exclamation mark question mark exclamation mark cellida diaresis em-dash full stop king of punctuation.

It is fantastic that not only did someone find my code useful but also that they were generous enough to take the time to improve it and give the changes back to me. I love tasty, delicious free software and the people like Tim who make it tastier and more delicious.

So here is the new version of filetimes.py incorporating Tim’s fixes.

The XPath 1.0 specification says '.' is short-hand for 'self::node()', selecting a node itself.

Parsing an XML document and selecting the context node with ElementTree in Python 2.5:

>>> from xml.etree import ElementTree
>>> ElementTree.VERSION
'1.2.6'
>>> doc = "<Root><Example>BUG</Example></Root>"
>>> node1 = ElementTree.fromstring(doc).find('./Example')
>>> node1
<Element Example at 10e0ed8c0>
>>> node1.find('.')
<Element Example at 10e0ed8c0>
>>> node1.find('.') == node1
True

See how the result of node1.find('.') is the node itself? As it should be.

Parsing an XML document and selecting the context node with cElementTree in Python 2.5:

>>> from xml.etree import cElementTree
>>> doc = "<Root><Example>BUG</Example></Root>"
>>> node2 = cElementTree.fromstring(doc).find('./Example')
>>> node2
<Element 'Example' at 0x10e0e3660>
>>> node2.find('.')
>>> node2.find('.') == node2
False

Balls. The result of node2.find('.') is None.

However! I have a kludgey work-around that works whether you use ElementTree or cElementTree. Use './' instead of '.':

>>> node1.find('./')
<Element Example at 10e0ed8c0>
>>> node1.find('./') == node1
True
>>> node2.find('./')
<Element 'Example' at 0x10e0e3660>
>>> node2.find('./') == node2
True

Kludgey because './' is not a valid XPath expression.

So we are back on track. Also works for Python 2.6 which has the same version of ElementTree.

Fortunately Python 2.7 got a new version of ElementTree and the bug is fixed:

>>> from xml.etree import ElementTree
>>> ElementTree.VERSION
'1.3.0'
>>> doc = "<Root><Example>BUG</Example></Root>"
>>> node3 = ElementTree.fromstring(doc).find('./Example')
>>> node3
<Element 'Example' at 0x107257210>
>>> node3.find('.')
<Element 'Example' at 0x107257210>
>>> node3.find('.') == node3
True

However! They also fixed my kludgey work-around:

>>> node3.find('./')
>>> node3.find('./') == node3
False

So I can’t code something that works for all three versions. This is annoying. I was hoping to just replace ElementTree with the C version, makes my code run in one third the time (the XML parts of it run in one tenth the time). And cannot install any compiled modules – the code can only rely on Python 2.5′s standard library.

To apply formatting to a cell you pass an instance of the xlwt.XFStyle class as the fourth argument to the xlwt.Worksheet.write method. The best way to create an instance is to use the xlwt.easyxf helper, which takes a string that specifies the formatting for a cell.

The other thing about using styles is you should only make one instance of each, then pass that same style object every time you want to apply it to a cell.

An example which uses a few styles:

import xlwt

styles = dict(
    bold = 'font: bold 1',
    italic = 'font: italic 1',
    # Wrap text in the cell
    wrap_bold = 'font: bold 1; align: wrap 1;',
    # White text on a blue background
    reversed = 'pattern: pattern solid, fore_color blue; font: color white;',
    # Light orange checkered background
    light_orange_bg = 'pattern: pattern fine_dots, fore_color white, back_color orange;',
    # Heavy borders
    bordered = 'border: top thick, right thick, bottom thick, left thick;',
    # 16 pt red text
    big_red = 'font: height 320, color red;',
)

I have no idea what it is based on, but 20 = 1 pt. So 320 = 16 pt text.

book = xlwt.Workbook()
sheet = book.add_sheet('Style demo')

for idx, k in enumerate(sorted(styles)):
    style = xlwt.easyxf(styles[k])
    sheet.write(idx, 0, k)
    sheet.write(idx, 1, styles[k], style)

book.save('Example.xls')

It isn’t included with the current distribution on the cheese shop, but there is a useful Excel spreadsheet demonstrating cell patterns in the source repository.

You can find the complete list of possible cell formats by reading the source for xlwt.Styles.

(If you were mad keen on Django’s shift to class-based views you might reckon life wouldn’t be complete with a Bottle-based application until you employ classes for views. However Bottle’s use of decorators for tying URLs to views means it is less a natural fit than the same thing in Django.)

The problem is that you can’t just decorate the method in your class using bottle.route because if you use that decorator on a method in a class you are telling Bottle to use the method before it has been bound to an instance of that class.

So although I wish it did, the following example will not work:

import bottle

class ViewClass(object):
    @bottle.route("/")
    def home_view(self):
        return "My home page."

obj = ViewClass()
bottle.run()

Running that will lead to errors about not enough arguments passed to the view method of your ViewClass instance.

Instead you need to register the route right after the object is created. This can be done in the class’s __new__ method:

import bottle

class ViewClass(object):
    def __new__(cls, *args, **kwargs):
        obj = super(ViewClass, cls).__new__(cls, *args, **kwargs)
        bottle.route("/")(obj.home_view)
        return obj

    def home_view(self):
        return "My home page."

obj = ViewClass()
bottle.run()

It works. It isn’t that pretty. You could achieve exactly the same thing by explicitly passing the obj.home_view method to bottle.route after the instance is created. The advantage to doing this in the __new__ method is it will happen automatically whenever ViewClass is instantiated.

And if you go down this path then you should be aware of threads. Hey! Nice threads! Also I have a cold.

Use MacPorts: relatively easy to install and the best thing is everything is contained in a directory that you can be confident won’t eff up Apple’s stuff and won’t be effed up by Apple’s stuff.

Install Xcode

You need the compiler and bits that are installed with Xcode. If you can’t find your Mac install discs (Xcode is included with every new Mac but not installed) you can download it from Apple’s developer website. Registration is required but is free.

The current version of Xcode is easy to find, while older versions are available in the downloads section under “Developer Tools”. Xcode version 3.1.4 is the last version that will work for Mac OS X 10.5 systems.

Install MacPorts

MacPorts have a nice pkg installer. You can also build it from source.

curl -O http://distfiles.macports.org/MacPorts/MacPorts-1.9.1-10.5-Leopard.dmg
hdiutil attach MacPorts-1.9.1-10.5-Leopard.dmg
sudo installer -pkg /Volumes/MacPorts-1.9.1/MacPorts-1.9.1.pkg -target /
hdiutil detach /Volumes/MacPorts-1.9.1

If for some reason MacPorts cannot fetch updates you may need to pull updates by hand.

Check your $PATH after installing ports to make sure /opt/local/bin is in there. If it isn’t your can do export PATH=/opt/local/bin:/opt/local/sbin:${PATH} to fix things, and even add taht line to ~/.profile so that bash picks it up every time (assuming you haven’t switched your shell).

Install software

The port command is used to manipulate the MacPorts installation. Use it to build and install the various bits we need. This takes a while, especially on old PowerPC machines. Make it more exciting by adding the --verbose flag. Exciting!

sudo port install python26
sudo port install apache2
sudo port install mysql5-server
sudo port install mod_python26
sudo port install py26-mysql
sudo port install py26-django
sudo port install py26-south

And if you want to hook Django into a network directory then you almost certainly want to use LDAP.

sudo port install py26-ldap

Cool kids these days say use mod_wsgi instead of mod_python for hosting Python applications with Apache, but I am not cool (and on 20 September 2010 I couldn’t persuade mod_wsgi to build from MacPorts on a clean installation).

Configuring and starting MySQL

UPDATED: commenter matea pointed to Jason Rowland’s MySQL on Mac posting that includes steps to secure a default installation, so I’ve updated this section with the appropriate steps.

I always seem to be the only person who cares about non-English visitors… anyway, so I want to have MySQL use UTF8 for everything. Edit the configuration so it does. As root, create a configuration at /opt/local/var/db/mysql5/my.cnf with these lines:

[mysqld]
init-connect = 'SET NAMES utf8'
character-set-server = utf8
collation-server = utf8_general_ci
skip-networking

[mysql]
default-character-set = utf8

One thing about the line skip-networking in the configuration file is that it means MySQL will not listen to any network clients, including connections to 127.0.0.1. Instead clients should connect to localhost or they should specify the path to the socket that MySQL uses for communication. If your MySQL “client” is a Django instance running on the same host then that should not be a problem.

Now initialize the database and start the server. (The use of -w in the second line tells launchctl to have the database daemon start at boot. If you don’t want to have MySQL running at boot use -F to force start just this one time instead of every time.)

sudo -u mysql /opt/local/bin/mysql_install_db5
sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist

And let’s check that the server is up and configured right.

/opt/local/bin/mysql5 -e "SHOW variables LIKE '%char%'"

You should see a table showing that the character set for the client and server is set to utf8.

Now run the secure installation script for MySQL. This will ask you to set a password for MySQL’s root account (the administrator) and ask whether to remove the test database and anonymous user access (you should do both):

/opt/local/bin/mysql_secure_installation5

Thaz better.

Configuring Postgresql instead of MySQL

If you want to use Postgres instead of MySQL then you need a couple different packages out of ports.

sudo port install postgresql84-server
sudo port install py26-psycopg2

Did you know Apple’s management tools use Postgres? Is true.

Configuring Apache to serve a Django project

Let’s suppose your Django project lives under /Library/WebServer/example.com/myproj, and the project’s settings file is /Library/WebServer/example.com/myproj/settings.py. Here’s how to configure Apache with mod_python to serve your project.

Create a separate site configuration for Apache in /Library/WebServer/example.com/site.conf.

<Location "/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE myproj.settings
    PythonOption django.root /
    PythonDebug On
    PythonPath "['/Library/WebServer/example.com'] + sys.path"
</Location>

<Directory /Library/WebServer/example.com>
    Order deny,allow
    Allow from all
</Directory>

Of course once everything is hunky dory you will go back and edit the site configuration so that PythonDebug Off.

And finally tell Apache to use mod_python and read the site configuration. Edit /opt/local/apache2/conf/httpd.conf and add a line at the end of the modules like:

LoadModule python_module modules/mod_python.so

And then a line like:

Include /Library/WebServer/example.com/site.conf

Now fire up Apache:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist

MacPorts has a convenient shortcut for this:

sudo port load apache2

You also want to save Apache a little grief by pre-compiling the Python source files for the project:

/opt/local/bin/python2.6 -m compileall /Library/WebServer/example.com

Hope this helps.

The @view decorator can also take keyword arguments. These are treated as default template variables – if the dictionary returned by your view function doesn’t have a key for one of the keyword arguments then the template will use the value passed into the decorator, like so:

from bottle import view

@view('default.html', author='David Buxton')
def home():
    return {'title': 'Home page'}

That would render any instance of '{{ author }}' as 'David Buxton'. And then you can have another view function that overrides the keywords by returning a different value in the dictionary:

from bottle import view

@view('default.html', author='David Buxton')
def music():
    return {'title': 'Thalassocracy', 'author': 'Frank Black'}

And at that point I wonder what is the advantage of using keyword arguments with @view: you have to decorate each function separately, and if you want to override a keyword in your return dictionary then it would be easier not to specify the keyword in the first place.

Thus the real point of using keywords with the @view function is only apparent if you curry the @view decorator with keywords first so that you can re-use the curried decorator and avoid repeating yourself.

Someday I will re-write the previous sentence. Until then, sorry.

Instead of passing a default author each time as in the examples above, let’s make a new @view decorator (using Python’s functools module) and then use that on each view function:

import functools
from bottle import view

view = functools.partial(view, author='David Buxton')

@view('default.html')
def home():
    return {'title': 'Home page'}

@view('default.html')
def music():
    return {'title': 'Thalassocracy', 'author': 'Frank Black'}

The new decorator means you get the default keyword arguments wherever you use @view while permitting any function to override those defaults in the dictionary it returns.

And if you wanted to get really lazy you could even pass in a template name when wrapping the decorator with functools.partial, however you would not be able to use your wrapped decorator to change the template name because it is a positional argument (like what it explains here in the functools documentation). You would also have to call the decorator with no arguments like '@defaultview()'. So forget I mentioned it.

I’m not saying you are lazy.

Updated: I have re-written this post and the example to make it simpler following Marcel Hellkamp’s comments (Marcel is the primary author of Bottle). My original example was needlessly complicated.

It is possible to have a Django-style urlpatterns stanza with a Bottle app. Here’s how it can work:

from bottle import route

# Assuming your *_page view functions are defined above somewhere
urlpatterns = (
    # (path, func, name)
    ('/', home_page, 'home'),
    ('/about', about_page, 'about'),
    ('/contact', contact_page, 'contact'),
)

for path, func, name in urlpatterns:
    route(path, name=name)(func)

Here we run through a list where each item is a triple of URL path, view function and a name for the route. For each we simply call the route method and then invoke it with the function object. Not as flexible as using the decorator on a function (because the @route decorator can take additional keyword arguments) but at least you can have all the routes in one place at the end of the module.

Then again if you have so many routes that you need to keep them in a pretty list you probably aren’t writing the simple application that Bottle was intended for.

(This was tested with Bottle’s 0.8 and 0.9-dev branches.)

The newly-decorated function can then do things like checking the called arguments before invoking the original un-decorated function.

Django provides decorators for authentication so that you can wrap a view function with a check for client credentials before deciding whether to return the original response or a deny access.

In this manner Django’s authentication decorators encourage orthogonal code: the logic for displaying a view is separated from the logic for deciding whether you should be permitted to see the view’s output. By keeping them separate, it becomes simpler to re-use the authentication logic and apply it to other views.

Suppose you have a view that accepts a Django request object and checks whether the user is signed in:

def administration_page(request):
    if request.user.is_authenticated():
        return HttpResponse("Welcome, dear user.")
    else:
        return HttpResponseRedirect("/signin/")

With a decorator you can simplify and clarify things:

@login_required
def administration_page(request):
    return HttpResponse("Welcome, dear user.")

For older versions of Python (pre 2.4) which don’t understand the @ operator one must explicitly decorate the view function like so:

def administration_page(request):
    return HttpResponse("Welcome, dear administrator.")

administration_page = login_required(administration_page)

Note in the example that the original administration_page function is passed to the decorator. The @ syntax in the first example makes that implicit but the two are equivalent.

The implementation of a decorator is interesting. It takes the function itself as an argument and returns a new function which does the actual checking. Here is how the decorator used above might do its stuff:

def login_required(view_function):
    def decorated_function(request):
        if request.user.is_authenticated():
            return view_function(request)
        else:
            return HttpResponseRedirect("/signin/")

    return decorated_function

The actual implementation of Django’s login_required decorator is considerably less idiotic. Python’s functools module has helpers for writing well-behaved decorators.

Because functions in Python are themselves objects the decorator can accept a function reference, construct a new function that checks for authentication and then return a reference to that new function.

Simples!

(Simples gets less simples when you want to write a decorator that accepts configuration arguments because you then need either another layer of nested function definitions or a class whose instances can be called directly, but I’m going to ignore you for a bit and wow is that Concorde…?)

My Python answer:

def readlines(filename, endings, chunksize=4096):
    """Returns a generator that splits on lines in a file with the given
    line-ending.
    """
    line = ''
    while True:        
        buf = filename.read(chunksize)
        if not buf:
            yield line
            break

        line = line + buf

        while endings in line:
            idx = line.index(endings) + len(endings)
            yield line[:idx]
            line = line[idx:]

if __name__ == "__main__":
    import sys, os

    FORMFEED = chr(12) # ASCII 12
    basename = os.path.basename(sys.argv[1])
    for num, data in enumerate(readlines(open(sys.argv[1]), endings=FORMFEED)):
        filename = basename + '-' + str(num)
        open(filename, 'wb').write(data)

This is also useful when reading data exported from some old-fashioned Mac application like Filemaker 5 where the line-endings are ASCII 13 not ASCII 10.

This post was inspired by Lotus Notes version 8.5, which is so advanced that to save a message in a file on disk you have to export it as structured text. And if you want to save a whole bunch of messages as individual files you must forget that drag-and-drop was introduced with System 7, that would be too obvious.

Django uses an instance of ModelAdmin (defined in django.contrib.admin.options) to handle the request for a model object add / change view in the admin site. ModelAdmin.add_view and ModelAdmin.change_view are responsible for populating the template context when rendering the add object and change object pages respectively.

Here are the keys common to add and change views:

• title, ‘Add ‘ or ‘Change ‘ + your model class’ _meta.verbose_name
• adminform is an instance of AdminForm
• is_popup, a boolean which is true when _popup is passed as a request parameter
• media is an instance of django.forms.Media
• inline_admin_formsets is a list of InlineAdminFormSet objects
• errors is an instance of AdminErrorList
• root_path is the root_path attribute of the AdminSite object
• app_label is your model class’ _meta.app_label attribute

The way that Django renders a form in the admin view is to iterate over the adminform instance and then iterate over each FieldSet which in turn yield AdminField instances. All I want to do is layout the form fields, ignoring the fieldset groupings which may or may not be defined in the model’s ModelAdmin.fieldset attribute.

This turns out to be easy once you know how. The regular form is an attribute of the adminform object. So if your model has a field named “king_of_pop” you can refer to the form field in your template like so:

{{ adminform.form.king_of_pop.label_tag }}: {{ adminform.form.king_of_pop }}

Or if you want to save your finger tips you can use the with template tag:

{% with adminform.form as f %}
{{ f.king_of_pop.label_tag }}: {{ f.king_of_pop }}
{% endwith %}

Delving through the Django source while I tried to understand all of this I was struck by how Python defines hook functions for iteration and accessing attributes. Half of Python’s attraction is in how easy it is from the program author’s point of view to treat objects as built-in types like lists, dicts, etc.; the other half is the responsibility of the author of a Python module to encourage that same ease of use by implementing the related iteration protocols. It is harder to write a good Python module than it is to write a good Python program that uses a good module.

The next problem was that MacPorts couldn’t update because rsync was blocked by the corporate access policy. Fortunately plain HTTP is permitted outbound. Here’s how to use a local ports tree.

Install MacPorts using the disk image for 10.5.

curl -O http://distfiles.macports.org/MacPorts/MacPorts-1.8.2-10.5-Leopard.dmg
hdiutil attach MacPorts-1.8.2-10.5-Leopard.dmg
sudo installer -pkg /Volumes/MacPorts-1.8.2/MacPorts-1.8.2.pkg -target /
hdiutil detach /Volumes/MacPorts-1.8.2

If the MacPorts install directories are not in your $PATH environment, you can add them to your .profile. This change will not take effect until you start a new terminal session.

cat >> ~/.profile <<EOF
PATH=/opt/local/bin:/opt/local/sbin:${PATH}
MANPATH=/opt/local/share/man:${MANPATH}
EOF

After you have installed MacPorts, create a directory for the ports tree and check it out using Subversion.

sudo mkdir -p /opt/local/var/macports/sources/svn.macports.org/trunk/dports
cd /opt/local/var/macports/sources/svn.macports.org/trunk/dports
sudo svn co http://svn.macports.org/repository/macports/trunk/dports/ .

N.B. In the last line beginning svn co ... the trailing directory separator is significant!

Now tell MacPorts to use the local checkout rather than rsync. Edit /opt/local/etc/macports/sources.conf and add a new line to the end with the path to the ports tree, then comment out the previous line that uses rsync. Here are the last lines from my configuration:

#rsync://rsync.macports.org/release/ports/ [default]
file:///opt/local/var/macports/sources/svn.macports.org/trunk/dports/ [default]

Finally you must create an index for the tree (otherwise you will see messages saying “Warning: No index(es) found!”).

cd /opt/local/var/macports/sources/svn.macports.org/trunk/dports
sudo portindex

Now go do great things.

A suitable ModelForm for your Django model will consume each row and do the conversion of each field to an appropriate Python type. Much simpler than explicitly converting each value yourself before creating a new model instance.

Suppose you have a model for an address book entry and its associated ModelForm (this works for Django 1.1):

# myapp/models.py
from django.db import models
from django import forms

class Contact(models.Model):
    first_name = models.CharField(max_length=100)
    second_name = models.CharField(max_length=100)
    telephone = models.CharField(max_length=50, blank=True)
    email = models.EmailField(blank=True)

class ContactForm(forms.ModelForm):
    class Meta:
        model = Contact

Here’s a script to run through a comma-separated list of contacts where each line looks something like “Smits, Jimmy, jimmy@example.com, 555-1234″:

from myapp.models import ContactForm

# Map columns to fields, adjusting the order as necessary
column_map = (
    'second_name',
    'first_name',
    'email',
    'telephone',
)

for line in open('tab-separated-data.txt'):
    row = dict(zip(column_map, (field.strip() for field in line.split(','))))
    form_obj = ContactForm(row)
    try:
        form_obj.save()
    except ValueError:
        for k, v in form_obj.errors.items():
            print k, row[k], ', '.join(map(unicode, v))

If a line doesn’t validate the script prints the validation errors and moves to the next line. If your data has columns you want to ignore then just name them in the column_map – the form class will ignore extra keys in the dictionary.

These are notes on how Radmind generates checksums for files on Mac OS X.

The fsdiff format is documented, however for files with Mac Finder info or a resource fork the checksum is for an AppleSingle-encoded representation of the file, which means a Python implementation needs to produce an equivalent AppleSingle-encoded byte stream for the file. Bummer.

Python 2.6 on Mac OS X includes a (deprecated) applesingle module that can read the format but cannot write it (and the module has been removed for Python 3). Therefore a pure Python implementation of Radmind’s checksum has to implement a compatible AppleSingle encoding routine too.

Radmind’s fsdiff command is written in C, which I can just about get the gist of, but I am missing something because my attempts at emulating Radmind’s checksums are wrong.

The meat of Radmind’s checksum is the do_acksum() function in cksum.c. The algorithm appears to be as follows:

1. Initialize a digest using the default cipher (MD5 I think).
2. Add the AppleSingle header, consisting of a magic number and version number and some padding.
3. Add the AppleSingle entry table, which has 3 entries for the Finder info, the resource fork info and the data fork info (in that order). Each entry is 12 bytes – an unsigned long for the entry type, an unsigned long for an offset into the file where the data will start and an unsigned long for the data length.
4. Add the Finder info data.
5. Add the resource for data.
6. Add the data fork data.
7. Return a base64 encoded version of the final digest.

Because the entry table in the AppleSingle header specifies data offsets and lengths you need to know the size of the Finder info data (always 32 bytes) and the size of the resource fork and the size of the data fork before you pass that data to the digest function.

So a working Python implementation needs to know the size of the resource fork and data fork before feeding that same data to the digest. It seems to me that this requirement might imply huge memory allocations while slurping file data – my wrong attempt tried counting bytes and later feeding the same data to the digest in manageable chunks.

Anyway…

Advice much appreciated. The workaround is to leave it to fsdiff to generate the checksum and parse the value from the output.

David

P.S. I still intend running A/UX 3.0.1 on my Centris 660av one day.

Update: using my eyes and brains and the fsdiff -V command I was able to read the fsdiff man page and deduce the preferred checksum cipher is actually sha1. My code is still wrong.

Previously the code was something like…

start(program1)
try:
    start(program2)
except:
    stop(program1)
    raise

try:
    start(program3)
except:
    stop(program2)
    stop(program1)
    raise

try:
    mainprogram()
finally:
    stop(program3)
    stop(program2)
    stop(program1)

Of course that could have been written with nested try / except / else / finally blocks as well, which I did start with but found not much shorter while almost incomprehensible.

With context managers the whole thing was written as…

# from __future__ import with_statement, Python 2.5

with start(program1):
    with start(program2):
        with start(program3):
            mainprogram()

So much more comprehensible! Here’s the implementation of the context manager (using the contextlib.contextmanager decorator for a triple word score):

import contextlib
import os
import signal
import subprocess


@contextlib.contextmanager
def start(program_args):
    prog = subprocess.Popen(program_args)
    if prog.poll(): # Anything other than None or 0 is BAD
        raise subprocess.CalledProcessError(prog.returncode, program_args[0])

    try:
        yield
    finally:
        if prog.poll() is None:
            os.kill(prog.pid, signal.SIGTERM)

For bonus points I might have used contexlib.nested() to put the three start() calls on one line but then what would I do for the rest of the day?

At least these days when I am writing some script (almost certainly in Python) I start out by intending to write tests. I usually fail because I haven’t learnt to think in terms of writing code that can be easily tested.

Mark Pilgrim‘s Dive Into Python has great stuff on how to approach a problem by defining the tests first and gradually filling in the code that satisfies the test suite. One day I may be able to work like that, until then I work by writing a concise docstring, then stubbing out the function. Once the function is in a state where it might actually return a meaningful result I can play with it in the Python interpreter and start adding useful doctests to the docstring.

What really helps is to break the logic out into tiny pieces where ideally each piece returns the result of transforming the input (which I think is known as a functional approach). By doing this I can have tests for most of the code and those functions that have a lot of conditional logic, those functions that are harder to write tests for, will at least be relying on sub-routines that are themselves well tested.

I can dream.

#!/usr/bin/python
"""
    Search for and kill app. 
"""
import os, sys
import commands
import signal


def main():
    if len(sys.argv) != 2:
        print 'Missing or too many arguments.'
        print 'One argument and only one argument is required.'
        print 'Pass in the app name to find and kill (i.e. "Safari").'
        return 0

    psCmd = '/bin/ps -x -c | grep ' + sys.argv[1]
    st, output = commands.getstatusoutput( psCmd )

    if st == 0:
        appsToKill = output.split('\n')
        for app in appsToKill:
            parts = app.split()
            killCmd = 'kill -s 15 ' + parts[0]
            #print killCmd
            os.system( killCmd )

if __name__ == "__main__":
    main()

(You can download the Acrobat 9.1.3 update and find this script at Acrobat 9 Pro Patch.app/Contents/Resources/FindAndKill.py.)

Was the author not aware of the killall command for sending a kill signal to a named process? The killall man page says it appeared in FreeBSD 2.1, which was released in November 1995. Adobe CS4 was released about 14 years later. How is it Adobe’s product managers approve these things for release?

What is particularly galling about Adobe’s Acrobat 9 updaters is that they seem to re-implement so much of what the Apple installer application does, even down to their use of gzipped cpio archives for the payload.

At some point in the process of re-writing each database for use with Django I have needed to convert all the records from Filemaker to Django. There exist good Python libraries for talking to Filemaker but they rely on the XML Web interface, meaning that you need Filemaker running and set to publish the database on the Web while you are running an import.

In my experience Filemaker’s built-in XML publishing interface is too slow when you want to migrate tens of thousands of records. During development of a Django-based application I find I frequently need to re-import the records as the new database schema evolves – doing this by communicating with Filemaker is tedious when you want to re-import the data several times a day.

So my approach has been to export the data from Filemaker as XML using Filemaker’s FMPXMLRESULT format. The Filemaker databases at work are old (Filemaker 5.5) and perhaps things have improved in more recent versions but Filemaker 5/6 is a very poor XML citizen. When using the FMPDSORESULT format (which has been dropped from more recent versions) it will happily generate invalid XML all over the shop. The FMPXMLRESULT format is better but even then it will emit invalid XML if the original data happens to contain funky characters.

So here is filemaker.py, a Python module for parsing an XML file produced by exporting to FMPXMLRESULT format from Filemaker.

To use it you create a sub-class of the FMPImporter class and over-ride the FMPImporter.import_node method. This method is called for each row of data in the XML file and is passed an XML node instance for the row. You can convert that node to a more useful dictionary where keys are column names and values are the column values. You would then convert the data to your Django model object and save it.

A trivial example:

import filemaker

class MyImporter(filemaker.FMPImporter):
    def import_node(self, node):
        node_dict = self.format_node(node)
        print node['RECORDID'], node_dict

importer = MyImporter(datefmt='%d/%m/%Y')
filemaker.importfile('/path/to/data.xml', importer=importer)

The FMPImporter.format_node method converts values to an appropriate Python type according to the Filemaker column type. Filemaker’s DATE and TIME types are converted to Python datetime.date and datetime.time instances respectively. NUMBER types are converted to Python float instances. Everything else is left as strings, but you can customize the conversion by over-riding the appropriate methods in your sub-class (see the source for the appropriate method names).

In the case of Filemaker DATE values you can pass the datefmt argument to your sub-class to specify the date format string. See Python’s time.strptime documentation for the complete list of the format specifiers.

The code uses Python’s built-in SAX parser so that it is efficent when importing huge XML files (the process uses a constant 15 megabytes for any size of data on my Mac running Python 2.5).

Fortunately I haven’t had to deal with Filemaker’s repeating fields so I have no idea how the code works on repeating fields. Please let me know if it works for you. Or not.

Download filemaker.py. This code is released under a 2-clause BSD license.

Under Win32 platforms, file times are maintained primarily in the form of a 64-bit FILETIME structure, which represents the number of 100-nanosecond intervals since January 1, 1601 UTC (coordinate universal time).

UPDATED New version with fixes by Tim Williams for preserving microseconds. See here for details.

It just so happens that Microsoft Active Directory uses the same 64-bit value to store some time values. For example the accountExpires attribute is in this format. Linked below is a module for Python with utility functions for converting between Python’s datetime instances and Microsoft’s FILETIME values.

Very handy if you enjoy querying Active Directory for login accounts that are due to expire. And who wouldn’t enjoy that? On a Monday.

Download filetimes.py module for converting between FILETIME and datetime objects. This code is released under a 2-clause BSD license.

Example usage:

>>> from filetimes import filetime_to_dt, dt_to_filetime, utc
>>> filetime_to_dt(116444736000000000)
datetime.datetime(1970, 1, 1, 0, 0)
>>> filetime_to_dt(128930364000000000)
datetime.datetime(2009, 7, 25, 23, 0)
>>> "%.0f" % dt_to_filetime(datetime(2009, 7, 25, 23, 0))
'128930364000000000'
>>> dt_to_filetime(datetime(1970, 1, 1, 0, 0, tzinfo=utc))
116444736000000000L
>>> dt_to_filetime(datetime(1970, 1, 1, 0, 0))
116444736000000000L

I even remembered to write tests for once!

So I fix it. The updated iCalendar files have events with time zone information.

Everyone’s happy.

Jon Udell’s use of Python to explore data manipulation on the Web was one of the reasons I thought I really ought to get stuck into Python.