Reliably Broken » unix

Split a file on any character in Python

david — Thu, 15 Apr 2010 10:38:12 +0000

I need to split a big text file on a certain character. I expect I am being thick about this, but split doesn’t quite do what I want because it includes the matching line, whereas I want to split right on the matching character.

My Python answer:

def readlines(filename, endings, chunksize=4096):
    """Returns a generator that splits on lines in a file with the given
    line-ending.
    """
    line = ''
    while True:        
        buf = filename.read(chunksize)
        if not buf:
            yield line
            break

        line = line + buf

        while endings in line:
            idx = line.index(endings) + len(endings)
            yield line[:idx]
            line = line[idx:]

if __name__ == "__main__":
    import sys, os

    FORMFEED = chr(12) # ASCII 12
    basename = os.path.basename(sys.argv[1])
    for num, data in enumerate(readlines(open(sys.argv[1]), endings=FORMFEED)):
        filename = basename + '-' + str(num)
        open(filename, 'wb').write(data)

This is also useful when reading data exported from some old-fashioned Mac application like Filemaker 5 where the line-endings are ASCII 13 not ASCII 10.

This post was inspired by Lotus Notes version 8.5, which is so advanced that to save a message in a file on disk you have to export it as structured text. And if you want to save a whole bunch of messages as individual files you must forget that drag-and-drop was introduced with System 7, that would be too obvious.

Removing printing restrictions in 10.5

david — Wed, 13 Jan 2010 17:38:32 +0000

I am trying to write a good one-liner for removing all restrictions on printing for Mac OS X 10.5. I had thought that sed would be perfect for this, but I can’t arrive at a simple syntax for appending new lines that works well when pasted into a terminal window. Here’s what I ended up with:

perl -p -0 -i '.bak' -e 's/(Policy default).*(Policy)/$1>\n\nOrder deny,allow\nAllow from all\n<\/Limit>\n<$2/s' /private/etc/cups/cupsd.conf

Rather brutal, it just guts the default policy and replaces it with the following:



Order deny,allow
Allow from all

Greg Neagle has a useful article about printing in the enterprise. Apple suggests adding the network group to the local lpadmin group, but points out that mobile users would need to be added individually. In my case most accounts are mobile accounts and we trust everyone to manage print queues on a Mac, so removing all restrictions is acceptable.

Using screen to log a terminal session

david — Tue, 06 Jan 2009 21:58:55 +0000

Note to self: use screen when doing a bunch of remote administration stuff over an SSH connection. You can tell it to log the session to a separate file, and can detach the session and log off without having the remote shell be terminated. Then later you can resume the session and haven’t lost anything.

To start a shell and log everything to a file, do

screen -L -S mysession

That drops you into a new shell, and lets you refer to that session later (in case you are ambitious and have many screen sessions running simultaneously). To detach the session type CTRL+a then d. From there you can exit cleanly. To resume the session later, type

screen -r mysession

The session history will be saved to a file named screenlog.0 or similar in the directory where screen was first invoked.

screen man page online (Hmmm… looks like the hmug man pages have renamed themselves.)

I should use screen more often. I should also get in the habit of going through the contents of $PATH every time Mac OS X gets updated.