jerakeen.org

I will be attending London Python meetup on Wednesday October 08

http://upcoming.yahoo.com/event/1135363/

PIL - the Python Imaging Library - documentation. PIL is a pain to work with, and needs to be installed from source on MacOS, but works and doesn’t make me as annoyed as ImageMagick does. So it wins.

http://www.pythonware.com/library/pil/handbook/index.htm

Exactly what I wanted. A pure-python standalone WSGI server that I can put behind an apache and host my Django jerakeen.org implementation on. Fast, too.

http://www.eflorenzano.com/blog/post/hosting-django-site-...

http://www.saurik.com/id/5

I went to DJUGL (pronounced ‘juggle’) yesterday, to watch tech talks and say hello to people. I learned the following things: (I know! Learning things! At a tech talk!)

• IPython, an improved Python shell. Does tab-completion, amonst other things. The Django ‘shell’ command will use it automatically if it’s installed.

• The SEND_BROKEN_LINK_EMAILS setting - sends mail to addresses listed in the MANAGERS config variable when the Django server serves a 404. Not something I particularly want to turn on, but I liked it. I also like the way Django will send mail on every server error. The absolute fastest way to get live crash bugs fixed is to mail all the developers every time they happen.

• There was some cool middleware that displayed profiling information. Must use it in something.

• The django-tagging application bears looking into at some point.

Simon has talk notes up.

As is my wont, I’m in the middle of porting jerakeen.org to another back-end. This time, I’m porting it back to the Django-based Python version (it’s been written in rails for a few months now). It’s grown a few more features, and one of them is somewhat smarter comment parsing.

This being a vaguely technical blog, I have vaguely technical people leaving comments. And most of them want to be able to use HTML. I’ve seen blogs that allow markdown in comments, but I hate that - unless you’re know you’re writing it, it’s too easy for markdown to do things like eat random underscores and italicise the rest of the sentence by accident. But at the same time, I need to let people who just want to type text leave comments.

The trick then is to turn plain text into HTML, but also allow some HTML through. Because the world is a nasty place, this means whitelisting based on tags and attributes, rather than removing known-to-be-nasty things. Glossing over the ‘turn plain text into HTML‘ part, because it’s easy, here’s how I use BeautifulSoup to sanitise HTML comments, permitting only a subset of allowed tags and attributes:

# Assume some evil HTML is in 'evil_html'

# allow these tags. Other tags are removed, but their child elements remain
whitelist = ['blockquote', 'em', 'i', 'img', 'strong', 'u', 'a', 'b', "p", "br", "code", "pre" ]

# allow only these attributes on these tags. No other tags are allowed any attributes.
attr_whitelist = { 'a':['href','title','hreflang'], 'img':['src', 'width', 'height', 'alt', 'title'] }

# remove these tags, complete with contents.
blacklist = [ 'script', 'style' ]

attributes_with_urls = [ 'href', 'src' ]

# BeautifulSoup is catching out-of-order and unclosed tags, so markup
# can't leak out of comments and break the rest of the page.
soup = BeautifulSoup(evil_html)

# now strip HTML we don't like.
for tag in soup.findAll():
    if tag.name.lower() in blacklist:
        # blacklisted tags are removed in their entirety
        tag.extract()
    elif tag.name.lower() in whitelist:
        # tag is allowed. Make sure all the attributes are allowed.
        for attr in tag.attrs:
            # allowed attributes are whitelisted per-tag
            if tag.name.lower() in attr_whitelist and attr[0].lower() in attr_whitelist[ tag.name.lower() ]:
                # some attributes contain urls..
                if attr[0].lower() in attributes_with_urls:
                    # ..make sure they're nice urls
                    if not re.match(r'(https?|ftp)://', attr[1].lower()):
                        tag.attrs.remove( attr )

                # ok, then
                pass
            else:
                # not a whitelisted attribute. Remove it.
                tag.attrs.remove( attr )
    else:
        # not a whitelisted tag. I'd like to remove it from the tree
        # and replace it with its children. But that's hard. It's much
        # easier to just replace it with an empty span tag.
        tag.name = "span"
        tag.attrs = []

# stringify back again
safe_html = unicode(soup)

# HTML comments can contain executable scripts, depending on the browser, so we'll  
# be paranoid and just get rid of all of them  
# e.g. <!--[if lt IE 7]><script type="text/javascript">h4x0r();</script><![endif]-->  
# TODO - I rather suspect that this is the weakest part of the operation..
safe_html = re.sub(r'<!--[.\n]*?-->','',safe_html)

It’s based on an Hpricot HTML sanitizer that I’ve used in a few things.

Update 2008-05-23: My thanks to Paul Hammond and Mark Fowler, who pointed me at all manner of nasty things (such as javascript: urls ) that I didn’t handle very well. I now also whitelist allowed URIs. I should also point out the test suite I use - all code needs tests!

Shockingly, python doesn’t seem to be able to parse iso8601 dates in core. But this works.

http://code.google.com/p/pyiso8601/

Two stupid fixes to two stupid problems. I thought MacOS was a sensible platform nowadays. Anyway, bookmarked so I can find it again, because it works.

http://www.davidcramer.net/code/57/mysqldb-on-leopard.html

Shelf 0.0.13 is available, though it’ll auto-update itself if you let it. Not a lot in this one, the most interesting thing for me is that Dopplr support is back in and works a lot better - there’s now a button to click that’ll get you a token, for instance. I’ve also squashed various crashes for cases where data was bad.

Full changelog for this version here.

Shelf is an app for MacOS that looks at the current foreground application, and tries to figure out if what you’re looking at corresponds to a person in your Address Book. Then it’ll tell you things about them.

What it says on the tin.

http://phildawes.net/microformats/microformatparser.html

EmusicR is a (very trivial) download manager for emusic

My syndicated links thumbnails seemed like such a good idea at the time. I loved what ma.gnolia was doing, but I didn’t want to switch bookmarking services just for that. And I wanted all my content on one site anyway. But the pain, oh the pain.

The script that actually does the thumbnailing uses python and gtkmozembed. It requires an X server (for the mozilla) and I run it on my colo, which is headless, so I call it from a wrapper script that starts up an xvfb headless server and runs it. All a little fragile, I’m afraid. But it works surprisingly well. I call it from a cronned delicious syndication script (danger! ugly!) that pulls my delicious links into /links here every so often.

It was based a long time ago on mattb’s script/fragment and basically it’s all a bit nasty. I’ve tried to re-write it in various languages, but you’ll quickly find that the gtkmozembed bindings are awful - on most platforms they flat-out don’t work. My wrapper script has to munge LD_LIBRARY_PATH to get it to work under Debian, my server platform. Having to run an X-server is a pain as well. And because you’re wrapping mozilla, and not the underlying renderer, you can’t automatically bypass the security stuff, so it’s impossible to thumbnail sites with bad SSL certificates, you’ll just get a screenshot of the security confirmation dialog. I also can’t find a way of getting a ‘page loaded’ callback, so I just have to sleep some arbitrary amount of time before just blindly taking a screenshot of whatever I’m managed to render so far. That sort of thing.

At the time I wrote this script, I wanted an automated solution. I looked at various 3rd-party thumbnailing services but most of them just thumbnailed the root path of the domain you asked for, not the page itself. Most services also want you to deep-link their thumbnails rather than pulling them to a local server and serving from there, and they charge/quota by number of thumbnails served, rather than generated. I don’t know why. And I wanted thumbnails taken at the time that I bookmarked the page, not at the time you looked at the page. Picky, I know.

Were I to do it again, I think I’d seriously consider 3 alternatives:

Doing the thumbnailing on a Mac

Paul Hammond has a webkit2png script that does the same as my script, but using Webkit on a Mac. It’s almost as annoying, because you still need a windowing server, but the overheads are smaller - there are sensible callbacks so you can thumbnail faster, and it’s a more reliable environment - the bindings work. Of course, there are downsides - you need a Mac, for a start. And if you want an automated solution you’ll need a Mac connected to the internet and turned on 24 hours a day. But I have one of those under my telly now, so it’s tempting. Not sure if I’ll be able to solve the SSL certificate problem for this one, but it’s not a deal breaker.

WebThumb

Simon Willison found a thumbnailing service that doesn’t suck for Oxford Geeks. It’ll actually thumbnail the page itself (or at least, it did last time I poked at it) so if you don’t want the overhead of running your own server, this might work.

A simpler version

I now have a pure-C version of the thumbnailer script. I don’t use this version, but only because I wrote it as a thought experiment some time after I got everything working, and I don’t want to mess with something that works. I see no reason why the C version won’t do just as well, and it avoids most of the bindings pain. It’ll still need the wrapper, but dropping the python side of things might help.

I have Leopard now, which includes, amongst other things, Python 2.5.1 and PyObjC built-in. Because of this, I’ve been able to package Flame as a much smaller application - 70k compressed rather than 3 megabytes. But it’ll only work on Leopard. So far, I haven’t been able to use Leopard to build a version of Flame that’ll run on Tiger (10.4), but I’m sure I’ll figure it out eventually.

And I thought source filters for Perl were bad enough. Now Python has them.

http://timhatch.com/projects/pybraces/

pyobjc demo showing how to grab global hotkeys - for future reference.

http://svn.red-bean.com/pyobjc/trunk/pyobjc/Examples/AppK...

I need to add this to jerakeen.org so I have have openid-authenticated comments.

http://simonwillison.net/2007/Apr/24/openidconsumer/

Good musings, including some non-obvious stuff that really should be documented better. Keeping apps out of the project tree is a really good idea.

http://www.b-list.org/weblog/2007/03/27/reusable-django-apps

DjangoKit is a framework that will take a Django application, and turn it into a stand-alone MacOS application with a local database and media files. It started as more of a thought experiment than an effort at producing a real application, but I have it working, and you can package perfectly usable stand-alone applications with it.

DjangoKit is currently hosted in a Google Code repository, so go there for downloads and source, or just get the svn tree directly at http://djangokit.googlecode.com/svn/trunk/.

for some reason, test-driven development in python sometimes seems a little hokey. distutils doesn’t have a built-in recipie for running tests, for instance. But this is nice.

http://www.nedbatchelder.com/code/modules/rees-coverage.html

Trawler is a (currently internal) Mac GUI client for editing templates, tests, code and files on a Zimki server. It’ll be public Real Soon Now.

Update: Zimki is dead. Dead dead dead. So Trawler is about as much use as a chocolate teapot now. I guess I’ll leave the page here for posterity.

Another day, another python-daap release. Version 0.6 allows you to connect to shares requiring a password, and fixes some problems with the source tarball. However, there are still outstanding licensing issues that may entail an annoying change to the library in the future.

linked because it lists how to install the python-fuse bindings under macos. And boy, is it nasty.

http://blog.macos.fr/post/2007/01/18/GMailFS-for-Mac-OS-X

I’m experimenting with a simple source code browser for jerakeen.org. Right now it’s trivial - just a list of folders and links to files, but what I’m aiming for is pages showing the check-in history of various folders, when they were last changed, etc - essentially, the sort of boring stuff I’d get for free were I to use svnweb or trac or something.

As usual, though, that’s not the point. I’d hate to have a web site that consisted of several different apps, written in different languages, needing hundreds of different apache modules, and all looking different - or needing different templates if I wanted to give them similar appearances. I’m not very good at design and building templates, so as a crazy insane developer, it’s easier for me to write a subversion browser than it would be to bully trac into looking the way I want it.

So, the pysvn bindings - Python bindings to the subversion client library. They’re lovely.

import pysvn
client = pysvn.Client()
projects = client.ls("https://jerakeen.org/svn/tomi/Projects")
for project in projects:
    print " * %s"%project.name

The logic behind the pages under /source isn’t much more complex than that. There’s no caching, I don’t have to have a local checkout, and it’s easily fast enough for a little website like this. The (fairly sparse) docs don’t make it sufficiently clear, to my mind, that you can point the client at a remote repository instead of a local checkout, but you can.

Another trick (hack) I use is providing a ‘short name’ method to the directory entry objects. I pass the objects returned from the ls call directly to the django template, but you’re not allowed to do anything clever in template space (the templates are touched by those designer people - can’t trust ‘em). To make it easier to print a human-readable name for the entries, I poke a short method into their namespace:

def short_name(self):
    offset = self.name.rfind('/') + 1
    return self.name[offset:]

PysvnDirent.short = short_name

Then the template needs a simple

<h2>files here</h2>
{% for file in files %}
  <p><a href="{{ file.name }}">{{ file.short }}</a></p>
{% endfor %}

Evil. I’m clearly still too much of a perl programmer…

I have a lovely shiny office MacBook Pro sitting in front of me, and in the middle of the top of the screen is this little annoying black square. It’s an iSight camera, and it can always see me.

It’s annoying for two reasons. Firstly, it can always see me. There’s a little light to tell you that it’s on, but it’s perfectly capable of blipping on briefly without you noticing. But mostly it’s annoying because I hate the thought of such a high-tech piece of technology just going to waste up there.

I really have no real use for this thing at all - I don’t obsessively catalogue my book collection, I don’t hold frequent multi-person videoconference sessions, and I already have a camera. I’ve also seen enough PhotoBooth output to last me my ENTIRE LIFE. But merely having no good reason isn’t a good enough reason to stop me, so I came up with a use for it.

DuckCall is an application that runs in the background, and takes a picture of you every 30 seconds using the iSight. Then it’ll set this picture to be your iChat or Adium ‘status picture’ thing - the little picture of your head that other people see in their contact lists. Setting away messages is so web1.0 - when I’m not at my computer, you know it, because you can’t see me. It’ll also try to be smart - it won’t do anything if neither app is running, for instance.

Naturally, it’s not perfect. For the Adium stuff to work, you need to be using a recentish beta - the ones with the single buddy icon shared between services. And it has a nice little todo list, like all my projects.

The biggest thing wrong with it, though, is the Frankenstein nature of the thing. I write my crazy application prototypes in PyObjC because it’s very easy to get something working. But there are no Cocoa bindings for reading from the iSight (why??) so I shell out to an external tool and save a file from the iSight to disk. Then I use AppleScript to load the file from disk and set the icon of either of the IM apps that happen to be running (or both, I guess, if you’re weird). Using Python as the wrapper makes the app about 3 megabytes larger than it really should be. If someone wants to re-implement it in Objective C, that would be good.. :-)

Weirdy, I’m not sure if I like the philosophy of the app. I keep getting scared that it’ll take photos of me at bad times. Given my stated reluctance to publicise my life it’s odd that I even considered writing this thing. I comfort myself with the thought that no matter how stupid I look in this photo, it’ll be gone in 30 seconds, and most people will never see it…

Get it here and tell me what you think..

oooh, shiny python/linux presentation software for linux. Display only - no authoring, but I can cope.

http://keyjnote.sourceforge.net/

I tend to reimplement the CMS that drives jerakeen.org more often than I add content to it, but the current Django based incarnation seems to have decent sticking power. A lot of this is Django’s magic admin interface middleware. When I add, say, a tagging engine to the site, I only need to worry about the object model and presenting it on the site itself. All the boring and much harder to write admin pages to add and remove tags just write themselves. But the other reason I’m staying with it is that I’ve now added so many features to it (because it’s easy!) that a re-write in another language would be a huge amount of effort.

This weekend, for instance, I’ve added an implementation of the metaweblog API to the site, using the excellent code on allyourpixel as a base. The main source of pain is the persistent weirdness of implementing the Movable Type extensions to the metaweblog extensions to the Blogger XMLRPC API. How can you call something a metaweblog API and not allow for post excerpts, for instance? So annoying.

While implementing it, I found the TextPattern API reference to be far more useful than the official spec, mostly because it covers everything up to the Movable Type extensions, which you need if you want to edit page excerpts. The other problem I encountered was that Ecto won’t talk to an endpoint over HTTPS with a self-signed certificate unless the SSL cert is in the local machine X509 database. The way it fails is incredibly unhelpful and annoying, too. The simplest way to fix it (assuming a recent macos) is to visit the endpoint in Safari. It’ll complain about the certificate - click the ‘always trust this site’ box, and it’ll stop.