jerakeen.org

I used to think that if I did my own thing to handle comment spam then I’d be low-hanging fruit and wouldn’t have a problem. And this worked for quite a long time. But a couple of weeks ago it stopped working, and last week I turned off comments on jerakeen.org just so I didn’t have to delete 30 spam comments every day. And I’m far too lazy to do anything properly to solve this.

Fortunately, Disqus have appeared recently, and they’re great. I can just embed someone else’s commenting framework and let them deal with the problem. The obvious downside is that the comments aren’t ‘really’ on my page, so I won’t get any google juice from them. But on the other hand, the comments aren’t really on my page, so noone else will get any google juice from them. Maybe this’ll make them less appealing as a spam target in the first place.

The other downside is that there’s no import capability for my old comments. I’ve settled for just displaying the old comments in-place, with Disqus comments under them. I’ve also take the opportunity to make it slightly clearer when I’m syndicating comments from flickr rather than allowing them on the local site. I haven’t managed to combine the count of old and new comments yet, but I’m sure I’ll get it soon. Until then, pages with old-style comments will just have 2 figures for comment count. You’ll live.

As is my wont, I’m in the middle of porting jerakeen.org to another back-end. This time, I’m porting it back to the Django-based Python version (it’s been written in rails for a few months now). It’s grown a few more features, and one of them is somewhat smarter comment parsing.

This being a vaguely technical blog, I have vaguely technical people leaving comments. And most of them want to be able to use HTML. I’ve seen blogs that allow markdown in comments, but I hate that - unless you’re know you’re writing it, it’s too easy for markdown to do things like eat random underscores and italicise the rest of the sentence by accident. But at the same time, I need to let people who just want to type text leave comments.

The trick then is to turn plain text into HTML, but also allow some HTML through. Because the world is a nasty place, this means whitelisting based on tags and attributes, rather than removing known-to-be-nasty things. Glossing over the ‘turn plain text into HTML‘ part, because it’s easy, here’s how I use BeautifulSoup to sanitise HTML comments, permitting only a subset of allowed tags and attributes:

# Assume some evil HTML is in 'evil_html'

# allow these tags. Other tags are removed, but their child elements remain
whitelist = ['blockquote', 'em', 'i', 'img', 'strong', 'u', 'a', 'b', "p", "br", "code", "pre" ]

# allow only these attributes on these tags. No other tags are allowed any attributes.
attr_whitelist = { 'a':['href','title','hreflang'], 'img':['src', 'width', 'height', 'alt', 'title'] }

# remove these tags, complete with contents.
blacklist = [ 'script', 'style' ]

attributes_with_urls = [ 'href', 'src' ]

# BeautifulSoup is catching out-of-order and unclosed tags, so markup
# can't leak out of comments and break the rest of the page.
soup = BeautifulSoup(evil_html)

# now strip HTML we don't like.
for tag in soup.findAll():
    if tag.name.lower() in blacklist:
        # blacklisted tags are removed in their entirety
        tag.extract()
    elif tag.name.lower() in whitelist:
        # tag is allowed. Make sure all the attributes are allowed.
        for attr in tag.attrs:
            # allowed attributes are whitelisted per-tag
            if tag.name.lower() in attr_whitelist and attr[0].lower() in attr_whitelist[ tag.name.lower() ]:
                # some attributes contain urls..
                if attr[0].lower() in attributes_with_urls:
                    # ..make sure they're nice urls
                    if not re.match(r'(https?|ftp)://', attr[1].lower()):
                        tag.attrs.remove( attr )

                # ok, then
                pass
            else:
                # not a whitelisted attribute. Remove it.
                tag.attrs.remove( attr )
    else:
        # not a whitelisted tag. I'd like to remove it from the tree
        # and replace it with its children. But that's hard. It's much
        # easier to just replace it with an empty span tag.
        tag.name = "span"
        tag.attrs = []

# stringify back again
safe_html = unicode(soup)

# HTML comments can contain executable scripts, depending on the browser, so we'll  
# be paranoid and just get rid of all of them  
# e.g. <!--[if lt IE 7]><script type="text/javascript">h4x0r();</script><![endif]-->  
# TODO - I rather suspect that this is the weakest part of the operation..
safe_html = re.sub(r'<!--[.\n]*?-->','',safe_html)

It’s based on an Hpricot HTML sanitizer that I’ve used in a few things.

Update 2008-05-23: My thanks to Paul Hammond and Mark Fowler, who pointed me at all manner of nasty things (such as javascript: urls ) that I didn’t handle very well. I now also whitelist allowed URIs. I should also point out the test suite I use - all code needs tests!