Python

Fun with ctypes

Simon Willison — 2006-10-31T11:36:25-00:00

This probably only works on Intel-based OS X machines:

>>> import ctypes
>>> print ctypes.c_char_p(
    -16 * 4096 + 0x1600
).value
Your karma check for today:
There once was was a user that whined
his existing OS was so blind,
he'd do better to pirate
an OS that ran great
but found his hardware declined.
Please don't steal Mac OS!
Really, that's way uncool.
   (C) Apple Computer, Inc.U??VWS?5P

Adapted from dsmos.c in Understanding Apple's Binary Protection in Mac OS X by Amit Singh.

What I'm excited about, post-conference edition

Simon Willison — 2006-09-22T00:25:11-00:00

Wow, I've had a really busy month. I've attended (and spoken at) BarCamp London, Media in Transition, d.Construct, RailsConf Europe, Euro Foo and EuroOSCON. All were excellent, and each one nicely complemented the others. I'm exhausted. I think my brain is full.

My favourite question to ask new people I meet at conferences is "what are you excited about?". It's better than "what do you do?" (their job might not be as exciting as what they do in their spare time) and often gets a really interesting reply. People often ask me the same back, so here are three things that have been catching my attention recently.

OpenID. It's criminal that so few people are playing with this. I gave talks about it at both BarCamp and Euro Foo - it's decentralised single sign-on that works, and it's trivial to implement thanks to really solid libraries for most programming languages. There's also a $5,000 bounty to help spur adoption. I'll be writing a lot more about this in the future.
Virtualization. This was a common thread at several conferences, and the recent popularity of Parallels for browser testing barely scratches the surface. Virtual servers have a bunch of advantages over physical servers: you can clone them instantly, you can migrate them between machines (while they are still running if you're using Xen) and Amazon's EC2 offers utility computing on an enormous scale.
Dynamic languages on virtual machines. IronPython 1.0 is out, Sun have hired the JRuby guys. It looks like dynamic languages are finally being taken seriously as useful and powerful alternatives to C# and Java. Programmers on those VMs get more productive languages, while users of those languages gain access to enormous existing class libraries, not to mention the promise of significant performance boosts.

Finally, since I've blogged the last two releases of Python I can't resist saying a few things about the new Python 2.5. It's all good, but the stuff that really stands out is the addition of sqlite3, ElementTree and ctypes to the standard library. Batteries included!

The YDN Python Developer Center

Simon Willison — 2006-08-08T20:56:26-00:00

I recently had the opportunity to put together the Python Developer Center for the Yahoo! Developer Network. YDN is one of my favourite parts of Yahoo! so I jumped at the chance, and the resulting mini-site is now online (YDN blog post here).

The bulk of the content is the HOWTOs, which discuss ways of accessing Yahoo! APIs using Python:

I had a lot of fun playing around with different ways of accessing the APIs and working out which ones were the most natural fit. The HOWTOs use urllib, urllib2 and xml.dom.minidom from the standard library, but also discuss httplib2, ElementTree and simplejson as third party libraries that are worth investigating. Naturally, feedparser is the recommended tool for accessing Yahoo!'s multitude of RSS feeds.

Python really is a fantastic language for exploring web service APIs. All of the example code for the HOWTOs was first written in an interactive prompt and then copied to a file once it was working. Test-first development is certainly an important technique, but the power of interactive development should never be underestimated.

Exciting stuff in Python 2.5

Simon Willison — 2006-04-06T22:15:14-00:00

Python 2.5 alpha 1 is out, and as usual the What's New in Python 2.5 document provides a pleasant overview of the new features. There are some real treats in there. While I'm hoping that the syntax for conditional expressions will grow on me, I'm looking forward to Partial function application becoming a common Python idiom. Relative imports are going to make Django applications a lot easier to redistribute, and I can't wait to see all the crazy hacks that result from the introduction of coroutines.

I've seen David Heinemeier Hansson speak a few times, and he often uses ActiveRecord's transaction support to illustrate the elegance that well-written Ruby has to offer:

Account.transaction(account1, account2) do
  account1.withdraw(100)
  account2.deposit(100)
end

Here a Ruby block is being used as syntax for a database transaction, guaranteeing that some code will be run before and after the withdraw and deposit lines. The new with statement makes something similar possible with an imaginary Python ORM:

with Account.transaction(account1, account2):
    account1.withdraw(100)
    account2.deposit(100)

The highlight of the release might well be the modules that have been added to the standard library - tucked away in section 13 of the document. ctypes, ElementTree, a new hashlib with support for SHA-224 through 512 and sqlite3 (a renamed pysqlite) are all included. ctypes is particularly interesting as it lets you call functions in compiled DLLs and shared libraries without having to compile a Python wrapper.

The final Python 2.5 release appears to be scheduled for June-July this year, which should give Apple ample time to incorporate it in to OS X Leopard.

Exciting developments with Django

Simon Willison — 2005-08-03T16:56:34-00:00

The amount of activity surrounding the Django web framework since its not-quite release a few weeks ago is amazing. Adrian, Jacob and Wilson have been working over-time, with 395 check-ins to source control since the 13th of July. They've added WSGI support, a development web server, unit-tests, a ton of documentation, SQLite support, database introspection and dozens of other feature tweaks and bug fixes. Check out the project Timeline for an idea of just how frenetic things have been.

The emerging Django community has been kicking in as well. There's a significant community-led initiative to get internationalisation and localisation going, and a wide number of unofficial tutorials have emerged to complement the one on the site.

Here's where things get really interesting: changes at the Journal-World have kick-started the Django job market. Rob Curley, formally in charge of the World Company's web activities, recently took up a new position at the Naples Daily News in Florida. Rob just hired Eric Moritz, a regular on the #django IRC channel, to work on Django-powered projects there.

Meanwhile, Adrian Holovaty has taken a new job at the Washington Post as "Editor, Editorial Innovations" - a role that is sure to involve some very innovative use of Django (Adrian built chicagocrime.org). Adrian's departure means that the Journal-World are looking for a new developer - here's why you should apply.

One thing's for certain: we're going to see some very exciting Django-powered sites in the next few months.

Introducing Django

Simon Willison — 2005-07-17T11:59:57-00:00

You may know that I spent a year working in Kansas for a local newspaper - the Lawrence Journal-World. I'm delighted to announce that a decent chunk of the software I worked on there is now available as open-source, in the form of the Django web framework.

Django is an MVC Python web development framework with a strong emphasis on content management. While comparisons with Rails are inevitable (I plan to post one myself shortly), it should be emphasized that Django is by no means a clone of Rails - in fact, development on Django started in October 2003, months before the first public Rails release. That the two frameworks share so many ideas is, I feel, a testament to the design of both.

Django was initially developed by Adrian Holovaty and myself, with Jacob Kaplan-Moss joining the team shortly before my departure. The framework evolved during the construction of a number of sites, which included:

6newslawrence.com, local TV news, the first site to launch using the then nascent Django framework.
lawrence.com, a local entertainment site. Every town should have a site like this; sadly, very few do.
VisitLawrence.com, a local tourism site.
KUSports.com, a sports news site covering the University of Kansas.
LJWorld.com, a local news site and the flagship for the Journal-World.

It's also purring along under the hood of chicagocrime.org and the Django project site itself.

Adrian and I developed Django to allow us to create reasonably complex database driven web applications in as little time as possible. A local newspaper produces a huge amount of information - news stories, events listings, obituaries, sports results, marriage announcements, photo galleries, polls, weather reports and dozens of other bits and pieces. To maximise their value and potential for reuse, these things need to be stored in a database. That database needs a user interface for people to add and modify content, and of course the information needs to be published to a website somewhere.

Django automates most of this - and makes the rest as easy as possible. You create a data description (kind of like an SQL table schema but with additional information about validation rules and interface widgets) and load it in to Django. Django then creates the database tables, model classes and a comprehensive web-based administration interface for your site's staff. All that's left for you to do manually is the code for the public site, which is generally a case of writing a few lines of controller code, configuring some URLs and knocking out some templates.

To give you an idea of how much this speeds up the creation of new sites, consider the case of Game. Game was a site created to provide detailed coverage of little-league games in and around Lawrence (for non-Americans, little-league is baseball and softball for children aged around 5 to 11 - it's amazingly popular). The entire Game site, including news, fixtures, match results, team profiles and even weather forecasts for forthcoming matches took two days to develop.

I haven't been involved with Django since leaving the Journal-World back in September, but now that the framework is open-sourced I look forward to contributing to its further development - both in terms of documentation and actual code. There's already plenty to look at on the Django site (designed by the ever-talented Wilson Miner); I suggest the overview as your first port of call.

Expect to hear a lot more about the framework in the next few weeks.

PyCon observations

Simon Willison — 2005-03-28T17:08:22-00:00

I'm back from my two week stint in the US, and currently suffering from vicious jet-lag (my body wants me to go to sleep at 5am and wake up just past noon). Herewith some observations on PyCon, SxSW and the differences between the two.

PyCon 2005 was a great conference, and a very different one from SxSW Interactive the week before. While SxSW was one big social party with panels thrown in to fill the gaps, the sessions in PyCon were the main event and the social stuff (with the exception of the sprints, which I didn't really experience) was much less prominent. For the first day of the conference I actually found it quite hard to spark up conversations with strangers, something I'd been doing for pretty much the whole of SxSW. Things got better on the second and third days, but the lack of any organised social events and more reserved atmosphere meant I didn't have nearly as many random social experiences as at SxSW.

The PyCon sessions really were excellent: three great keynotes (the IronPython keynote was my favourite), an excellent web track and a whole smorgasbord of interesting topics spread over the three days. I have only one big complaint: all sessions apart from the keynotes were half an hour in length. For most sessions this worked fine, but some of the more experienced presenters were obviously shackled by the half hour requirement. Bruce Eckel's presentation was the most noticable in this regard - I love the stuff he covered, but it's obvious he could have gone on for a lot longer without losing the attention of the crowd (he obviously thought the same).

My suggestion for next year would be to keep most of the sessions at half an hour, but schedule a small number of 45 minute sessions for presenters who are obvious candidates for longer talks. I talked to Steve Holden (this year's organiser) briefly about this and he mentioned that 45 minute sessions lead to scheduling difficulties, particularly with respect to coordinating the different tracks. I personally think that the benefits of longer sessions for certain key topics would outweigh the scheduling disadvantages.

A few other PyCon observations:

There were over 400 attendees, at least a hundred more than last year. This slightly exceeded the capacity of the conference center, and they'll be mobing to a larger (as yet undecided) venue for 2006.
I only attended one of the two lightning talk sessions, but it was great fun and a refreshing change from the regular panels. The highlight for me was the guy who strapped a computer to the back of his motorcycle and drove 7,000 miles across America... with Python to coordinate all of the pieces. You can read more on his site, or in this article on Py.
The two (sometimes three) tracks were well arranged, with few clashes between things that I wanted to see. This was in contrast to SxSW's 5 tracks which had serious clashes pretty much all the time.
Everyone was hiring! The conference package we got was stacked with job brochures from the conference sponsors, and the whiteboard by the registration desk had new jobs added to it every day. Sure-fire evidence that Python is finally starting to gain significance in the job market.
The lunches, included in the conference price, were excellent. The price itself was great value too - early bird for students was $125, and $175 for regular attendees. Even late registration was only (from memory) $275.
The largest venue at the center, used for the keynotes, had no WiFi! Coverage throughout the rest of the conference was good however.
I finally got to join Ted Leung and friends in a SubEthaEdit session during the Python at Google keynote. It was an electrifying experience watching each slide transcribed in to the notes within seconds of it appearing on screen, with multiple lines developing at the same time. The results of our labour can be seen here. Someone really needs to put together a screencast of this kind of thing so the rest of the world knows what they're missing.
Despite my observations about the less social nature of the conference above, I met some very interesting people and had a really great time.

It seems to me that Python and SxSW could learn some tricks from each other. Lightning talks and Birds-of-a-feather sessions would be a great addition to the SxSW lineup, while PyCon really does need some more thought put in to the social side of the conference. I hope to attend both again next year.

Casting out getters and setters

Simon Willison — 2004-12-03T15:52:41-00:00

Python Is Not Java by Phillip J. Eby (via Ned) is the most useful article on programming I've read in ages. If you have any interest at all in either language, go and read it. It's all good, but the part that really struck a nerve for me was this:

Getters and setters are evil. Evil, evil, I say! Python objects are not Java beans. Do not write getters and setters. This is what the 'property' built-in is for. And do not take that to mean that you should write getters and setters, and then wrap them in 'property'. That means that until you prove that you need anything more than a simple attribute access, don't write getters and setters. They are a waste of CPU time, but more important, they are a waste of programmer time. Not just for the people writing the code and tests, but for the people who have to read and understand them as well.

Writing getters and setters in Java has always bugged me, but I've never quite been able to articulate the reason until now. Python's property syntax (and Ruby's similar attr_reader and attr_writer methods) are so much more elegant that writing those things by hand, or even auto-generating them with Eclipse, leaves a nasty taste in my mouth.

Running Pydoc under mod_python

Simon Willison — 2004-09-28T03:34:46-00:00

I've written about pydoc before. In my opinion it's one of Python's best kept secrets: a way of instantly browsing the properties, methods and documentation strings of any module available to the Python environment. It can even run a local HTTP server to allow for easy browsing of available documentation.

That's all well and good for code running on your local machine, but most developers won't want to run a Pydoc server on a remote machine. We ran in to this problem recently at work; we wanted to run Pydoc on our development server but didn't want it exposed to the world at large.

Initially we considered running the regular Pydoc server locally on some random port, blocking that port to external IPs with a firewall rule and then using Apache's mod_proxy to open it up to outside access in a controlled manner. The downside with this is that we would have to run a special process and a firewall rule just to enable a relatively minor function. Instead, I spent some time working through the pydoc.py source code and came up with a mod_python wrapper for the HTML documentation part of the module.

modpydoc.py

(Aside: Safari will try to render the above as HTML, even though the Content-Type header says text/plain. This is because Safari is brain-dead - it inherited this particular grotesqueness from Internet Explorer. I still use it though.)

Since we were already running mod_python on our development server, adding this extra module required no extra server configuration or additional server processes. It also allowed us to tie authentication for the resulting Pydoc instance to our existing authentication database, using mod_auth_pgsql.

For the inquisitive, here's a rough approximation of the relevant parts of our httpd.conf file for that server:


LoadModule access_module modules/mod_access.so
LoadModule auth_module modules/mod_auth.so
LoadModule alias_module modules/mod_alias.so
LoadModule auth_pgsql_module modules/mod_auth_pgsql.so
LoadModule python_module modules/mod_python.so

RedirectMatch permanent /pydoc$ /pydoc/

<Location "/pydoc/">
    SetHandler python-program
    PythonHandler utils.modpydoc
    PythonPath "sys.path+['/path/to/our/codebase/']"
    AuthName "Code Browser"
    AuthType basic
    Auth_PG_database ourdb
    Auth_PG_user dbuser
    Auth_PG_pwd_table users
    Auth_PG_uid_field username
    Auth_PG_pwd_field password_md5
    Auth_PG_pwd_whereclause " AND can_browse_code = 't' "
    Auth_PG_hash_type MD5
    require valid-user
</Location>

Python2.4 highlights

Simon Willison — 2004-09-21T02:36:47-00:00

A.M. Kuchling's "What's New in Python X" documents are always a treat, and his guide to the forthcoming Python 2.4 is no exception. Among other things, 2.4 elevates sets to built in type status, dramatically improves the usability of Python's list sort method (for easier application of DSU, aka the Schwartzian transform), makes reverse iteration easier and introduces an alternative string substitution method.

All that's before you get on to the really exciting stuff: generator expressions, the new decimal type (because floating point numbers are such a nuisance) and the controversial function decorators.

I have to admit I didn't understand the significance of half of this stuff until I read about them in "What's New", which explains the use-cases for the new features with great clarity.

Python seems to advance at just the right rate; new features are introduced fast enough to keep me interested (and keep the language feeling alive) but not so fast as to leave me feeling left behind.

Command line blacklisting

Simon Willison — 2004-09-09T05:59:46-00:00

Just over a year ago, I started blacklisting domain names from links featured in comment spam. My idea then was that these blacklists could become a shared resource: people would publish their own blacklist and subscribe to those of people they trust, thus making it much harder for spammers to operate. While the sheer volume of spam domains meant that the technique was much less useful than I originally anticipated, I've continued to maintain my blacklist ever since as a preventative measure against repeat spammers.

I have a confession to make: all of my blog administration (with the exception of adding entries and blogmarks) is performed using phpMyAdmin. The trouble with writing your own software is that it's very easy to skimp on the backend tools, since you're the only person who will ever see them. Incidentally, this is the main reason I plan to switch to WordPress just as soon as I find the inspiration to write the necessary import scripts. Comments are deleted in phpMyAdmin, and domains are blacklisted by manually editing the blacklist.txt file via FTP.

This has been really bugging me, especially since I have so little other use for FTP that my only installed client is an unregistered version of Transmit (closes after ten minutes, won't save passwords along with account details). I've been muddling along with that for longer than I care to admit, but today I decided to take 10 minutes out to solve the problem once and for all. I could have put together a web interface for adding new domains but I wasn't really in the mood, so I decided to put time spent reading The Art of Unix Programming to good use and knock out a simple command line application.

The result (minus my login details) can be found here. Sample usage: ./blacklist.py www.domain.org www.domain2.com. It follows the Unix ideal of being the simplest-thing-that-could-possibly-work, and ended up taking longer to write than I expected thanks mainly to the craziness of Python's ftplib. I've seen complaints about this before, and it thoroughly deserves its bad reputation.

Here's one example: retrlines is the method used to retrieve ascii text from the server. Bizzarely, it doesn't actually return the text receieved; instead, it expects you to provide it with a callback function that will be fed each line in turn, minus the newline. Sounds like a job for StringIO, but StringIO objects don'y have a writeline method (required to add the newline back on). I ended up writing my own extension of the StringIO2 class and adding a writeline method just to preserve the newlines returned from the server!

Strange APIs aside, I'm pretty pleased with the final result. It follows a bunch of Unix design patterns (and skips others such as those related to configuration, but I'm not overly bothered about those) including the following:

A usage note is displayed if no arguments are provided.
Multiple domains can be blacklisted at once, by providing them as multiple command line arguments.
Domains that are already in the blacklist are skipped, and a message is written to standard error.
If the script suceeds, it doesn't say anything at all.

It also uses the common Python idiom of wrapping the principle logic in a function and then calling that from a block that runs only if the file is executed directly (the __name__ == '__main__' idiom) so that other Python code can import the module and reuse its functionality if required.

There's plenty of room for improvement: being able to pipe a list of domains in via standard input would be nice, and hard coding the (unencrypted) username and password is sloppy (as is expecting the blacklist.txt file to live in the FTP home directory). Even better, with SSH access the whole thing could be replaced with an infinitely more secure one-liner: echo www.domain-to-ban.org | ssh username@server "cat - >> blacklist.txt". I'm happy though: an irritating task has become much less irritating and I have some example code to fall back on next time I need to get mucky with ftplib.

Instant authentication against an existing web application

Simon Willison — 2004-07-15T00:12:14-00:00

I was thinking today about the problem of querying an existing authentication database from a new application - exactly the kind of thing web services are useful for. Then I realised that any web application protected by HTTP Basic authentication already provides a standard API against which queries can be run. Here's the Python code to do exactly that:


def auth_against_url(url, username, password):
    import urllib2, base64
    request = urllib2.Request(url)
    b64 = base64.encodestring('%s:%s' % (username, password))[:-1]
    request.add_header('Authorization', 'Basic %s' % b64)
    try:
        urllib2.urlopen(request)
    except urllib2.HTTPError:
        return False
    return True

To check a username and password against an existing application's user database, just call the above function with the URL of a page within the existing application as the first argument. The function returns True if the username and password are valid, and False otherwise. It doesn't get much simpler than that.

A nice side effect of using Python's standard library modules is that they transparently support HTTPS, so authentication can take place over an encrypted channel provided the target application supports it.

Compile everything with a one-liner

Simon Willison — 2004-07-01T17:59:39-00:00

The other day, we noticed that the .py files in our main mod_python application at work did not have corresponding compiled .pyc files. mod_python runs as the unprivileged apache user, which on our server doesn't have the required permissions to write the compiled .pyc files in to the directories in which our code lives.

A bit of digging came up with the compileall module, which provides functions for byte-compiling Python files in bulk. We invoked it using the following shell one-liner, executed at the upper most directory of our code tree:

python2.3 -c "import compileall; compileall.compile_dir('.')"

Executed as root, the result was a full set of .pyc bytecode files.

Backporting from Python 2.3 to Python 2.2

Simon Willison — 2004-06-09T04:58:22-00:00

We have a home-grown templating system at work, which I intend to dedicate an entry to some time in the future. We originally wrote it in Python 2.2, but upgraded to Python 2.3 a while ago and have since been evolving our code in that environment. Today I found a need to load the most recent version of our templating system on to a small, long neglected application that had been running the original version ever since it had enough features to be usable.

Unfortunately, this application was running on a server that only had Python 2.2. Installing Python 2.3 would have been somewhat more painful here than on other servers we run for reasons I won't go in to, so I decided to have a go at getting our current code to run under the older Python version.

In the end, I only had to make three minor changes, all at the top of the file in question.

I added from __future__ import generators as the very first line of the file. We use generators (with the yield statement) in a few places - this feature was only properly added in Python 2.3, but was made available in Python 2.2 as a "future enhancement" through the aforementioned obscure import.
I added True, False = 1, 0 on the next line down. Surprisingly, Python 2.2 had no support for a boolean type and instead used a test for non-zero. The above line defines constants that behave enough like Python 2.3's True and False to avoid any problems.
I defined an enumerate function, which was introduced for real in Python 2.3. Here's the code I used:
```
def enumerate(obj):
    for i, item in zip(range(len(obj)), obj):
        yield i, item 
```

All in all it only took around ten minutes to put the above together, after which the script worked just fine. It was interesting to see how our code had grown to rely on Python 2.3 features without us realising it.

Update: Check this entry's comments for improvements to the above code snippets.

Switch statements in Python

Simon Willison — 2004-05-07T02:43:14-00:00

Python doesn't support a native switch statement. I've found myself using the following coding idiom instead recently which seems to work pretty well:

{'option1': function1,
 'option2': function2,
 'option3': function3,
 'option4': function4}[value]()

This works with lambdas as well, for inline calculations. Here's a switch statement that assigns to a variable in PHP:

switch ($value) {
    case 'a':
        $result = $x * 5;
        break;
    case 'b':
        $result = $x + 7;
        break;
    case 'c':
        $result = $x - 2;
        break;
}

And here's the equivalent code in Python:

result = {
  'a': lambda x: x * 5,
  'b': lambda x: x + 7,
  'c': lambda x: x - 2
}[value](x)