Systems Administration

Problems with Trac? Switch to FSFS

Simon Willison — 2005-03-04T01:11:26-00:00

I'm head over heels in love with Trac, and have been for about 6 months now. It really is best-of-breed software: it neatly integrates a wiki, a simple bug tracker and a Subversion repository browser with clean markup, a nice default design and a learning curve for new users that can be measured in minutes. No wonder it's started to show up all over the place.

The software has only two disadvantages. The first is that it's hideously difficult to install thanks to the myriad of dependencies (although apt-get on Debian or Fink on OS X go a good way towards lessening the blow). The second is that if you're using BDB and you're not careful it can corrupt your repository. Thankfully this corruption isn't permanent (you can revert it with svnadmin recover) but it's still very, very annoying.

Happily, a solution exists to problem two. Subversion 1.1 introduces a new storage mechanism called FSFS. The advantages and disadvantages are discussed in this advocacy document, but the key advantages as far as Trac is concerned appear to be "Write access not required for read operations", "Little or no need for recovery" and "No umask issues". Switching a BDB Subversion repository over to FSFS is short and painless, and provided you have up-to-date Subversion/Python bindings Trac will be able to access the new repository without needing any other changes at all. I've made the switch on my local machine and the weird corruption and performance issues I was having have vanished without a trace.

For the record, the trick to successfully installing Trac with Fink is to make sure it's pulling from the "unstable" tree. That will give you Subversion 1.1 (with FSFS support) and allow you to install the trac-py23 package which should sort everything out for you. The instructions on the Trac wiki cover the essentials, but be sure not to miss the troubleshooting note about using a custom shell script in place of the regular trac.cgi.

Command line blacklisting

Simon Willison — 2004-09-09T05:59:46-00:00

Just over a year ago, I started blacklisting domain names from links featured in comment spam. My idea then was that these blacklists could become a shared resource: people would publish their own blacklist and subscribe to those of people they trust, thus making it much harder for spammers to operate. While the sheer volume of spam domains meant that the technique was much less useful than I originally anticipated, I've continued to maintain my blacklist ever since as a preventative measure against repeat spammers.

I have a confession to make: all of my blog administration (with the exception of adding entries and blogmarks) is performed using phpMyAdmin. The trouble with writing your own software is that it's very easy to skimp on the backend tools, since you're the only person who will ever see them. Incidentally, this is the main reason I plan to switch to WordPress just as soon as I find the inspiration to write the necessary import scripts. Comments are deleted in phpMyAdmin, and domains are blacklisted by manually editing the blacklist.txt file via FTP.

This has been really bugging me, especially since I have so little other use for FTP that my only installed client is an unregistered version of Transmit (closes after ten minutes, won't save passwords along with account details). I've been muddling along with that for longer than I care to admit, but today I decided to take 10 minutes out to solve the problem once and for all. I could have put together a web interface for adding new domains but I wasn't really in the mood, so I decided to put time spent reading The Art of Unix Programming to good use and knock out a simple command line application.

The result (minus my login details) can be found here. Sample usage: ./blacklist.py www.domain.org www.domain2.com. It follows the Unix ideal of being the simplest-thing-that-could-possibly-work, and ended up taking longer to write than I expected thanks mainly to the craziness of Python's ftplib. I've seen complaints about this before, and it thoroughly deserves its bad reputation.

Here's one example: retrlines is the method used to retrieve ascii text from the server. Bizzarely, it doesn't actually return the text receieved; instead, it expects you to provide it with a callback function that will be fed each line in turn, minus the newline. Sounds like a job for StringIO, but StringIO objects don'y have a writeline method (required to add the newline back on). I ended up writing my own extension of the StringIO2 class and adding a writeline method just to preserve the newlines returned from the server!

Strange APIs aside, I'm pretty pleased with the final result. It follows a bunch of Unix design patterns (and skips others such as those related to configuration, but I'm not overly bothered about those) including the following:

A usage note is displayed if no arguments are provided.
Multiple domains can be blacklisted at once, by providing them as multiple command line arguments.
Domains that are already in the blacklist are skipped, and a message is written to standard error.
If the script suceeds, it doesn't say anything at all.

It also uses the common Python idiom of wrapping the principle logic in a function and then calling that from a block that runs only if the file is executed directly (the __name__ == '__main__' idiom) so that other Python code can import the module and reuse its functionality if required.

There's plenty of room for improvement: being able to pipe a list of domains in via standard input would be nice, and hard coding the (unencrypted) username and password is sloppy (as is expecting the blacklist.txt file to live in the FTP home directory). Even better, with SSH access the whole thing could be replaced with an infinitely more secure one-liner: echo www.domain-to-ban.org | ssh username@server "cat - >> blacklist.txt". I'm happy though: an irritating task has become much less irritating and I have some example code to fall back on next time I need to get mucky with ftplib.

Backporting from Python 2.3 to Python 2.2

Simon Willison — 2004-06-09T04:58:22-00:00

We have a home-grown templating system at work, which I intend to dedicate an entry to some time in the future. We originally wrote it in Python 2.2, but upgraded to Python 2.3 a while ago and have since been evolving our code in that environment. Today I found a need to load the most recent version of our templating system on to a small, long neglected application that had been running the original version ever since it had enough features to be usable.

Unfortunately, this application was running on a server that only had Python 2.2. Installing Python 2.3 would have been somewhat more painful here than on other servers we run for reasons I won't go in to, so I decided to have a go at getting our current code to run under the older Python version.

In the end, I only had to make three minor changes, all at the top of the file in question.

I added from __future__ import generators as the very first line of the file. We use generators (with the yield statement) in a few places - this feature was only properly added in Python 2.3, but was made available in Python 2.2 as a "future enhancement" through the aforementioned obscure import.
I added True, False = 1, 0 on the next line down. Surprisingly, Python 2.2 had no support for a boolean type and instead used a test for non-zero. The above line defines constants that behave enough like Python 2.3's True and False to avoid any problems.
I defined an enumerate function, which was introduced for real in Python 2.3. Here's the code I used:
```
def enumerate(obj):
    for i, item in zip(range(len(obj)), obj):
        yield i, item 
```

All in all it only took around ten minutes to put the above together, after which the script worked just fine. It was interesting to see how our code had grown to rely on Python 2.3 features without us realising it.

Update: Check this entry's comments for improvements to the above code snippets.

It's only going to get worse

Simon Willison — 2004-03-27T00:32:59-00:00

This analysis of the spread of the witty worm is fascinating for a whole bunch of different reasons.

Firstly, the analysis was made possible by USCD's Network Telescope, a network monitoring system on a massive scale which takes advantage of the fact that IP arranges were handed out like candy back when the 'net was in its infancy. USCD controls a huge chunk of all potential IPv4 addresses, and their network telescope tracks data sent to 1/256th of all IPv4 traffic. Since most worms target random IP addresses this makes the telescope a unique tool in analysing the spread of hostile code in the wild.

Next, Witty Worm was no ordinary worm. It targeted an exploit in ISS firewall products, which include the popular BlackICE product targeted at home users; this means the worm was actively attacking people who had made an effort to secure their machines! It also carried a destructive payload - a rarity for worms in the wild. Additionally, the exploit it used had only been publically announced the day before. It's possible the authors new of the vulnerability in advance, but it's far more likely they had already written the payload and were just waiting for a new vulnerability to use as the carrier.

From reading the report, it seems that the worm managed to infect virtually every one of its potential targets that were connected to the internet. This critical point is what makes the worm so interesting, because it destroys the idea that non-Windows users are made more secure by their relatively lesser numbers. If a worm came out with a similar methodology to Witty Worm but that targeted Linux, OS X or even something with a truly tiny statistical footprint like BeOS it could still achieve almost total infection of its chosen target audience.

The worm also appears to have used a number of techniques that had previously been hypothesized by the security communit, such as spreading from a number of pre-infected hosts.

If a worm can spread this fast, with this little notice, and infect almost all of the vulnerable population, we're in a pretty precarious state.

Related reading: The Peon's Guide to Secure System Development, Slashdot's thread on the Witty Worm analysis (some of the +5 comments are pretty good).

In praise of Apache documentation

Simon Willison — 2004-03-02T05:11:41-00:00

I spent much of today upgrading a distinctly hairy Apache 1.3 server to Apache 2.0 as part of a routine OS upgrade. It certainly wasn't plain sailing - I still have a few crinkles to iron out - but that's more down to the weirdness of the existing configuration than any problems with Apache 2.

Apache 2 is a beautifully designed piece of software. The documentation is superb - the migration guide proved invaluable but the real gem was the directive quick reference. Armed with the quick reference and Firefox's Type Ahead Find the previously inpenetrable httpd.conf file becomes a living tutorial on the wild and wonderful ways of Apache configuration. A wise old sysadmin once told me that the best way of learning Linux is to ls /bin and run man <command> for every command in there - then do the same thing for /sbin, then /usr/bin and so on until you run out of things to read. The same appears to be true of Apache configuration directives.

Bizex

Simon Willison — 2004-02-27T23:30:28-00:00

I'm going to try not to turn this in to a blog about Windows security exploits but this one is genuinely interesting in that it actively tries to steal financial information and important passwords. Bizex spreads itself by spamming messages over ICQ advising the recipient to visit a specific URL. When they visit it, Internet Explorer exploits are used to download and execute the main payload which then infects their ICQ program and uses it to message their contacts. The worm also scans their hard drive for information relating to a number of well known financial services which it then uploads to a server via FTP, and it apparently snoops on their browser for any passwords travelling over HTTPS connections as well.

It seems that the sole purpose of this worm was to steal a bunch of cash quickly, and it looks very likely to succeed. The servers used to spread the worm have since been taken offline but you can be sure the person or people behind the worm were smart enough to cover their tracks.

More on Bizex can be found here and here; Thors Larholm's analysis on BugTraq is particularly insightful.

Novel security measures

Simon Willison — 2004-02-25T23:40:06-00:00

An article on SecurityFocus led me to this site about Port Knocking. Port Knocking is an interesting security technique in which a box sits online with no ports open to connections and awaits a specific sequence of connection attempts. A user wishing to connect to the box must first attempt to initiate connections to ports in a specific, secret order. Once they do, the box starts up the required service (such as an SSH daemon) on a designated port and allows the user to connect properly.

It's a pretty neat trick, and one that may well start showing up in backdoors and trojans in the future. It reminds me of a couple of other novel firewall related tricks: invisible firewalls and firewalls that are effectively turned off.

"I'm Brian and so's my wife"

Simon Willison — 2004-02-24T01:26:57-00:00

I'm subscribed to a whole bunch of mailing lists, mostly as a lurker as I have a hard enough time just keeping up with some of them. One of those lists is Bugtraq, which is pretty much required reading for anyone with sysadmin responsibilities for a server connected to the public internet. Bugtraq is the central hub of the "public disclosure" security community and is actually surprisingly low traffic with only twenty or so messages a day. It's fascinating to watch the latest exploits for all manner of popular software packages tick by on an hourly basis.

Last week, someone posted to the list asking if anyone knew of a contact address for the security team at Bank of America. Today, they posted a follow-up which included the following gem:

I'd also like to thank the 0-day social engineers for their variety of approaches used to attempt to gain access to this exploit. We received responses ranging from fraudulent "Bank of America" employees to phone calls from people claiming to be from Bank of America's IT Security. (One caller claimed to be from Bank of America's IT Security but didn't know what PGP is and then said he couldn't give his PGP key due to security restrictions. And when we asked him to provide information so we could verify the contact, he said he would call back but never did. To this caller: Yes, your social engineering failed and your caller-id spoofing was almost perfect. Emphasis on "almost".)

For some reason, I'm reminded of a classic scene from Monty Python's Life of Brian.

Defending web applications against dictionary attacks

Simon Willison — 2004-01-22T01:02:07-00:00

Over at Reflective Surface, Ronaldo M. Ferraz discusses the usability of an authentication system that locks down an account for a certain period of time after three failed login attempts. Ronaldo sees this as a trade off between usability and security, but I see it more as an added security issue in that it allows malicious third parties to lock other user's accounts armed only with their username.

The problem then is how best to defend web applications against brute force password guessing attacks without enabling denial of service attacks at the same time. The largest risk is from automated scripts that try every possible password until they get in. Identifying these attacks should be trivial - a real user could potentially fail a dozen or so times, but would be unlikely to try hundreds of combinations in quick succession. Assuming a malicious cracking attempt has been identified, what steps should a system take to foil the attack while still allowing the real user to access the site?

I can think of a few options, none of which seem like the ideal solution:

Ban login requests from the attacker's IP address. This introduces the usual problems with IP banning, namely the risk of banning a whole bunch of people indiscriminately but leaving the attacker free to skip the ban using open web proxies.
Lock the user's account and email them a warning of the attack and a special key needed to unlock the account again. This relies on the user having access to their email account when they next have a need to access the system. It also assumes that the user's email account is secure, but since both the user's password and the secret unlocking key will be required to access the system email security is of less importance (the user's password is not sent with the unlock key).
Send an automated alert to a system administrator so they can analyze the situation in real time and take any necessary action. This relies on administrators being available 24/7 - hardly a safe assumption for most systems.
After a certain number of failed attempts, challenge the user to "prove their humanity" with one of those obscured-text-as-image things. This comes with accessibility issues which have as yet been unresolved.

If anyone has any better solutions, please leave a comment.

Installing psycopg on Red Hat 9

Simon Willison — 2003-12-31T04:43:03-00:00

Adrian Holovaty and I spent some time today figuring out how to get the psycopg Postgres module to install on Red Hat 9. It took a while, but eventually we tweaked the spec file and used it to compile our own RPM. I've posted our modified spec file to the psycopg mailing list. More for my own record than anything else, the arcane incantations needed to create the RPM went roughly as follows:

Copy the modified psycopg.spec file in to /usr/src/redhat/SPECS/
Copy psycopg-1.1.11.tar.gz in to /usr/src/redhat/SOURCES
In the SPECS directory, run rpmbuild -ba psycopg.spec
Install the resulting RPM, which materialised in /usr/src/redhat/RPMS

The above was recreated by digging through my bash history - hopefully I haven't missed a step. The technique appears to work for Fedora as well.

Open Mosix

Simon Willison — 2003-12-19T23:44:04-00:00

I can't remember how I stumbled across it, but Open Mosix looks like a really interesting project. It's a Linux kernel extension that makes creating a Linux cluster is as simple as installing a kernel module on a number of machines and supplying each one with a shared config file.

Once the cluster is set up, any of the machines on the network has the ability to farm long running processes out to a different box. The clustering only kicks in when a process is running in the background takes more than a few seconds to execute. Once that happens, Open Mosix checks the load averages on the machines in the cluster and, if a better host is found, migrates the process over to the other machine. The whole process is completely transparent; as far as the end user is concerned the processes they run just keep on running until they terminate.

Since clustering only kicks off for longer running processes this woudn't be much use for something like a web server farm, but could be ideal for tasks such as compilation or rendering where individual processes perform computationally intensive work for a long period of time.

Even more fascinating is ClusterKnoppix, a modified Knoppix distro that uses an Open Mosix enabled kernel. Burn a bunch of CDs, boot some standard networked PCs with them and you've got an instant cluster.

I've been doing some pretty intense log crunching today on a year's worth of daily web server logs, each one between 15 and 20 MB in size. Since most of the work is in running a regular expression on each line of each file it's likely a cluster would have speeded the whole thing up considerably. Unfortunately, I doubt my co-workers would have been overjoyed with me turning their machines in to cluster nodes for the day.

Repartitioning with Knoppix

Simon Willison — 2003-11-30T00:36:02-00:00

I've been long bemoaning the fact that if you want to repartition your hard drive to install Linux as a dual boot with an existing Windows system the most frequently recommended method is to buy a copy of Partion Magic. You would have thought the open source software world would have provided a free alternative by now.

Via Andy Todd, it turns out that they have. GNU Parted is a repartioning tool for Linux. QtParted wraps it in a GUI with a Partition Magic style interface. And the awesome Knoppix comes with QtParted included on the disk. So instead of shelling out for an expensive package that you are unlikely to ever use more than once, you can download and burn a Knoppix CD, boot in to Linux and repartition from there. I'll be trying this out for real on Monday, and I'll report back with the results when I do.

As an aside, has anyone ever found a web page that lists all of the software included on the Knoppix CD?

Update: Closer inspection reveals that Parted can't resize NTFS. Thankfully, ntfsresize can - and ntfsresize is integrated in to QtParted. Magic.

Why run Windows on an ATM?

Simon Willison — 2003-11-26T05:16:33-00:00

So you're writing the software for an ATM. It needs to display something pretty on the screen, control the hardware that serves out the money and talk securely to your central servers. It also needs to be stable, secure, reliable and allow remote administration. Why on earth would you choose Windows as the operating system?

Check out this article on The Register: Nachi worm infected Diebold ATMs. This just beggars belief. How a Windows worm spread on to a network with ATMs connected to it is beyond me - even if you take in to account employee laptops plugged in behind the firewall it's still incredible that the ATMs weren't on their own separate secure network.

Here's the best bit:

Billett defended the company's patching process, which he said involves testing each new bug fix, and deploying at a wide variety of institutions with a mix of network architectures. "A lot of those machines actually have to be visited by a service technician" to be patched, said Billett. "Our experience in the past is we are able to turn those around in one or two days."

What do you have to do to patch these things, plug in a keyboard and mouse?

Contribute / ProFTPd problem solved

Simon Willison — 2003-11-19T23:05:30-00:00

After further analysis of the Contribute problem described earlier, we discovered that Contribute was opening a new FTP connection every time we clicked a link within the application even before we had hit the "edit page" button to fire up the editing mode. Switching the connection over to use SFTP instead of FTP had the same problem, with a secure connection being opened for each link we clicked instead. The connections remained open until we shut down Contribute.

My hunch is that this could be an obscure bug that only surfaces when Contribute is used with ProFTPd 1.2.9. At any rate, we've solved the problem by setting the MaxClientsPerUser directive in the ProFTPd configuration file. Contribute doesn't seem to mind in the slightest.

Contribute hammering FTP servers?

Simon Willison — 2003-11-17T23:34:25-00:00

We're having a problem at work with Macromedia Contribute. We host sites for a number of local companies, and one of them wants to use Contribute to update its site. The problem is that whenever Contribute tries to connect to our FTP server, it opens up 30 simultaneous connections, effectively running a denial of service that prevents other clients from logging in during peak times. I've searched the 'net and haven't found any references to this problem; does anyone know anything about the issue? We're running ProFTPD 1.2.9 and the client is using Macromedia Contribute 2.