Online Issues

Freeing the postcode

Simon Willison — 2006-11-17T17:29:59-00:00

UK postcodes have some interesting characteristics: a full six character post code identifies an average of around 14 house holds, and postcodes are mainly hierarchical - W1W will always be contained within W1 for example. They're useful for a huge range of interesting things.

The problem is that the postcode database (of nearly 1.8 million postcodes) is owned by the Royal Mail and licensed at a not inconsiderable fee of between £150 and £9,000 per year.

Free the postcode was set up a while ago to try to remedy this situation, by asking people to enter their postcode along with the latitude/longitude coordinates collected from a GPS. Having people enter coordinates from online mapping services is no good as EU database law may see that as a derivative work. It's had some success, but the GPS requirement has seriously stunted its growth.

Then a few weeks ago, npemap.org.uk launched. It's an interface for browsing scans of out-of-copyright maps from the 1950s (credits at the bottom of the FAQ). The site asks people to enter post codes based on that old mapping data, which can then be placed in the public domain.

If you haven't already done so, you should go and add any postcodes that you know about now. It takes no time at all, and is especially important if you live in one of the 230 districts for which no data has yet been collected.

You can grab the data they've already collected from here. There's a really cool interactive visualisation of their data here, based on previous work by Chris Lightfoot using the commercially licensed postcode database.

Social engineering and Orange

Simon Willison — 2005-11-09T20:52:12-00:00

I had a call on my mobile earlier today from a lady claiming to be from Orange (my phone service provider) who told me that my contract was about to expire. She asked me for my password.

Alarm bells instantly went off in my head, so I told her (truthfully as it happens) that I didn't know my password. Then she asked for my postcode instead.

At this point I was pretty sure this was a social engineering attack, so I started to quiz her about why she needed the information. She said it was for a "security check". I told her I was uncomfortable giving out information like this to a cold caller over the phone and she said it was nothing to worry about because it was all covered by "the data protection act".

I said that I would rather conduct my business in an Orange shop, and she told me that she would have to put a mark on my record that I had failed a security check. I interpreted this as a threat, which convinced me that the call was an attempted con. I asked for her name and ended the call.

I e-mailed Orange customer support via their website with details of the call and the number it came from (07973 100 194, which looked like a mobile number to me and had further fuelled my suspicions). I just received their reply - the call really was from them!

Banks and other online services have learnt to repeatedly tell their customers that they will never contact them and ask for their password. Orange are leaving themselves wide open to social engineering attacks. This incredible lack of attention to basic security has given me serious second thoughts about trusting them with my business at all.

Fighting RFCs with RFCs

Simon Willison — 2005-05-06T20:39:45-00:00

Google's recently released Web Accelerator apparently has some scary side-effects. It's been spotted pre-loading links in password-protected applications, which can amount to clicking on every "delete this" link - bypassing even the JavaScript prompt you carefully added to give people the chance to think twice.

"Aah," I hear you cry, "but RFC 2616 clearly states that you shouldn't perform state changing operations with a GET or HEAD method!"

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval.

I'll see your RFC 2616 and raise you an RFC 2119:

SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

Hiding your dangerous delete links behind an authentication scheme is a perfectly acceptable compromise. Web Accelerator is B.A.D.

Update: Be sure to read the excellent discussion brewing in the comments. Hiding behind authentication may not be as acceptable a compromise as I had first thought.

Update 2: If you haven't been following the comments, I've had a change of heart. Even in the absence of Web Accelerator, hiding behind authentication leaves your application open to some very nasty security vulnerabilities (malicious pages can piggy-back your session and cause havoc making dangerous GET calls). I still think the RFC language covers people who thought long and hard before implementing a dangerous GET, but if you haven't thought about security and accelerating caching proxies such as Web Accelerator you haven't been thinking hard enough.

Update 3: So, it turns out using POST is no defence at all against CSRF attacks. I've been learning a whole bunch of interesting stuff this evening.

Giving away the index

Simon Willison — 2005-05-04T01:16:45-00:00

My final year project is due in two weeks, and I'm going to be running on silent for most of them. I have, however, upgraded to Tiger and playing with Spotlight has given me plenty to think about.

Giving away the index

The great benefit of having an electronic version of a book you own in dead-tree format to hand is that you can search it. Publishers generally don't hand out free digital copies because, well, they want you to buy the books, not freely distribute electronic copies.

The thing is, you don't need a digital copy of a book to be able to search it; you just need a full-text index of it (if you don't understand what this means, go and read Tim Bray's series On Search). An index isn't enough to reconstruct the book, but it is enough to answer questions like "on what pages of Eric Meyer on CSS are float layouts discussed?"

Imagine if technical publishers made binary full-text index files of their titles available for download, for free in some kind of open standard format. Readers could query them using Spotlight or similar technologies, and gain the ability to search the titles they own all without needing to rely on centralised, artificially limited services such as Amazon's Search Inside the Book.

O'Reilly, I'm looking at you.

Full-text phishing

On a darker note, one thing about Spotlight that has given me pause is the immense ease with which it can uncover passwords saved amongst my email. Lost password reminders, new account details, invitations to sign up for services - they're all hidden away in my mail archive. Spotlight makes it trivial to dig them back up again, and offers the APIs for applications to do so as well. Combine this with a piece of spyware / some trojan horse and you've got the ultimate vector for phishing attacks.

This problem isn't limited to Macs either; Google and MSN's Desktop Search engines could be used for much the same purpose, and full-text search is bound to end up built in to Windows sooner or later. For the moment, the safest thing to do is either delete those pesky emails or move them to a folder that is excluded from Spotlight's index. Somehow I doubt many people will think to take such precautions.

And with that off my chest, it's time to get back to my dissertation.

rel="nofollow"

Simon Willison — 2005-01-17T01:39:26-00:00

Reading between the lines (which in this case isn't particularly hard), this and this (don't forget to view source) suggest that Google are soon to announce that they won't be calculating PageRank for links with a rel="nofollow" attribute. Finally, an official way of fighting the economics of comment spam by denying PageRank on user-submitted link content. Sam Ruby points to Mark Pilgrim's prediction that spammers won't care - they'll spam anyway, on the offchance that they hit somewhere undefended. I'm optimistic - if the major weblog (and wiki) vendors get behind this one it could help stem the tide.

As an aside, I have exams starting in a week and plenty to revise, so I'll probably be on hiatus until the end of the month.

Some notes on Wikipedia

Simon Willison — 2004-12-23T00:57:42-00:00

I've been driving myself crazy with coursework over the past couple of weeks, and since it's always good to have something to take your mind off things I've also been spending a fair amount of time lurking around the beautiful Wikipedia. Here are a few things about Wikipedia you may have missed:

It's not just Wikipedia any more; there's also Wiktionary (a multi-lingual dictionary), Wikibooks (developing open content books on various topics), Wikiquote (quotations), Wikisource (a repository of public domain source texts), Wikispecies (a biological species database), Wikicommons (free images and other media) and Wikinews (a new Wikipedia-style news site). Not to mention the huge numbers of projects in other languages.
You can view live stat graphs of the Wikipedia squid cache servers and see an overview of the status of all Wikipedia servers.
Last year's drive for donations was mostly spent on new hardware, and a detailed list of hardware orders is available.
Wikipedia's awesome TeX engine for presenting mathematical formulae may soon be expanded to support rendering of musical scores, SVG graphics, chemical formulae and more, thanks to the brilliant Wikitex module for MediaWiki.
Wikisource has a bunch of stories by H. P. Lovecraft!
Wikipedia's Periodic table links to detailed descriptions of every single element.
Live recent changes feed is a page that shows edits to Wikipedia in real time. It works by keeping the HTTP connection to your browser open and sending updates packaged as JavaScript calls (I think this is the same trick used by CGI:IRC).
The channel #enrc.wikipedia on irc.freenode.net carries a bot-produced live feed of recent changes to Wikipedia. Edits occur so frequently that the bot had to be split in to five to avoid being flooded off the channel!
Wikipedia has a huge vandalism problem, but malicious edits are cleared up so fast that you'd be hard pressed to spot it.
The Wikimedia foundation has an attractive quarterly newsletter, the Wikimedia Quarto. September's issue includes an interview with Ward Cunningham.
Wikipedia provides a great way to sharpen your language skills; not only does Wikibooks have guides to teaching yourself French and German (among others) but the multi-lingual versions of Wikipedia provide excellent practise in reading comprehension. Compare the English and French entries on Bath, for example.
The Wikimedia foundation recently received a small grant to develop a series of children's books.

The deeper I dig in to Wikipedia, the more amazed I become. I see it as more than just a collaborative encyclopaedia; it's a testament to humanity's ability to work together for the greater good. I guess you could say I'm in WikiLove :)

Update: Fixed links, thanks to corrections posted in the comments. If this entry had been a wiki page, people could have fixed them themselves...

No EU Software Patents

Simon Willison — 2004-11-23T14:26:12-00:00

Munich, Germany (23 November 2004). The three most famous European authors of open-source software have issued an appeal against software patents on NoSoftwarePatents.com. Linus Torvalds (Linux), Michael Widenius (MySQL) and Rasmus Lerdorf (PHP) urge the EU Council, which will convene later in the week, not to adopt a draft directive on software patents that they consider "deceptive, dangerous, and democratically illegitimate". They also call on the Internet community to express solidarity by placing NoSoftwarePatents.com links and banners on many Web sites.

It would be nice if someone with some serious design credentials would knock up some more aesthetically pleasing banners.

The Register hit by XSS

Simon Willison — 2004-11-22T08:32:08-00:00

Here's a nasty one: popular tech news site The Register was hit on Saturday by the Bofra exploit, a nasty worm which uses an iframe vulnerability in (you guessed it) Internet Explorer to install nasty things on the victim's PC. Where it gets interesting is that the attack wasn't against the Register themselves; it came through their third party ad serving company, Falk AG.

This is a classic example of a cross site scripting attack, in which malicious client-side code (usually JavaScript) is uwittingly served up by an otherwise innocent site. Usually, cross site scripting is caused by a badly written server-side application failing to properly escape data sent in a query string before displaying it on a page. This allows attackers to create links which, when followed, steal cookies or cause other nasty effects for the user following the link. Attacks on third parties with scripts served up on a target website's pages (ad serving companies are a classic example) are less common but much more damaging as the malicious code involved will be distributed to everyone who visits that site, whether they click on a hostile link or not.

This problem isn't restricted to ad servers; any service where web pages point to a JavaScript file hosted on an external site are potentially at risk should the external site be compromised by crackers or abused by its legitimate owner.

An aside: users of alternative browsers (Get Firefox!), as well as those who had upgraded to Windows XP Service Pack 2, were unaffected.

Improving online credibility

Simon Willison — 2004-07-29T05:45:37-00:00

If you've browsed Amazon's product reviews recently you may have noticed an interesting new feature: Badges, little icons displayed below certain people's names. This isn't a new idea by any means - many online communities use special icons as rewards for members who make valuable contributions (SitePoint is a good example). What's interesting about Amazon's badges is that one of them is "Real Name". Amazon's Real Names FAQ explains the badge, and includes the following:

Why is Amazon.com encouraging the use of Real Names?

In general, we believe that a community in which people use their Real Names will ultimately have higher quality content, since an author willing to sign his or her real-world name on a piece of content is essentially saying "With my real-world identity, I stand by what I have written here."

Real names certainly add credibility to online discussion: I for one find it much easier to trust information if the author appears to have signed their real name to it. The challenge is verifying that the name is accurate, and Amazon's solution is so simple-but-smart that I kicked myself: they match the name against the user's credit card details. Genius.

I wouldn't be surprised to see this trick taking on a key role in the field of online identity management - provided Amazon's patent lawyers don't get there first.

Jimmy Wales on battling wiki spam

Simon Willison — 2004-07-29T02:38:45-00:00

Jimmy Wales of Wikipedia was interviewed recently by the Slashdot community. One of the questions regarded protecting Wikis from spammers:

Sure, I think it's pretty simple to solve problems like that. One of the first tricks I would try is to parse the wiki text that someone inputs to see if it contains an external link. If so, then only in those cases, require an answer to a captcha.

Second step, keep editing wide open for everyone, but restrict the ability to post external links to people who are trusted by that community. Make it really easy for trusted users to extend the zone of trust, because you want to encourage participation.

Basically what I think works in a wikis is to trust people to do the right thing, and trust them as much as you can possibly stand it, until it hurts your head and makes you scared for what they're going to break. Because that is what works.

People are not fundamentally bad. It only takes the smallest of correctives to take care of that tiny minority that wants to disrupt the community.

I'm glad to say that so far the css-discuss wiki spam problem has been effectively tackled by a hard working group of dedicated spam fighters. Helping out is as easy as signing up for the recent changes RSS feed.

Site-specific extensions

Simon Willison — 2004-07-20T05:46:23-00:00

I've been thinking about per-site user stylesheets for a while now, but my colleague Adrian has gone one better: his All Music Guide Corrector extension for Firefox fixes their horrible JavaScript links, hides the useless Flash navigation and improves their unpopular "read more" links, causing them to load content on the current page rather than navigating to a new page entirely.

I believe that extensions like this have a significant role to play. Bugzilla's Tech Evangelism project is overflowing with badly designed sites that through ignorance or choice refuse to work with standards compliant browsers, many of which have available patches just waiting to be implemented. Per-site extensions at least allow users to choose to fix the problem locally and route around the damage - and their use should send a powerful message to the sites in question.

This kind of extension also introduces some interesting questions. How will site owners react to their users tweaking their websites in this way? Is it ethical to modify a site without the user's permission to improve functionality? What about to block irritating ads?

Pop-up blockers have only scratched the surface. Let's see some innovation.

News site registration

Simon Willison — 2004-07-16T16:16:09-00:00

The single hottest topic in the online news industry at the moment is that of required registration. A number of large news sites (the New York Times, the Washington Post, the Chicago Tribune) have moved to this model, and many local newspapers are following suit.

If you haven't seen BugMeNot, go and check it out now. It's a simple service for sharing free news site accounts, and it's started to upset some people in the news industry. A post to the online-news mailing list inquiring about possible legal action against the site prompted me to reply with the following:

The flaw here is not with BugMeNot - it's with the entire concept of user registration in its present form. The reason BugMeNot works is that there is absolutely no value to an end user in keeping their account to themselves. If you want to stop people from sharing their accounts, give them an incentive not to. This is not a difficult thing to do - I have a large number of accounts on different community sites which are used to contribute to discussions and manage my personal information. I would never dream of sharing those accounts with others - it would allow other people to impersonate me and damage my reputation. An account that only allows me to read content (a one-way interaction) is of no value to me, so why not share the account with others?

BugMeNot is not a new idea by any stretch of the imagination: shared accounts have existed for as long as sites have required registration for spurious reasons. For as long as I can remember, members of the MetaFilter community have worked together to set up username/password combinations of metafilter/metafilter on sites that require registration to bypass the irritation of setting up yet another account.

If you want to fight BugMeNot, the solution is to monitor the site and ban any accounts for your own site that appear there - but that's just fighting the symptoms. The core problem is the whole idea of registration itself: it's anti-web, anti-user, it doesn't scale and it's a sign of extreme short term thinking. Imagine if every site on the web required registration - no one would use it!

As a web user, I see registration as nothing more than an unnecessary irritation. Before BugMeNot I would simply hit "Back" whenever I saw a registration screen; now I use it to carry on through to the articles and accompanying ads. As a heavy web user who buys online almost as frequently as offline I'm exactly the kind of demographic sites should be trying to attract.

Reading the above a few days later, I think it still accurately represents my thoughts on the free registration model.

Adrian has also posted his thoughts on registration, which run along very similar lines to mine.

For a great example of the mentality behind registration, check out this spiel from the Toronto Star (via Craig Saila):

Our main goal of asking you to become a registered member of thestar.com is to improve and enhance your online experience with us. Registration is an important piece of our long-term strategy in building a valuable audience for our advertisers and helping us in setting the priorities for future site development and enhancements.

[...]

By asking you to share some information with us we are able to increase the value of our site to advertisers, who help support the cost of producing one of Canada's top news sites, by offering them the ability to target their advertising messages based on the information you provide.

And that's the problem right there: as a user, the value proposition of having more targetted ads thrown at me just isn't a good enough incentive for me to jump through their hoops.

Domain Keys Explained

Simon Willison — 2004-05-19T02:04:40-00:00

Via Jeremy Zawodny,, Yahoo's Anti-Spam Resource Center have published an explanation of their proposed Domain Keys spam fighting technique. At first glance it looks very promising. There's no centralised authority, no requirements for changes to existing protocols and the central concept is extremely easy to understand. Essentially, mail servers generate a public/private key pair and sign outgoing messages with the private key, while publishing the public key as part of their DNS record. Because only they can publish to their public key in this way the signature can be used to confirm that the sender of the email has not been spoofed. The presence or lack of a signature can be used as part of the process of identifying spam.

The FAQ covers all the bases I could think of, and explains how Domain Keys can help fight phishing attacks as well.

Democratised Namespaces

Simon Willison — 2004-03-21T19:22:20-00:00

The New York Times: Get out of my Namespace (via Diego Doval) - a well-researched look at the huge problems (and frivolous lawsuits) being generated by the global quest for ownership of unique names.

Unsurprisingly, the fundamentally broken domain name system is a recurrent theme. I hadn't heard about bodacious-tatas.com vs the Tata Group of India but it seems pretty indicative of the whole crazy situation. Fans of TypeKey take note: this is what you get when you build a centralised system in a decentralised world.

An interesting side effect of all of this is the role of Google's PageRank algorithm in creating a democratised namespace. FireFox and Safari, the two browsers I use on a daily basis, both include a "search Google" box to the right of the standard URL bar. If I'm going to a site about a certain topic or person for the first time I will almost always enter the relevant terms in that box rather than trying to guess a domain. PageRank usually ensures that the first returned result is the definitive resource on the entered term - a democratic process, achieved by general consensus of the billions of pages that make up the web.

Of course, this also strengthens the whole URLs as identity idea, where you need a well ranked web presence (generally a weblog) to confirm your place in internet society. Owning your name on Google becomes more important than owning your name with ICANN.

Grey Tuesday

Simon Willison — 2004-02-24T18:25:18-00:00

I'm supporting Grey Tuesday.