Blogging

New weblog, new location

Simon Willison — 2006-12-15T14:47:06-00:00

I've just launched my new weblog over at simonwillison.net. I will no longer be updating simon.incutio.com, and will be putting redirects in place for old content over the next few days. You can read more about the new site (powered by Django, naturally) in this entry.

Blogmarks on del.icio.us

Simon Willison — 2004-12-02T00:13:22-00:00

I'm horribly ill again: having defeated the mumps I now seem to have come down with some kind of 'flu thing. Lovely. In between whinging about my state of health and watching episodes of Frasier I've been playing with del.icio.us as part of my research in to web annotation. The connection between the two isn't particularly strong but it's clear that something very exciting is happening over there.

This evening I wrote a script to import my blogmarks in to del.icio.us. I don't plan to replace them with a feed from the site for a couple of reasons: firstly, I like to keep my data somewhere I control and secondly, del.icio.us doesn't support my "via" fields. I will however be adding tag support to my blogmarks and some kind of functionality to ensure that anything I post to them is added to del.icio.us as well.

The problem I have now is that I've added nearly 1200 untagged links to del.icio.us, and anyone who has played with the service for more than a few minutes knows that it's the tags that make it so much fun. Does anyone know of a good tool for bulk-tagging of items in del.icio.us? I've tried Cocoal.icio.us but unfortunately it only lets you assign tags to items one at a time; what I really want to do is run full text searches against my del.icio.us links and mass-apply tags to dozens of items at a time. If one doesn't exist I can always knock up a custom tool with the lovely API but I'd rather not duplicate the effort if I don't have to.

Keeping up appearances

Simon Willison — 2004-10-29T11:41:39-00:00

Wow, I think this is the longest gap in my blogging since I started! I wish I could say I've been enjoying the sunshine or taking up a new hobby, but the truth is that the weather's been horrible and I've just been run off my feet readjusting to life in England and at University.

One thing I haven't been doing is reading blogs. My aggregator (currently the excellent NewsFire, although NetNewsWire 2.0 could easily steal my affections) has been lying dormant, and aside from occasionally checking a few sites (congratulations Matt on the new job!) I've been reading more academic papers than weblogs. My final year project is tentatively titled "Collaborative annotation of web resources" and looks set to take up a big chunk of my time over the next six months or so.

I certainly miss the information flood of blogging, but there's something very liberating about dipping in every now and then rather than following several hundred constant streams of consciousness, Scoble style. I guess you could say I've been re-evaluating my priorities. I'll certainly be scaling back some commitments in the near future, though which ones and by how much I have yet to decide.

1000th Blogmark

Simon Willison — 2004-08-26T00:30:54-00:00

I just posted my 1000th blogmark. I can't emphasize enough how much of an impact this 15 minute hack has had on both my browsing and my blogging habits. While I still tend to leave browser windows open for days at a time, I now at least have a procedure for getting rid of the ones that still interest me. More importantly, having blogmarks has eliminated the temptation to write a full blog entry (with quotation) just to share a link. This has dramatically reduced my posting rate, but has meant that when I do post an entry I usually have something moderately interesting to say.

To celebrate this personal milestone, I've linked up the rudimentary LIKE query search engine I've been using for a while on the blogmarks index page. My long term aim is still to integrate them with my main content and add comments in the style of photomatt, but that would require more time spent hacking on my blogging system (or switching to WordPress) than I have to spend right now.

A snarky note from the administrator

Simon Willison — 2004-08-23T14:59:35-00:00

No, you can't have a Gmail invite. No, I won't hack your email account for you. And if you can't find your hotmail inbox, you shouldn't be using a computer.

Semantic HTML is a two-edged sword.

Google approved PageRank stripping

Simon Willison — 2004-05-11T07:56:08-00:00

Blogger are now using the redirect-without-PageRank technique to protect their hosted blogs against comment spam (also used by Moveable Type). At the risk of sounding incredibly pleased with myself (which I am), that was my idea! Sweet.

Here's the real kicker: the URL redirector they are using is hosted on Google's primary domain. This is great news for people like myself who are running their own redirector, as the problem with having a redirect on your site is that it can be abused to make it look like people are visiting a specific site from a link on your domain: or even worse, to trick people in to visiting an unpleasant link (see Slashdot comments, where links are displayed along side the domain on which the "real" site is hosted). Now I can point my own PageRank stripper at http://www.google.com/url?sa=D&q=URL and let Google handle the redirects for me. Lovely.

Peter van Dijck recently joined the ranks of victims of wiki spam. Let's start rolling this technique out on Wikis as well. The trade-off in lost PageRank for linked sites is more than worth it.

Personalisation? We've already got it

Simon Willison — 2004-04-05T01:20:41-00:00

Vin Crosbie, a highly respected commentator on the online news industry, recently published his long awaited essay What Newspapers and Their Web Sites Must Do to Survive. It's long but captivating and well researched; if you have any interest in the role of traditional newspapers on the web you should take the time to read it.

Vin believes that customisation of both online and offline editions to serve reader's individual interests is critical to the survival of big-J media. He makes an excellent case; the problem is we are already there.

I would guess that at least 90% of my news intake comes from reading blogs - the 130 blogs you can see listed in the sidebar on my front page. That's not to say that I only read 130 sites - blogging is about linking, and those 130 sites in turn link me out to a huge network of news sources around the web. The blogs I read work as the ultimate personalised filtering mechanism: I read them because their authors have similar interests to me, and are the people most likely to direct me to content that I will personally find interesting.

No computerised system could possibly compete with 130 hand-picked human editors, working around the clock to channel interesting information in my direction. Blogs are conversations, but they are also filters. Never before in my life have I had to invest so little effort in finding so much diverse and fulfilling content. As a certified infovore I don't know how I survived without them.

What this means for traditional news media is anyone's guess. It's certainly not going to die out: someone has to collect the news and there's only so much unpaid bloggers can do in that regard. We certainly live in interesting times.

As an aside, if you're interested in online media you should read Steve Yelvington's essay Ten years in new media: Looking back, looking forward, which includes a call to arms to the online news industry to get over the novelty of new technology and finally start taking advantage of it.

Microsoft "get" blogging

Simon Willison — 2004-04-05T00:11:06-00:00

Who would have thought a year ago that Microsoft would be the company that took corporate blogging to the next level? Say what you like about the company itself, you can't fault the quality and quantity of bloggers coming out of Redmond at the moment. Yesterday I stumbled across this fascinating blog that provides an insight in to Microsoft's recruitment techniques. If you're looking for a job at a high-tech company you can't afford not to read this - they already have a bunch of resume advice, tips on what to wear to interviews and posts on subjects such as employee referrals, international recriting, phone screening and more.

The authors are two Microsoft recruiters and display an incredible enthusiasm for their work, which is probably what makes the blog such a great read. It's fun to compare their more recent entries with their opening entry, which Robert Scoble rightly criticised as being far too corporate. The great tragedy of the official Bush and Kerry blogs is that they frequently read like glorified press releases. One of the core messages of the Cluetrain Manifesto is that people want to be communicated with in a human voice. I don't know if the Microsoft recruiters have read it, but they've certainly taken that core value to heart.

Omit needless words, codified

Simon Willison — 2004-03-27T22:06:00-00:00

I continue to try to improve my writing. "Omit needless words" is all well and good, but identifying needless words can be a difficult task for the untrained eye. Paul Ford's Passivator bookmarklet highlights adverbs and passive verbs, both of which can indicate weak writing.

Avoiding protracted debates

Simon Willison — 2004-03-21T04:48:25-00:00

I love Charles Miller's Fishbowl. His latest entry introduces his rules for argument. Read them, follow them and save a truck-load of time avoiding protracted debates in the future. Heck, if everyone stuck to them the overall productivity of the internet would probably increase by a factor of ten.

Ghost town, sponsored by Google

Simon Willison — 2004-03-06T00:30:00-00:00

Via Boing Boing, this fascinating and utterly chilling photographic journey through the abandoned ruins of the Chernobyl dead zone.

As an aside, the free hosting provide used by the site appears to be inserting Google ads. Make sure you look out for them as you explore the site; the relevance algorithm gets stretched to the limit.

Attribution

Simon Willison — 2004-03-05T23:13:12-00:00

Via Kevin Fox, Wired are running an article that claims that authors of popular blog sites regularly borrow topics from lesser-known bloggers -- and they often do so without attribution.

The first part - that bloggers borrow topics from each other - isn't a new observation. Link commentary is one of the clasic forms of blogging, right up there with filling in quizzes and posting pictures of your cat. What's worrying is the lack of attribution.

Attribution is a critical part of good blogging etiquette. It provides "discovery credit" to the blogger who directed you to something, but more importantly it acts as a tool for communication. It's easy to tell when someone is linking to you, via referral logs or services such as Technorati. By attributing something to another blogger you make two useful statements: I read your blog (or at least stumbled across it somehow), and I'm interested in that particular entry. This is valuable feedback.

My favoured attribution method is the "via" link, as demonstrated at the start of this entry; I even use it in my blogmarks. One problem that I used to have with attributing interesting links, described here by Meri, is that when you browse with multiple tabs or browser windows it's easy to lose track of how you got to a certain page thanks to a "broken" back button. Thankfully there's a simple solution to this: the show referrer bookmarklet (adapted from a similar bookmarklet by Jesse Ruderman) which shows the page that referred you to the current page in an easily copy-and-pastable Javascript prompt.

This still doesn't help you if you clicked on a link in an aggregator such as NetNewsWire, but I've started habitually opening the real entry in a browser before clicking any links to ensure I maintain referral information. This is particularly important as the bookmarklet I use for updating my blogroll grabs the referrer for automatic "via" link generation.

The dangers of PageRank

Simon Willison — 2004-02-06T16:58:23-00:00

A well documented side effect of the weblog format is that it brings Google PageRank in almost absurd quantities. I'm now the 5th result for simon on Google, and I've been the top result for simon willison almost since the day I launched. High rankings however are not always a good thing, especially when combined with a comment system. A growing number of bloggers have found themselves at the top position for terms of little or no relevance to the rest of their sites, which in turn can attract truly surreal comments from visitors from search engines who may never have encountered a blog before.

I know of a couple of entries on my own blog that are attracting this kind of traffic. The most interesting is probably this entry on artifical diamonds, which has attracted comments from both buyers and sellers of artificial gems. My entry on MSN messenger usability problems from 2002 has drawn a steady stream of hilarious comments, no doubt caused in part by its top rating on Google for msn messenger sucks. Amusingly, for a long time Microsoft's own search engine was giving my page a high rank for a wide variety of less negative messenger related terms.

My own experiences of this phenomenon pale in to significance to some of the others I've seen. The most impressive example has to be Jason Kottke's brief review of the Matrix Reloaded, which drew over 900 comments from Google strays, developed its own micro-community and resulted in Jason pondering who owns the conversation on my web site? Jason eventually deciding to close and archive the thread after the page grew to more than a megabyte in size.

The problem can take on a far more disturbing twist. I won't link directly to these entries for fear of adding to their predicaments, but searches for crime scene cleanup and suicide chat rooms both return blogs in the first two results. The former thread is mostly crime scene cleanup companies marketing their services, but the latter is quite frankly disturbing. It's certainly lead me to double check the titles of my entries before posting them.

Thankfully, avoiding this kind of unwanted comment traffic is pretty simple. One way is to simply disable comments for entries older than a certain time (generally a couple of weeks), although personally I like to see the occasional comment on old entries. A neater solution proposed by Russell Beattie last year is to simply hide comments from search engine referrals, thus ensuring that random strays won't leave their mark without understanding the nature of your site first.

Solving comment spam

Simon Willison — 2004-01-28T03:24:56-00:00

There are two main schools of thought concerning comment spam: the optimists and the defeatists. Optimists believe that comment spam can be beaten with technology; defeatists (maybe I should call them pessimists) believe that comments are as doomed as email and we're all going to hell in a hand basket.

The story so far

I fall squarely in to the techno-optimist category. Back in September I started blacklisting domains linked to from spam comments, defending against return visits from spammers and allowing others to syndicate my block list to run on their own site. Then in October I tweaked my comment system to eliminate PageRank from links in comments, making spamming for search engine optimisation a futile exercise. Of course, this measure only works if spammers realise it's there (I know at least one has) which is why I'm personally very happy to see that the latest release of Moveable Type has adopted the technique - to mixed reviews from the MT community.

There have been a whole bunch of other technological innovations over the past few months. Sam Ruby has implemented throttling to ban people who post three consecutive comments, and has some great ideas about guarding against strangers. Jay Allen's MT-Blacklist makes the blacklisting concept available to a wide audience. Meanwhile, James Seng's MT-Bayesian introduces trainable spam filters adapted from the fight against email spam.

The challenges ahead

So those are the solutions so far; the critical question is whether they work. The amount of spam I've been getting has definitely decreased, but as I run a completely custom blogging system I'm safe from the automated scripts that target more widespread systems - other sites make easier targets. Now that the less ethical search engine optimisers have started to catch on to the potential of comment spam to improve their PageRank the amount of spam can only increase. Some bloggers have already started to disable comments entirely (thankfully Dan turned them back on again shortly afterwards), setting a worrying precedent for the elimination two way interactions comments allow between bloggers and non-bloggers.

I'll put it in writing now: I will never disable comments on this blog. In the past few months the comments here have proved far more interesting and valuable than my actual posts, and I really appreciate the quality of the discussions that have arisen here. I will take whatever steps are necessary to keep this a useful environment for discussion.

Many people have hailed user registration as the ultimate solution to spam. It isn't, because the value of PageRank is just too high - and writing a script to automatically create accounts (even with email confirmation required) is child's play to anyone who is competent in an internet-aware scripting language. Even accessibility-impeding captchas are no defence against spammers who can afford to employ cheap labour to defeat them - and with search engine rankings as critical as they are there's no shortage of spam dollars.

With those ruled out, let's look at the remaining solutions:

The killer

Without links, comment spam has no purpose. To eliminate spam, eliminate links. Redirecting them through a PageRank killer already achieves this, but proves too subtle for spammers intent on spreading their links as widely as they can. Too truly eliminate spam, strip out links and anything that even looks like a URL and force the spammer to preview their carefully crafted advertisement before hitting submit. Seeing as hyperlinks are the single most important feature of the web this may seem draconian - and indeed it is. But on a site that serves more as a discussion forum than a farm and where the alternative to killing links is killing comments entirely this could be the saving factor.

For most blogs however links are an essential part of the discourse - I certainly wouldn't want to disable them here. Now only do they add huge value to the discussions, but more importantly they act as a "signature" for many commenters - knowing a comment is by "Dan" is far less useful than knowing that it's by Dan from www.simplebits.com.

Finding a compromise

Draconian measures such as the above wouldn't be necessary if spammers would wise up to the fact that their carefully crafted missives were having no effect on their precious PageRank. The real challenge then is to make anti-PageRank measures obvious to even the most brain-addled viagra peddlers. I've taken the first step towards this by turning on compulsory previewing for comments, which should have the added benefit of reminding legitimate commenters to use paragraph tags. I'll be working on ways of making the anti PageRank measures more obvious over the next few days, as and when work permits.

I've seen people argue that depriving legitimate commenters of PageRank is a poor compromise. I disagree: if the only cost of eliminating the incentive to spam is the loss of some Google ego then I see it as a price well worth paying. Of course, I say that as someone who's already built up their Google ego but at the end of the day it's my blog, my rules. One solution I've considered is creating a whitelist of sites that frequent commenters use in their signatures, causing them to be displayed without a redirect.

Comment spam is a solvable problem. Furthermore, blogging about comment spamming is almost as dull as blogging about blogging. Let's hurry up and solve it so we can go back to blogging about cats.

Moveable Type now kills PageRank on comment links

Simon Willison — 2004-01-21T20:05:17-00:00

This is pretty cool: Moveable Type 2.661 is out and includes a whole bunch of comment spam fighting features, including one inspired by my own anti-spam measure of disabling PageRank on links from comments by sending them through a redirect. This is great news for me as the redirect acts as a deterrent, and deterrents are only worthwhile if people know about them in the first place. With the most popular blogging system (at least amongst comment spammers) now featuring the same deterrent hopefully SEO spammers will start to get the message.