Google

Google Base is interesting

Simon Willison — 2005-11-16T12:34:46-00:00

I'm still trying to get my head around Google Base. Here's a brain-dump of my thinking so far. First, some links.

Google Base FAQ
Google Base introduction on the Google Blog (includes testimonials)
Tom's first impressions

Base is a very interesting product for a whole bunch of reasons. The data model is surprisingly simple on the surface: all items have a title, description, (optional) external URL, a "type" and a set of labels (a.k.a. tags) and "attributes". Attributes are something for tag enthusiasts to get excited by - they're name/value pairs that are kind of like tags in that you can apply them to anything, but more structured and with a greater level of implied meaning.

Attributes instantly made me think of geotagging on Flickr, where tags are overloaded to store latitude and longitude values (example here). Having first class support for this kind of extensible data is a very powerful concept.

Another interesting problem that the Google Base data model could be used to tackle is Wikipedia's WikiProjects. If you look at any US Navy ship entry on Wikipedia (example) you'll see a table on the right hand side of standard attributes relating to that ship - things like Length, Displacement, Armament and so on. This data isn't really structured - it's just a wiki table, manually maintained by participants of the Ships WikiProject.

Obviously this data would be more valuable if it was structured in a way that allowed queries to be made against it. Base-like attributes provide a way of doing this.

There's definitely a trend towards this kind of loose data model at the moment. JotSpot allows all pages within a wiki to have as many extra name/value attribute pairs as you like (even the wiki body itself is internally implemented as a special attribute), and Ning works along similar lines.

Base currently allows bulk importing of data using tab delimited files, RSS or Atom. There are no outward bound APIs which is a notable omission - I wouldn't be at all surprised to see them added in the next few weeks.

Dissecting the Google Firefox Toolbar

Simon Willison — 2005-07-08T10:10:12-00:00

Google have finally released a Firefox version of the Google Toolbar, with some nice praise for XUL in to the bargain. Of course, the most interesting part of the toolbar from a geeky point of view is the bit that queries Google's servers for PageRank. Sure enough, if you download the google-toolbar.xpi file, unzip it, then unzip the google-toolbar.jar file within there's a file called pagerank.js with all of the juicy details.

To query PageRank, the toolbar makes a standard HTTP request to toolbarqueries.google.com, with the page to query in a parameter along with a hash (presumably to discourage scraping). pagerank.js includes the hash algorithm, with some amusing implementation details:

var GPR_HASH_SEED = "Mining PageRank is AGAINST GOOGLE'S TERMS OF SERVICE. Yes, I'm talking to you, scammer.";

function GPR_awesomeHash(value) {
  var kindOfThingAnIdiotWouldHaveOnHisLuggage = 16909125;
  ...
}

The spell check feature (spellcheck.js) is interesting as well. When you click the "Check" button, the toolbar packages any content in form fields up in XML and POSTs it to http://www.google.com/tbproxy/spell. It get backs a simple XML document providing the offset, length and confidence for each spelling error along with a list of suggested alternatives. The user interface stuff is all handled by the extension.

If you want to watch the toolbar in action, I recommend the fantastic LiveHTTPHeaders extension.

Fighting RFCs with RFCs

Simon Willison — 2005-05-06T20:39:45-00:00

Google's recently released Web Accelerator apparently has some scary side-effects. It's been spotted pre-loading links in password-protected applications, which can amount to clicking on every "delete this" link - bypassing even the JavaScript prompt you carefully added to give people the chance to think twice.

"Aah," I hear you cry, "but RFC 2616 clearly states that you shouldn't perform state changing operations with a GET or HEAD method!"

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval.

I'll see your RFC 2616 and raise you an RFC 2119:

SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

Hiding your dangerous delete links behind an authentication scheme is a perfectly acceptable compromise. Web Accelerator is B.A.D.

Update: Be sure to read the excellent discussion brewing in the comments. Hiding behind authentication may not be as acceptable a compromise as I had first thought.

Update 2: If you haven't been following the comments, I've had a change of heart. Even in the absence of Web Accelerator, hiding behind authentication leaves your application open to some very nasty security vulnerabilities (malicious pages can piggy-back your session and cause havoc making dangerous GET calls). I still think the RFC language covers people who thought long and hard before implementing a dangerous GET, but if you haven't thought about security and accelerating caching proxies such as Web Accelerator you haven't been thinking hard enough.

Update 3: So, it turns out using POST is no defence at all against CSRF attacks. I've been learning a whole bunch of interesting stuff this evening.

Google cruft

Simon Willison — 2005-02-24T00:34:00-00:00

New Google feature: Google Movies. Displays aggregated movie reviews (like Rotten Tomatoes), looks up local movie times based on your zip code saved in Google Local (more evidence of the fabled Google cookie), and even handles recommendations.

The downside is that it's yet more cruft for the search results page. Here's a screenshot, with the cruft in red and the actual search results in green:

Ben Hammersley has more commentary.

Google Maps and XSL

Simon Willison — 2005-02-08T14:19:37-00:00

I'll probably write more on this later, but it seems that Google Maps is using XSL. I spotted it loading the following pages while sniffing its activity with LiveHTTPHeaders:

This is in addition to the (now expected) XMLHttpRequest stuff. There even appears to be some of Microsoft's weird VML, although as I'm on a Mac I don't have access to IE/Windows to see what it's doing with it.

The bulk of the Google Maps JavaScript appears to be hidden away in maps.1.js, which becomes a lot more readable if you feed it through PrettyPrinter.de

As for Google Maps itself, it's an amazing piece of work but it's a shame they didn't follow map.search.ch's lead in degrading gracefully to a static version for unsupported browsers.

If anyone has any further insights in to how it all works, please post them in a comment.

rel="nofollow"

Simon Willison — 2005-01-17T01:39:26-00:00

Reading between the lines (which in this case isn't particularly hard), this and this (don't forget to view source) suggest that Google are soon to announce that they won't be calculating PageRank for links with a rel="nofollow" attribute. Finally, an official way of fighting the economics of comment spam by denying PageRank on user-submitted link content. Sam Ruby points to Mark Pilgrim's prediction that spammers won't care - they'll spam anyway, on the offchance that they hit somewhere undefended. I'm optimistic - if the major weblog (and wiki) vendors get behind this one it could help stem the tide.

As an aside, I have exams starting in a week and plenty to revise, so I'll probably be on hiatus until the end of the month.

Google Print

Simon Willison — 2004-12-14T15:44:38-00:00

I'm probably late to the party on this one, but I just noticed that Google Print results are now included in any Google search that starts with "books on". I can't say I like the lousy discoverability of the interface much - a search box at print.google.com would be a welcome addition - but the results are pretty impressive. It's also a shame that they're using a nasty obfuscation technique to disable copying and printing (based on serving book pages up as background images), if only because it will fuel yet more questions from newbie web developers asking how to do exactly that. Still, with today's announcement that Google are to team up with five leading libraries to scan more books this service is going to get a whole lot more important over the next few years.

I have some gmail invites

Simon Willison — 2004-06-18T23:57:15-00:00

I have four gmail invites left. The first four people to leave a comment with their email address can have them (put it in the email field and fill in a URL as well if you're worried about spam harvesters - the email won't be displayed but I'll still have access to it). No random gift or good deed necessary - consider it a "thank you" for reading.

In related news, Adrian's Python gmail export script is really rather neat.

Update: They all went in under 15 minutes! To avoid being bombarded with requests, I'll state now that I have no plans of handing out any more in this manner. Still, it was a fun way of drawing some of the lurkers out of hiding.

Supplemental Results

Simon Willison — 2004-05-13T07:22:11-00:00

Does anyone know what Google means when it says that something is a "Supplemental Result"?

Google, circa 1998

Simon Willison — 2004-05-02T20:43:05-00:00

Thanks to the ever impressive Internet Archive (did you know they host old public information films as well?) here's Google's homepage from 1998. Their searchable index was slightly less than 25 million pages, their hardware was less than a dozen machines apparently held together with lego and their crawler was called BackRub. Following the links will take you to Sergey and Larry's homepages, where digging a little deeper will even uncover the now infamous Sergey-in-drag photo.

My favourite insight though comes from the Legos page (why do Americans insist on adding an 's'?) on Larry's site:

I attriubute a great deal of my understanding and ability with mechanical devices to Legos and similar construction toys.

I've often thought that the most important factor in leading me to geek-hood, apart from a C64, was a serious obsession with Lego Technic from an early age.

I've avoided posting about the Google IPO because know nothing of the world of finance, but you've gotta love that Google's initial IPO valuation is e * 10 ^ 9.

What is Google?

Simon Willison — 2004-04-05T08:06:09-00:00

Via John Battelle, Rick Skrenta's remarkable piece on what Google have actually built. They don't just have the world's best search engine, they have the world's largest and most scalable platform for developing huge web-based applications.

Google has taken the last 10 years of systems software research out of university labs, and built their own proprietary, production quality system. What is this platform that Google is building? It's a distributed computing platform that can manage web-scale datasets on 100,000 node server clusters. It includes a petabyte, distributed, fault tolerant filesystem, distributed RPC code, probably network shared memory and process migration. And a datacenter management system which lets a handful of ops engineers effectively run 100,000 servers. Any of these projects could be the sole focus of a startup.

[ ... ]

While competitors are targeting the individual applications Google has deployed, Google is building a massive, general purpose computing platform for web-scale programming.

Fascinating stuff.

1GB of webmail from Google

Simon Willison — 2004-04-01T02:35:15-00:00

Provided this story about a new 1 GB webmail service from Google isn't a lame early April fool, I'm really psyched about it. A decent amount of space combined with Google's search technology could really help me keep up to date with my email. Just off the top of my head, here's my ideal hosted webmail feature list:

Costs between two and ten dollars a month. This is a critical service I'm entrusting to a company - I want the reassurance that there's cold hard cash on the line if they screw anything up. Paying for a no-ads option is fine too.
Smart filtering. I'm on a bunch of different mailing lists and can't live without filters.
"Virtual folders" - things like "all emails from this person", "emails sent to this person" should be on hand at all times.
Import and export mail as mbox files. Don't lock me in!
Decent spam filtering, obviously.
Send plain text, receieve plain text. No HTML in either direction.
Kick-arse search.

I don't really have very complex needs, which is why I'm so frustrated that so far nothing I've tried has done what I need it to. Here's hoping Google can hit the mark.

Update: The Google press release is dated April 1st. Given their history, it's almost certainly a hoax. Aah well, we can dream.

Ghost town, sponsored by Google

Simon Willison — 2004-03-06T00:30:00-00:00

Via Boing Boing, this fascinating and utterly chilling photographic journey through the abandoned ruins of the Chernobyl dead zone.

As an aside, the free hosting provide used by the site appears to be inserting Google ads. Make sure you look out for them as you explore the site; the relevance algorithm gets stretched to the limit.

The dangers of PageRank

Simon Willison — 2004-02-06T16:58:23-00:00

A well documented side effect of the weblog format is that it brings Google PageRank in almost absurd quantities. I'm now the 5th result for simon on Google, and I've been the top result for simon willison almost since the day I launched. High rankings however are not always a good thing, especially when combined with a comment system. A growing number of bloggers have found themselves at the top position for terms of little or no relevance to the rest of their sites, which in turn can attract truly surreal comments from visitors from search engines who may never have encountered a blog before.

I know of a couple of entries on my own blog that are attracting this kind of traffic. The most interesting is probably this entry on artifical diamonds, which has attracted comments from both buyers and sellers of artificial gems. My entry on MSN messenger usability problems from 2002 has drawn a steady stream of hilarious comments, no doubt caused in part by its top rating on Google for msn messenger sucks. Amusingly, for a long time Microsoft's own search engine was giving my page a high rank for a wide variety of less negative messenger related terms.

My own experiences of this phenomenon pale in to significance to some of the others I've seen. The most impressive example has to be Jason Kottke's brief review of the Matrix Reloaded, which drew over 900 comments from Google strays, developed its own micro-community and resulted in Jason pondering who owns the conversation on my web site? Jason eventually deciding to close and archive the thread after the page grew to more than a megabyte in size.

The problem can take on a far more disturbing twist. I won't link directly to these entries for fear of adding to their predicaments, but searches for crime scene cleanup and suicide chat rooms both return blogs in the first two results. The former thread is mostly crime scene cleanup companies marketing their services, but the latter is quite frankly disturbing. It's certainly lead me to double check the titles of my entries before posting them.

Thankfully, avoiding this kind of unwanted comment traffic is pretty simple. One way is to simply disable comments for entries older than a certain time (generally a couple of weeks), although personally I like to see the occasional comment on old entries. A neater solution proposed by Russell Beattie last year is to simply hide comments from search engine referrals, thus ensuring that random strays won't leave their mark without understanding the nature of your site first.

Google's Internal Blogs

Simon Willison — 2003-10-22T04:58:32-00:00

Evan Williams on Google's intranet weblogs:

How many people blog at Google?

Not sure what the count is, but I know there's a couple hundred or more. It's really interesting to see the network grow from scratch.

Do you use that to get to know one another or to keep up-to-date on projects?

A lot of people use it to keep up-to-date on projects and to share pointers or expertise. I've heard people comment on how it's way easier to know what's going on internally now. You can find out what's going on when you go there or when you're curious about it, but you don't have to be deluged or distracted from your normal day.

Markup question: I used a definition list in the above quotation - was this appropriate? Is there a better way of marking up this information?