Search Engines

Yahoo News Search RSS feeds

Simon Willison — 2003-10-08T00:29:46-00:00

It's not a new idea (Feedster has been doing it for a while) but it's a first for a major search engine: Yahoo are now offering RSS feeds of the results of searches within Yahoo news. The feeds are advertisement free, probably because you have to click through to the news stories to read them in full. I wonder how long it will be before someone starts offering custom feeds like this with advertising in the feed itself? As RSS is an XML format parsing out adverts before they get to the user is a much more obviosu step than ad-blockers in web browsers.

Google conspiracy theories

Simon Willison — 2003-09-17T00:51:40-00:00

Microdoc News have a poorly researched story suggesting that Google have been engineering their search results to favour their own properties:

It could be argued that the most important site that should appear when searching for the word blogs would be the generic site where anyone with a blog can get listed for her/his three minutes of fame, which includes any blog, anywhere in any system. Weblogs.com is a directory of sorts to any current post and is like, if you please, a central nervous system to the world of blogs. However, Google does not list weblogs.com as the primary site -- blogger.com is listed as the prime, first-up site in the listings that result from the blogs search. Is that because Google Inc., owns blogger.com, or is it that blogger.com is really what one would expect as the first result?

Time to break out Python again. I won't explain the following code in detail, but essentially it downloads the HTML source of the front pages of both Blogger.com and Weblogs.com, strips out the HTML tags (defined as anything between two angle brackets) and counts the number of occurrences of the individual word 'blogs'.

>>> import urllib, re
>>> striptags = re.compile('<[^>]+>')
>>> blogs = re.compile(r'\bblogs\b', re.I)
>>> blogger = urllib.urlopen('http://www.blogger.com/').read()
>>> weblogs = urllib.urlopen('http://www.weblogs.com/').read()
>>> len(blogger), len(weblogs)
(26369, 394323)
>>> blogs.findall(striptags.sub('', blogger))
['blogs', 'blogs', 'blogs', 'Blogs']
>>> blogs.findall(striptags.sub('', weblogs))
['Blogs', 'Blogs', 'blogs']

The above code shows that while Blogger.com mentions the word 'blogs' four times in 26,000 characters, Weblogs.com only mentions it three times in 394,000 characters! Blogger has a far higher 'blogs' word density - in fact, the only occurrence of the word on Weblogs.com is when it happens to be a part of the name of one of the several thousand blogs listed on the page at any one time.

Although word density is a reasonably useful metric for telling if Google will like something, everyone knows that Google's secret sauce is PageRank, which is based in part on the number of pages linking to a site. Two quick link: searches reveal 7,840 links to Weblogs.com, but a whopping 61,500 links to Blogger.com (no doubt helped by all those "powered by blogger" stickers).

So Blogger.com not only has a higher word density for the designated search term, it also has far more links to it overall. Is it really so surprising that it's coming out on top?

Further more, if you run a search for 'weblogs', Weblogs.com comes out as the number one result. It's all in the name.

Dave Winer finds it strange that the Google Weblog (unaffiliated with Google the company) comes out as the first result in a search for 'weblog'. My guess is that this is a result of the blog's name influencing the text of links made to it - when you link to Doc Searls or myself (both of whom have 'weblog' in their site title) you can abbreviate it to "Doc Searls" or "Simon Willison", but when you link to the Google Weblog you have to use the fully qualified name or your link won't make sense. Google can be strongly affected by link text, as last year's Google bombing epidemic aptly demonstrated.

On Metadata

Simon Willison — 2003-08-02T21:06:50-00:00

Tim Bray's series On Search now has a table of contents page linking to each of the previous entries. The most recent article covers metadata, and includes some insightful commentary on the huge problem of gathering metadata from users in the first place.

Learn to search!

Simon Willison — 2003-07-24T16:17:35-00:00

Slate: Digging for Googleholes:

Type in the make and model of a new DVD player, and you'll get dozens of online electronic stores in the top results, all of them eager to sell you the item. But you have to burrow through the results to find an impartial product review that doesn't appear in an online catalog.

sony DVP-S550D - shopping sites come out top

sony DVP-S550D review - review sites come out top

Search for "apple" on Google, and you have to troll through a couple pages of results before you get anything not directly related to Apple Computer - and it's a page promoting a public TV show called Newton's Apple. After that it's all Mac-related links until Fiona Apple's home page. You have to sift through 50 results before you reach a link that deals with apples that grow on trees: the home page for the Washington State Apple Growers Association.

apple - lots of stuff about Apple computers

apple fruit - lots of stuff about Apples, the fruit

These are not even advanced search techniques. It's a basic rule of searching: if your first set of results aren't what you are looking for, enter more specific terms and try again.

So, when you're doing research online, Google is implicitly pushing you toward information stored in articles and away from information stored in books. Assuming this practice continues, and assuming that Google continues to grow in influence, we may find ourselves in a world where, if you want to get an idea into circulation, you're better off publishing a PDF file on the Web than landing a book deal.

I'd say that day has already come (but replace PDF with HTML), but I'm not sure I understand how this is a bad thing. Surely information is more valuable if it is searchable? Books are not going to die out because of the internet (how many people prefer reading from a screen?) but if you have an idea to share the internet is obviously a better medium - you reach millions more people for a fraction of the cost of traditional publishing.

There are a lot of legitimate concerns about Google relating to its size and massive influence over the web's traffic, but concerns about skewed results are often the fault of the user rather than the tool. Learn to search!

time_since() on Feedster

Simon Willison — 2003-07-01T23:20:17-00:00

This is pretty cool: Scott's taken Nat's time-since function and added it to Feedster, giving a quick indication of how long ago an item was posted.

Tim Bray on search

Simon Willison — 2003-06-16T15:37:43-00:00

I love it when bloggers stick to their word. The other day, while describing a quick Perl hack that really impressed a major client a few years ago, Tim Bray mentioned the following:

Then I turned on Microsoft's search engine, at that time called Index Server, now I believe called Index Services, which is a pretty nice tool (we don't have the equivalent in the Open Source world, more on that another time).

And sure enough, he's just posted the first in a series of essays on full-text search. Go read it: it's really interesting stuff. Tim's conclusion is:

What we need is for Apache to come out-of-the-box with a built-in search capability that you just push a button and it works, and it's fast, and doesn't need much care and feeding, and it's internationalized, and it has the right API for when you want to get fancy.

Until that happens, I will happily recommend MySQL's built in fulltext search indexing for quickly adding a relatively powerful search facility to a site. I use it on this blog and my only real criticism is that it insists on search words of at least 4 letters, which is less than ideal when most of your entries include TLAs. Hopefully they'll provide a way around this limitation in a future release.

Feedster AND searching

Simon Willison — 2003-05-01T15:12:16-00:00

Feedster finally supports AND as the default search operator. This is a very good thing. I've decided to leave this site's search engine as using OR, mainly because I feel for a small search set (approximately a thousand entries) more search results is better than fewer search results and because relevancy algorithm used by MySQL to order the results appears to be working extremely well. For large data sets such as Feedster or Google I definitely prefer to only see results containing all of my search terms.

More fun with Search

Simon Willison — 2003-04-28T19:58:18-00:00

While browsing around my phoenix/ directory I spotted a sub-directory called searchplugins, which appears to control the list of search engines available in the very useful search box at the top right corner of the browser. A bit of digging later and it turns out that adding new search engines to Mozilla based browsers is remarkably easy: The Mozilla Search Project.

I'm a sucker for new toys, so click here to add the simon.incutio.com search engine to your (Mozilla or Firebird) browser :)

Site search finally available

Simon Willison — 2003-04-25T16:55:02-00:00

I've finally got around to adding a search page to this site. It uses MySQL's full text indexing, which is extremely fast and provides good results but comes at the expense of flexibility. Search terms less than 4 letters long are ignored, and multi-word searches are handled using OR rather than AND. This nearly put me off using it, but the relevancy algorithm is excellent which I think outweighs the disadvantage of not being able to use pure AND queries.

MySQL 4.0 introduces far more powerful boolean mode full text searches which allow all kinds of modifiers and extra syntax, but this site currently runs on 3.23.54 so I can't play with those just yet. Jeremy Zawodny's article on MySQL 4 explains boolean mode and describes many other exciting new MySQL features as well.

100 random pictures

Simon Willison — 2003-04-13T14:11:39-00:00

100 random AltaVista pictures is fascinating, if not guaranteed work-safe.

Yahoo Search uses CSS

Simon Willison — 2003-04-12T20:11:37-00:00

In all the fuss about Yahoo's new search interface over the past few days, the extensive use of CSS in the results pages was almost completely overlooked, probably because the page still contains a small layout table for the top and bottom navigation. The results themselves are served up as a styled ordered list, at least for modern browsers (thanks to a server side browser sniffer). More information in this message from Yahoo's Brian Ghidinelli, who is seeking feedback.

More on the new Yahoo

Simon Willison — 2003-04-07T21:47:43-00:00

Unsurprisingly, the new Yahoo is generating a whole load of commentary. There's a good thread going on Signals vs Noise, and ia/ has coverage as well. I've been playing with it a bit and it's definitely an immense improvement on the current Yahoo, although it's still not quite as usable or responsive as Google. I also noticed that the search results are exactly the same as Google's (even for image search) so it looks like Yahoo haven't switched over to Inktomi just yet.

It's worth clicking through the tour to get an overview of the new interface. The "open in new window" icon for each search result is a clever addition, but the smartest feature in my opinion are the specialised Yahoo shortcuts. mail! takes you to Yahoo mail, calendar! to Yahoo calendar and so on. It looks like are Yahoo hoping to out do Google by capitalising on their many other services, which seems like a very sensible approach.

A new Yahoo

Simon Willison — 2003-04-07T18:25:26-00:00

New York Times: Yahoo Plans Improvements in Effort to Regain Lost Ground. I'm guessing this is what it's going to look like (via thelist).

Thirty five year old cookies

Simon Willison — 2003-03-09T14:58:49-00:00

I'm finding myself slightly confused about the Google backlash washing around the blogosphere, which is summarised quite well by Gavin Sheridan. Most of the arguments against using Google unsurprisingly centre around privacy issues, in particular the "35 year cookie". I was under the impression that cookies could only be set for a maximum of a year, but having checked Netscape's Cookie Specification and RFC 2965 it appears I was mistaken.

So let's take a look at the cookies in question, via the Mozilla project's handy Web Sniffer utility (the front page for this tool is here):

HTTP/1.0 200 OK Content-Length: 3403 Connection: Keep-Alive Server: GWS/2.0 Date: Sun, 09 Mar 2003 14:34:32 GMT Content-Type: text/html Cache-control: private Set-Cookie: PREF=ID=05ba0c124de8df6e:TM=1047220472:LM=1047220472:S=Ke2RQCqjCEowS1x-; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com

There it is - a 35 year cookie. Now let's take a look at some of Google's competitors.

AllTheWeb:

HTTP/1.1 200 OK Date: Sun, 09 Mar 2003 14:36:42 GMT Server: Apache/1.3.27 (Unix) PHP/4.2.3-atw Set-Cookie: atw-uid=CgVSBj5rUXoAAQnFAwSFAg==; path=/; domain=.alltheweb.com; expires=Sat, 09-Mar-13 02:36:42 GMT X-Powered-By: PHP/4.2.3-atw Last-Modified: Sun, 09 Mar 2003 14:35:00 GMT Expires: Thu, 19 Apr 2001 04:25:21 GMT Cache-Control: max-age=0, private Set-Cookie: PREF=frschk=1:_lm=1047220602; expires=Fri, 07-Mar-08 14:36:42 GMT; path=/ Connection: close Content-Type: text/html; charset=iso-8859-1

That's two cookies - one for 5 years and one for 10 years. Interesting to see that they're using their own modified version of PHP 4.2.3 :)

Teoma:

HTTP/1.1 200 OK Server: Microsoft-IIS/5.0 Date: Sun, 09 Mar 2003 14:38:50 GMT Connection: Keep-Alive Content-Length: 6629 Content-Type: text/html Set-Cookie: CTST=yes; expires=Sun, 09-Mar-2003 15:03:50 GMT; path=/ Cache-control: private

That cookie lasts for about half an hour and doesn't contain a unique identifier. Plus they're running IIS!

Altavista:

HTTP/1.0 200 OK Set-Cookie: AV_POS=pos=1047220999574; path=/; domain=.altavista.com; Set-Cookie: AV_USERKEY=AVS03b87123ae55d80a1c21250000022; expires=Tuesday, 31-Dec-2013 12:00:00 GMT; path=/; domain=altavista.com; Server: AV/1.0.1 MIME-Version: 1.0 Cache-Control: no-cache,no-store,max-age=0 pragma: no-cache Expires: Sun, 09 Mar 2003 14:43:19 GMT Set-Cookie: AV_MKT=1; Domain=altavista.com; Path=/; Expires=Thu, 01-Dec-1994 16:00:00 GMT Content-Type: text/html; charset=ISO-8859-1 Content-Length: 10020 Date: Sun, 09 Mar 2003 14:43:19 GMT

What a mess! There's a session cookie (which only lasts until the browser s closed) recording what looks like the time I first visited the front page, a 10 year cookie with a unique ID and another cookie set to expire in 1994, possibly in an attempt to wipe out cookies set by an older version of the site.

So what have we learnt? Both AllTheWeb and Altavista set 10 year unique identifier cookies, while Teoma appears not to set any. At the end of the day though, what is the difference between a 10 year and a 35 year cookie? How many people are going to go a whole ten years without losing their browser's cookies, through a browser upgrade, PC upgrade, change of job or just wiping the cookie directory? Thee answer to that question is self evident, so in practise a 10 year unique identifier cookie is just as big an invasion of privacy as a 35 year cookie.

On the privacy front, AllTheWeb and Altavista are just as guilty as Google.

Roogle

Simon Willison — 2003-03-08T23:40:48-00:00

Scott Johnson has put together a blog search engine with a difference: it indexes RSS feeds rather than crawling the blogs themselves. Roogle is still under heavy development (and Scott is blogging it as he goes) but is shaping up to be a very neat tool. If your blog isn't already being indexed, you can add it using this form.