RSS and Syndication

Browser innovation is alive and well

Simon Willison — 2004-09-14T16:28:33-00:00

Here's a feature that caught me by surprise (maybe I haven't been keeping my ear close enough to the ground): the new Firefox 1.0 preview release supports Live Bookmarks, a novel twist on RSS aggregators where feeds look just like bookmark folders, displaying a list of bookmarks corresponding to the headlines from the feed. Best of all, the feature support RSS autodiscovery. Sites with auto-discoverable feeds display an attractive RSS icon on the right hand side of the status bar, allowing for one click subscriptions.

There are certainly some issues to be ironed out. Subscribing to a feed without auto-discovery isn't very easy, as the "New Live Bookmark" button is missing from the bookmark manager's toolbar (look in the File menu there instead). I also haven't found the interface for setting subscription options such as polling intervals, so I'll have to trust that Firefox is doing more or less the right thing. Those points aside, it's great to see Firefox driving forward browser innovation yet again.

Update: Here's a killer app for this feature: if you use blogmarks/b-links/del.icio.us instead of bookmarks, subscribing to the accompanying RSS feed as a live bookmark will give you access to your most recently added items within the browser UI.

How to track an RSS feed

Simon Willison — 2004-09-01T00:53:28-00:00

According to the HTTP specification, RSS/Atom aggregators should obey the HTTP 301 Moved Permanently header by altering the stored subscription URL for the feed they are attempting to retrieve.

This behaviour can be used to track repeat aggregator hits to a feed, in essence the equivalent a setting a permanent cookie. The first time an aggregator hits the published feed address, a 301 header is served redirecting that aggregator to a new URL incorporating a unique ID. The aggregator permanently changes the stored subscription URL, meaning future request to that feed will carry the unique ID that was assigned the first time the feed was retrieved.

At its most innocent, this could allow people to track their number of unique subscriptions - although the value of this would be severely diluted if people started deliberately subscribing to the same redirected feed URL. I'm sure there are more insidious uses for this as well.

Maybe aggregators should prompt users when a feed has permanently moved, to prevent them from being tracked without their knowledge.

The orange XML icon sucks

Simon Willison — 2004-06-16T18:28:48-00:00

I'm not a fan of the orange XML icon, even though I use it on my categories page. I love the concept of a universally recognised icon for RSS feeds, but XML is simply the wrong acronym to put on it. Don't think it's liable to cause confusion? Check out this excerpt from Journalism.co.uk (emphasis mine):

The Guardian is planning to launch more XML-based services in the next few months, adding to the news feed services the site already provides to both commercial clients and individual readers.

XML, also known as RSS, is a method of streaming information from a website. Many sites now offer a free XML feed enabling readers to receive headlines from news sites via a desktop software program. XML also avoids problems involved with sending email newsletters, such as junk mail and out-of-date addresses.

Doh!

They Work For You

Simon Willison — 2004-06-06T21:51:21-00:00

Today is/was (you never can tell with these wretched time zone differences) NotCon 2004, London's premiere low-cost, informal, one-day technology conference. Friday's MiniNTK promised the unveiling of a new project from the people behind FaxYourMP and PublicWhip and sure enough, here it is: TheyWorkForYou.com.

TheyWorkForYou.com is the finest example of a political hack I've ever seen. It's basically an ultra-user-friendly front-end to the Hansard public record of all speeches and debates in the houses of commons, with each MP getting their own page complete with a summary of their recent performance and member's interests and a list of their recently recorded parliamentary appearances.

The community features are pretty impressive as well: you can add comments to any speech made in any debate, and there's also a user-populated glossary which is automatically linked in to the speech transcripts.

But with all that, the most useful feature is probably the most subtle: you can subscribe to an RSS feed of your MP's appearances, meaning you can keep track of everything they say on your behalf. It's simple, powerful and a perfect example of the political hacker ethic at work.

Atom discussion minutes

Simon Willison — 2004-05-18T22:24:53-00:00

The minutes from the Atom/W3C discussion in New York have been posted online. Unfortunately the default formatting is pretty difficult to follow. I found it a lot easier to figure out who was saying what after applying the following CSS (using the test styles bookmarklet):


abbr {
  display: block;
  margin-top: 1em;
  margin-bottom: 0.5em;
  font-weight: bold;
}
abbr:after {
  content: " - " attr(title);
}

Advanced Python network programming

Simon Willison — 2004-02-18T03:36:18-00:00

Understanding Network I/O, Part 2 by George Belotsky (via The Farm) is the best tutorial on the subject of network programming I've seen yet. It provides a detailed explanation of simple threaded network clients, thread pools using the Queue module and asynchronous I/O using both Twisted and Python's asyncore library - then discusses the strengths and weaknesses of each approach.

I've also just noticed that O'Reilly provide RSS 1.0 and Atom feeds for each of their authors. Neat.

RSS vs Atom, condensed

Simon Willison — 2004-02-12T20:31:11-00:00

Jeremy Zawodny:

Since everyone's jumping up and down about the Atom vs. RSS (or "Google vs. RSS" if that's your paranoia) debate, it's probably worth pointing out that My Yahoo's RSS module also groks Atom. It was added last night. It took about a half hour.

To borrow a phrase, it's just data.

Hot Links

Simon Willison — 2004-02-05T23:53:50-00:00

The thing I love about RSS, and by extension web services, is that they allow people to take publically available data and combine it in ways never thought of by the originator of the feed. The internet is awash with examples of this, from useful services such as Feedster to useless amusements like I despise you and your so-called taste, the most insulting extension of the Amazon API I've seen to date.

Hot Links is the neatest new serice of this kind I've seen in quite a while, spotted a few weeks ago thanks to a vanity PubSub subscription. Hot Links subscribes to a bunch of RSS feeds for link logs (or blogmarks or whatever you want to call them) and watches them for newly linked sites. When it spots one, it grabs a thumbnail sized screenshot using khtml2png and posts it along with the description cribbed from the link log. Sites linked to from more than one link roll are promoted, allowing Hot Links visitors to restrict their "level" and view only sites with a certain minimum number of citations.

Syndicating link logs like this is a smart idea, and the people behind Hot Links have pulled it off in a stylish manner with great attention to detail. I particularly like the ability to view links from just my blogmarks as it provides a neat visual reminder of the sites I've recently recorded.

Atom autodiscovery test suite

Simon Willison — 2003-12-20T21:45:20-00:00

Mark Pilgrim has released the Atom autodiscovery test suite, comprising 148 tests:

When we say that Atom is going to have better specs, validators, and conformance tests than anything you've ever seen before, this is what we're talking about.

Mark's atomautodiscovery.py (based on Python's sgmllib module) passes all of the tests using a surprisingly small amount of actual code. The neatest thing about the test suite is that each test provides a <link rel="next" ... attribute pointing to the next test in the suite. atomautodiscovery.py uses these links in its test() function to run through every test in the suite, meaning that new tests can be added without modifying the test execution code.

Feed you

Simon Willison — 2003-11-26T01:55:12-00:00

Wow, that's what I call feedback! It's a shame pretty much everyone hates the new design but I like it so it stays. I've taken a few tips though and tweaked the link colours a bit, as well as making a few other small changes such as a darker green for the header and a 1em margin around the page.

In an attempt to satiate the voracious appetite for RSS displayed by some of my visitors I've set up two new feeds: Blog comments and Blogmarks. I don't use an aggregator myself so I'd appreciate feedback on how well they work. I've also put together a blogmarks archive - no search engine yet, but it's on the list.

Yahoo News Search RSS feeds

Simon Willison — 2003-10-08T00:29:46-00:00

It's not a new idea (Feedster has been doing it for a while) but it's a first for a major search engine: Yahoo are now offering RSS feeds of the results of searches within Yahoo news. The feeds are advertisement free, probably because you have to click through to the news stories to read them in full. I wonder how long it will be before someone starts offering custom feeds like this with advertising in the feed itself? As RSS is an XML format parsing out adverts before they get to the user is a much more obviosu step than ad-blockers in web browsers.

Sitting nervously on the fence

Simon Willison — 2003-07-11T21:02:11-00:00

Today's hot topic is the Winer Watcher, Mark Pilgrim's new tool that tracks and highlights edits made to Dave Winer's Scripting News. The blogosphere is pretty much evenly split on this: some people think it is a blatant attack on Dave Winer, tantamount o blogger bullying, while others see it as a neat technical solution to a very real problem.

I've been using Technorati to follow the discussion, but the real action is in the comments attached to this entry by Don Park. To save you having to read all 80+ comments, here are some highlights (I've tried to pick out entries which represent the different opinions on display). Please note that by the very nature of this post I am quoting these people out of context. If that makes you uncomfortable, read the whole thread.

Már Örlygsson

IMO Mark is acting like a school-yard bully, picking on someone he feels will make him look big. Hrrmphf! (Surely Mark has every right to dislike Dave - or anyone else for that matter - but constantly picking on people is just plain nasty and can have terrible psychological effects on even the strongest people.)

Greg Ritter

There's editing posts and then there's what I call "de-publishing." http://www.tenreasonswhy.com/weblog/archives/2003/07/10/the_ethics_of_depublishing.html

De-publishing is when an author deletes or substantively changes a post without any sort of retraction or notice that the change has taken place. Note that I'm talking about *substantive* changes -- not fixing grammer or spelling or text formatting, but changes that affect the meaning or impact of the post.

Winer regularly writes something inflammatory and then later tries to "erase" it from existence by de-publishing it. I disapprove of that because with publishing should come accountability.

Mark Pilgrim is using Winer's RSS feeds to track the "virtual paper trail" to reveal the kind of de-publishing that takes place on Scripting News.

I find de-publishing far more unethical and detrimental to the blogosphere (especially when it comes from such a prominent blogger as Winer) than what Mark Pilgrim is doing.

Blake Winton

Am I the only person here who finds the Winer Watcher a fascinating look into the mind of a popular and experienced weblogger as he writes his posts? I read it compulsively, and find myself thinking "What changed there? Why did he rephrase that particular statement? How is the new phrasing better than the old?".

Dave Winer

"there's never a concept of a final posting"

Not true. 10PM is the final, that's when the people who subscribe via email get their copy of Scripting.

BTW, I've deleted a few paragraphs as I'm writing this post. Think about it. Did I do something wrong? That's how ridiculous this discussion is.

Mark Pilgrim

1. Diff-like highlighting of changes within posts is an incredibly useful feature that all news aggregators should support.

2. Dave's bandwidth claim is bogus. Winer Watcher uses a system of distributed mirrors and never touches scripting.com directly. If WW ceased to exist, Dave's bandwidth bill would change by precisely zero.

3. Dave's copyright claim is bogus. Winer Watcher is no more infringing than these existing syndication services:

http://www.oreillynet.com/meerkat/?c=584&t=ALL http://newsisfree.com/sources/info/336/

[...]

For the record, Winer Watcher was started because Dave wrote a series of posts totally lashing out at Blogger, Movable Type, Google, Tim Bray, and myself, and then edited them within hours to erase all traces of his own slanderous flaming. This kind of slander is NOT ACCEPTABLE UNDER ANY CIRCUMSTANCES, and the fact that he seems to know this at some level and edits/deletes it later only makes it worse. WW tracks this kind of Orwellian rewriting of history and displays it. It would be more useful if it could distinguish between a relevant edit and a typo correction, but sometimes even a single word is relevant, so I don't know how it could tell.

Bill Brown

Dave: you're free to delete anything you want and do whatever you want on your blog. It's not wrong, it's just a pain in the ass for your regular readers. I might read in the morning and revisit by afternoon to a completely different page, one who's mood and tone have changed thoroughly. I personally don't like that because I'm hardly confident in my previous recollection -- it's unsettling.

Mark Pilgrim

For the record, here is a Scripting News post he posted on July 8 2003 and subsequently deleted (but Winer Watcher caught it):

""" There's more to the story, in re Mark's control of the RSS validator. It seems people who accuse me of controlling RSS may have missed that Mark and Sam have actually been exerting silent control by changing key aspects of the validator, without telling anyone they were doing it. Mark's flaming in this thread, which caught the attention of quite a few people as being extrordinarily mean, even for Mark, was in exactly the area he wouldn't want you to look in the validator. I want to disclaim that I control RSS, folks, because since the RSS 2.0 spec was frozen, it was Mark and Sam that controlled it, not me. Ironically, no one knew. """

Rogers Cadenhead

Regardless of the legality, though, it seems particularly ill-timed. If the Echo Project is going to move anyone beyond the intractable political fights over RSS, it's counterproductive to find novel new ways to piss each other off.

The thread then devolves in to an argument about whether Mark's tool is a copyright infringment or is protected by "fair use", at which point I tuned out (there are good arguments either way on that one).

Having thought things over, I love the functionality of the tool (Dave's edits have caught me out on more than one occasion) but I am uncomfortable with the way it is being used to attack Dave's personality. Point 4 of Rebecca Blood's guide to weblog ethics is worth reading here: editing entries is best avoided, and when they are edited they should be accompanied by an addendum. Weblogs are a personal medium, but that does not absolve people from responsibility for what they have written. My own policy is to clearly mark any alterations I make to posts (with the exception of spelling mistakes), and I usually avoid making any edits at all. Dave's policy is to edit his blog "live" until 10pm. when the day's entries become frozen. It is not so much this policy that is under fire as the scale of Dave's edits, and the nature of the material he later deletes.

If Winer Watcher was available as a standalone tool, I would use it. As a public resource, it does feel a little below the belt.

RSS Links

Simon Willison — 2003-07-11T18:43:50-00:00

Here's one for budding RSS historians: Ken MacLeod's RSS Links is a collection of links relevant to the development of RSS between March 15, 1999 and August 14, 2000. Ken also provides a distilled list of the more important discussion points. Combine that with Mark Pilgrim's History of the RSS Fork (covering July 2000 to November 2000) and you've got more knowledge about RSS history than anyone could possible want to know.

Clearout

Simon Willison — 2003-07-10T13:52:43-00:00

Tristan Louis' RSS to Necho convertor puts paid to the idea that the success of one format will be detrimental to the usefulness of the other.
O'Reilly's RegExp Power series (part one and part two) demonstrate some powerful tricks for use with Perl compatible regular expressions.
Norman Walsh explains Content Negotiation and some of the pitfalls with modern browser implementations.
So that's what happened to Digitiser. See also a Digitiser Tribute and a Mr Biffo interview from 2001 for background information. I cuss you bad.
George Orwell: Politics and the English Language
Clay Shirky: A Group Is Its Own Worst Enemy. The title is misguiding; this is an essay about how online groups behave and how to look after them.
A Java HttpClient Class.
Some good stuff on Boxes and Arrows: Ten Quotable Moments: Challenges and Responses for UI Designers and Views and Forms: Principles of Task Flow for Web Applications (Part 1).
Inside our notions of "document" and Inside our documents II - the Runoff model.
5 days worth of XSLT observations from Simon St. Laurent: One, Two, Three, Four, Five.
Windows programming with open source tools: Minimalist GNU For Windows and Win32 Programming with GNU C and C++.

Throwing your money around

Simon Willison — 2003-07-09T01:18:54-00:00

Adam Curry is a dangerous man: He's throwing $10,000 at a problem he clearly doesn't understand. Quote from June 29th:

I wonder if netscape came up with the Really Simple Syndication when they hijacked the name for what they called rss1.0.

Anyone who's been following RSS even a little bit should know that Netscape had absolutely nothing to do with RSS 1.0 - they defined RSS 0.91 and that was it. Netscape certainly didn't hijack the name, and it was Dave Winer who came up with "Really Simple Syndication".

Despite questionable understanding of the situation, Adam is taking a stand by promising to spread $10,000 around in payments to aggregator authors to include him as a default feed, but only if they omit support for (N)Echo. As Rafe Colburn points out, even if you assume a complete understanding of the issues behind the creation of (N)Echo, this simply doesn't make sense.