Content Management

Exciting developments with Django

Simon Willison — 2005-08-03T16:56:34-00:00

The amount of activity surrounding the Django web framework since its not-quite release a few weeks ago is amazing. Adrian, Jacob and Wilson have been working over-time, with 395 check-ins to source control since the 13th of July. They've added WSGI support, a development web server, unit-tests, a ton of documentation, SQLite support, database introspection and dozens of other feature tweaks and bug fixes. Check out the project Timeline for an idea of just how frenetic things have been.

The emerging Django community has been kicking in as well. There's a significant community-led initiative to get internationalisation and localisation going, and a wide number of unofficial tutorials have emerged to complement the one on the site.

Here's where things get really interesting: changes at the Journal-World have kick-started the Django job market. Rob Curley, formally in charge of the World Company's web activities, recently took up a new position at the Naples Daily News in Florida. Rob just hired Eric Moritz, a regular on the #django IRC channel, to work on Django-powered projects there.

Meanwhile, Adrian Holovaty has taken a new job at the Washington Post as "Editor, Editorial Innovations" - a role that is sure to involve some very innovative use of Django (Adrian built chicagocrime.org). Adrian's departure means that the Journal-World are looking for a new developer - here's why you should apply.

One thing's for certain: we're going to see some very exciting Django-powered sites in the next few months.

Static content generation

Simon Willison — 2003-12-13T01:10:55-00:00

Ian Bicking has an interesting pieces on using static publishing in a CMS. The choice between static and dynamic when building software for the web is a critical one, and one that I think deserves in-depth discussion.

In a dynamic site, pages are assembled "on the fly" as and when they are requested. Most PHP powered sites do this and as PHP as a technology actively encourages dynamic content creation. Generating pages dynamically allows for all sorts of clever applications, from random quote generators to full on web applications such as Hotmail.

In a static publishing system, HTML pages are pre-generated by the publishing software and stored as flat files on the web server, ready to be served. This approach is less flexible than dynamic generation in many ways and is often ignored as an option as a result, but in fact the vast majority of content sites consist of primarily static pages and could be powered by static content generation without any loss of functionality to the end user.

The most widespread example of a static publishing system I've seen is Moveable Type, which rebuilds static files for a site each time a weblog entry is added or modified - although it can be configured to serve content dynamically instead.

At first glance, the benefits of dynamic publishing are obvious. What is frequently ignored are the benefits of static publishing, at least for content-driven sites which don't have any heavy need for dynamic features. The most obvious benefit is performance; serving static files is what web servers such as Apache are optimised to do, and they can do it fast. A second advantage is reliability, as Ian explains:

A big part is that it takes the pressure off of going live. I can be sure before going live that the public website is correct. The actual CMS may explode in flames, but the site will be fine. Going live with a web application is always a stressful process, and anything that reduces the stress of that is a great benefit. As time goes on, static publishing is also a big stress reduction for the system administrator, since a simple Apache configuration is a lot more reliable under different loads and configurations than any dynamic site will be.

I've been developing dynamic sites almost exclusively for the past two or three years, but a couple of my most recent projects were static rather than dynamic. These were the LJWorld.com Coupons site and the KUSports.com photo galleries. I wanted to write both of these in Python, because doing so would make the process of transferring them over to our new mod_python powered CMS (currently in development) far less involved. Unfortunately our main production servers don't currently have mod_python configured, and we weren't overly keen on setting it up there for the sake of a couple of small projects. Instead I decided to write the administration interfaces using Python CGI scripts, but generate the actual front end pages (which would see far heavier traffic) as static files.

In addition to the performance and reliability benefits, an additional benefit is that static generation provides a simple "staging area" style feature for free. Both the coupons and the gallery interfaces allow users to make multiple changes to site content safe in the knowledge that none of the changes will become visible until the "Publish Site" button is selected. At first I was worried that this extra step could prove confusing, but in practise it allows our content producers to make changes in a safe environment, without fear of accidentally breaking the public site while they are working.

Static content generation certainly isn't appropriate for every project, but for plain content sites sites that don't need dynamic features it's a much more viable option than many people think.

Simpler content managment

Simon Willison — 2003-12-05T01:36:28-00:00

Perls of wisdom in a sea of site mismanagement, via the ever-excellent Column Two:

The great surprise of the past five years of content management is that, despite all the hundreds of systems, no clear winners have emerged. Instead, there's a growing dissatisfaction with the ongoing technical burden that such systems impose.

Some influential voices are starting to argue that many sites should, in effect, wait out this immature phase of website management. For the moment, they should content themselves with limited automation.

The article concludes with the idea that many sites can do perfectly well with a few simple Perl scripts and maybe a relational database on the back end, rather than investing in an expensive super-package that claims to be able to do anything you could possibly want. This is very sound advice. The simple fact of the matter is that many sites really don't need a complex content management platform with support for templating, user logins, workflow, versioning and a dozen other high end features. Most sites just need someone to be able to easily update them, when necessary. This is why Macromedia Contribute has been such a success - people want the ability to hit "Edit This Page", make a few changes and publish straight to their site.

I've worked on my fair share of content management systems (in fact I'm helping develop one at the moment) and out of all of the ones I've been involved in, the one I got the biggest kick out of took the shortest time to develop. It was based on Tavi Wiki, and consisted of a password protected Tavi install for the back end and a slightly modified separate install for the front end. Both installs pointed to the same database, but the front end was altered to disable all editing features and make the site look less like a Wiki. You can see the end result here.

All in all, the CMS took less than an hour to put together from start to finish. It made it easy enough for contributors with no previous knowledge of HTML to update the site (using Wiki markup) and provided us with full versioning on all content contained within the project. The final site gives very few clues that the underlying engine is a Wiki, and thanks to Tavi's ease of customisation the site design can be easily changed to look even less wiki-like. It's close to the simplest thing that could possibly work and it works just fine.

Of course, if you don't have a competent server-side programmer to hand your only option is to buy a pre-made solution, but with a half-decent programmer and a good set of tools a simple home built CMS customised to fit your needs could be a much better investment than some $100,000 one-size-fits all monstrosity.

Nvu

Simon Willison — 2003-10-30T04:46:03-00:00

Launched today by Lindows, Nvu is a new project to develop a complete "web authoring system" (aka Dreamweaver/Frontpage style WYSIWYG editor) for the Linux platform. Reading around the marketing hyperbole, What it actually is is a standalone version of Mozilla's Composer with a whole bunch of improvements and extra features, scheduled for release in early 2004.

The really good news is that Lindows have hired Daniel Glazman to work on the project full time. Daniel is the principle maintainer of Mozilla Composer and it's great to see his work being funded in this way. Improvements to Composer made for Nvu seem certain to make it back in to the main Mozilla trunk.

The not-so-good news is that the Nvu site and FAQ make no mention of a Windows version of the software. Since development is being sponsored by Lindows this shouldn't come as a surprise, but it is disappointing because the great strength of the Mozilla framework is that it supports cross-platform development.

It's also going to be interesting to see how this pans out from a web standards point of view. On the one hand, Daniel is certainly on the side of web standards. On the other hand, the current Nvu site's markup seems to come right out of 1998. I very much doubt it was created with the tool it promotes but it doesn't give off a particularly reassuring message. A free WYSIWYG HTML editor with strong standards support out of the box would be a valuable tool in helping authors create standards compliant sites. As it is, Dreamweaver is getting close but Dreamweaver also costs nearly $400.

Wild conjecturing apart, the most interesting thing about Nvu (the name's pretty terrible but at least they got a three letter domain name) may well be the features that are added to the current Composer code base. The screenshots already hint at intelligent FTP publishing support; what would be really interesting would be the integration of a simple templating system and/or client side CMS capabilities - something like a cross between Fog Creek's CityDesk and Macromedia's Contribute. Interested developers are invited to sign up for a mailing list for notification of when the first development builds are made available. This is going to be an interesting project to watch.

New content management blog

Simon Willison — 2003-09-15T11:06:25-00:00

Ideas in Technology and Publishing is a great new blog covering content management, XML and other publishing related technologies. It's less than a month old so it's still possible to read through the archives in full, which I've just done and recommend to anyone with an interest in content management.

Neat tip for clean URLs

Simon Willison — 2003-08-06T20:18:04-00:00

Here's one of the neatest tips for clean URLs I've seen yet, from Thijs van der Vossen. He's come up with a mod_rewrite rule that checks to see if the requested file exists if you add .html on to the end of it, and serves it up if that's the case. I'm posting the full code snippet here because it's just too good to risk losing to link-rot in the distant future:

RewriteEngine on 
RewriteBase /
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule (.*) $1\.html [L]

XHTML for future-proof content

Simon Willison — 2003-08-03T21:39:09-00:00

Don Park questions the benefits of emitting XHTML. In one sense, Don is right; publishing a whole site using XHTML in this day and age brings very little benefit and can cause a great deal of grief. But just because XHTML doesn't provide advantages when publishing whole sites does not mean it should be written off entirely. As I've said on this blog many times before, XHTML offers an excellent format for future-proofing site content, especially chunks of content kept in a database. Keith D. Robinson makes some excellent points along the same lines in his latest essay, Standards, Semantic Markup, Distributed Authorship and Knowledge Management:

XHTML is, at it's most basic, much simpler and easier to learn that traditional HTML 4.0. With a simple style guide, standard markup and CSS styles you can accomplish almost all the formatting a content author would need, just by knowing a handful of markup tags. Instead of trusting the CMS to sort out code from Word, for example, you can hand a content owner a cheat sheet with the basic tags outlined and trust that they can code their own content. I mean, really, how hard is is to learn 10 or so tags? Team this technique with a tool like Contribute and you've got a nice, simple and cheap process that, while doesn't store you content in a database, keeps it in a clean, standard form you can repurpose down the road.

As for ensuring entered XHTML is valid, I think this site's comment system does a pretty good job of showing how that can be achieved with only a small amount of server side effort.

Textile 2

Simon Willison — 2003-07-09T00:08:04-00:00

Textile 2 is now available for testing, courtesy of Dean Allen. Textile is one of the more popular structured-text style markup languages, which translate a simple markup language in to HTML. What made the original Textile special was that it concentrated squarely on structural markup, providing intuitive shortcuts for most structural XHTML elements. Textile 2 takes this further, but also introduces a number of presentational effects such as block alignment. Beta PHP code is available in addition to the demo.

That said, the killer feature of Textile 2 in my opinion is the fact that it has been developed as a collaborative effort between Dean Allen and Brad Choate. Brad created a Perl port of the original Textile (Mark Pilgrim did the same thing for Python) but this time round they have been working together to define the format. If they are successful, Textile could become a useful mini-standard for authoring structural markup. At any rate, since Textile is intended to be a shorthand technique to complement HTML rather than replace it it is well worth a look.

Knowledge Representation Timeline

Simon Willison — 2003-07-02T16:27:00-00:00

This is pretty impressive: A Timeline of knowledge-representation that starts at the dawn of the Universe and continues through the whole of human history right up to the present day.

Site search finally available

Simon Willison — 2003-04-25T16:55:02-00:00

I've finally got around to adding a search page to this site. It uses MySQL's full text indexing, which is extremely fast and provides good results but comes at the expense of flexibility. Search terms less than 4 letters long are ignored, and multi-word searches are handled using OR rather than AND. This nearly put me off using it, but the relevancy algorithm is excellent which I think outweighs the disadvantage of not being able to use pure AND queries.

MySQL 4.0 introduces far more powerful boolean mode full text searches which allow all kinds of modifiers and extra syntax, but this site currently runs on 3.23.54 so I can't play with those just yet. Jeremy Zawodny's article on MySQL 4 explains boolean mode and describes many other exciting new MySQL features as well.

PHP and Javascript spell checker

Simon Willison — 2003-03-18T21:50:40-00:00

Last week I commented that Sam Ruby's spell checking feature could be made even funkier with the addition of a javascript powered "corrections" menu. I spent a few hours this afternoon playing with the idea, and I've now got quite a nice proof of concept:

Spell Checker Demo

I've tested it on Phoenix and IE5 on Windows - I'll check it on IE6 later on this evening. IE5 gets the menus in the wrong place but other than that it seems to work fine in both browsers. I adapted Sam's Python code for PHP on the server side, while the client side bit is a whole lot of messing around with the DOM.

If you want to nose around the source code, take a look at this lot:

There are still a few bugs, and I haven't quite worked out an elegant way to get the menus to behave more like menus, but on the whole it's worked out pretty well.

More nukes

Simon Willison — 2003-03-12T23:55:54-00:00

[PHP|Post|myPHP]-Nuke has to be one of the most-forked open source projects in history! Xaraya appears to be a fork from Post-Nuke, which itself forked from PHP-Nuke several years ago (and I'm pretty sure there are more). They've got an interesting set of RFCs on how they intend to build the next big open source content / community management system (nothing about generating pretty URLs yet). While browsing their site I found a link to PHPXref, a powerful looking tool for generating PHP source code documentation. Unsurprisingly for such a lot of text munging, it's written in Perl ;)

Update: I obviously wasn't paying attention: RFC 0023: Short URL Support

Spell check in web applications

Simon Willison — 2003-03-08T17:23:17-00:00

Sam Ruby has enabled spell checking for the preview comment tool on his blog. I wonder how it works... I've lost track of the scripting language Sam uses for Intertwingly (PHP? Python? Perl?) but I know PHP can be compiled with support for the Pspell module.

Sam's user interface is pretty neat - misspelled words are marked up with a span, underlined in dashed red and have suggested spellings listed in the span's title attribute. Theoretically, it should be possible to build a javascript right-click menu offering alternatives instead (preferably dynamically generated from the list of words in the title attribute using the DOM). Actually modifying the preview textarea text based on the menu selection would be quite a lot harder - it could be done with a simple search-and-replace operation, but doing so might change other words with the same "incorrect" spelling without the user realising.

It would be fun to integrate something like this with a rich text editor, such as the recently announced htmlArea 3.0 that works with Mozilla 1.3b as well as IE (more information here).

Vector search engines

Simon Willison — 2003-03-01T13:07:18-00:00

Building a Vector Space Search Engine in Perl:

Vector-space search engines use the notion of a term space, where each document is represented as a vector in a high-dimensional space. There are as many dimensions as there are unique words in the entire collection. Because a document's position in the term space is determined by the words it contains, documents with many words in common end up close together, while documents with few shared words end up far apart.

To search our collection, we project a query into this term space and calculate the distance from the query vector to all the document vectors in turn. Those documents that are within a certain threshold distance get added to our result set. If all this sounds like gobbledygook to you, then don't worry - it will become clearer when we write the code.

Having done a course on Linear Algebra last term, it's interesting to see how it can be applied to the search problem. The technique described lends itself well to finding "similar documents" as well, as documents with similar word content will end up "near" to each other when projected on to the vector space.

The article is also yet another demonstration of how Perl's modules make it such a powerful tool. Lingua::Stem is used to find word "stems", providing a free algorithm for eliminating related words like cat and cats. The performance overhead of using Perl arrays to represent large vectors is avoided with the PDL module, which implements a whole set of matrix algebra functions in compiled C for high performance. Without these two modules the technique described would be a great deal less powerful. Of course, neither of them are available for PHP or Python, my scripting languages of choice.

Safe HTML checker

Simon Willison — 2003-02-23T15:04:37-00:00

I've finally enabled a subset of HTML in my comments. In doing so, I had several requirements that needed to be fulfilled:

Entered markup must be valid to XHTML strict, to stop comments form breaking validation and keep things nice and tidy.
No presentational markup! I want to maintain control over how things look via my stylesheets - comments posted should only be able to use structural HTML elements.
Attributes should be restricted to those that add semantic meaning. Javascript event attributes and CSS related attributes should not be allowed.
I should retain full control over the tags and attributes allowed in the comments.
Submitted HTML must be kept free from anything that could pose a security risk, such as javascript: URLs.

The system I have implemented works by running submitted posts through an XML parser, which checks that each element is in my list of allowed elements, is nested correctly (you can't put a blockquote inside a p for example) and doesn't have any illegal attributes. My initial test have shown it to work pretty well, but if anyone wants to have a go at breaking it please, be my guest.

The code for the main class is available here: SafeHtmlChecker.class.php