DHTML and Javascript

Tamarin

Simon Willison — 2006-11-09T12:24:45-00:00

On Tuesday, the Mozilla Foundation and Adobe announced the Tamarin project, an open-source ECMAScript virtual machine based on the ActionScript engine used by Flash Player 9.

Frank Hecker's overview of what this means is useful, but the Tamarin source code itself provides this interesting piece of historical insight:

AVM+ is the ActionScript Virtual Machine

AVM+ offers an order of magnitude performance increase over the "Classic AVM" in Flash Player 7. Our performance target is 10X.

AVM+ implements ActionScript 3.0, the new version of the ActionScript language that is compliant with the ECMAScript Edition 4 standard.

AVM+ is also built for modularity. It will be part of the Flash Player, but is a self-contained module which can be incorporated into other programs with ease. It may also be submitted to the ECMA standards organization as a reference implementation of ECMAScript Edition 4.

Adobe's reputation for solid engineering shines through here - it seems that what is now Tamarin was designed for integration with other applications from the very start.

The most important thing we can expect from this is a serious improvement in JavaScript performance in the Mozilla family of products, thanks to Tamarin's JIT compiler that can convert ECMAScript bytecode to machine code at runtime. JavaScript/Ajax applications will run faster, and the Mozilla applications themselves will perform better as much of their UI is written in JavaScript and XUL.

This performance boost will benefit other applications as well. Tamarin is being integrated with SpiderMonkey, which is used in a variety of applications such as the Yahoo! Widget Engine.

Reading through Brendan Eich's technical overview of Mozilla 2, it looks like the Mozilla team also plan on taking advantage of this performance boost to move code from C++ to JavaScript 2 in many places, simplifying their code base and reducing the likelihood of security flaws in the code.

Even in these buzzword filled days of Ajax and Web 2.0, JavaScript is still seen as a poor cousin to so-called "real" programming languages. With a high performance open-source VM like Tamarin available, maybe more developers will start to re-examine JavaScript's role outside the browser.

Notes on JavaScript Libraries

Simon Willison — 2006-06-26T15:03:00-00:00

@media 2006 was a blast. Great talks, great people and some of the highest production values I've ever seen at a conference (check out the bags!).

I sat on the JavaScript Libraries: Friend or Foe? panel, with PPK, Dan Webb, Stuart Langridge and Cameron Adams chairing. It was my first participation in a panel and I really enjoyed it - I've always liked the Q&A bit of giving talks. JavaScript Libraries are an enormous topic but I felt we did them justice considering the time available. For the record, here are the key points that I wanted to get across:

Large JavaScript applications need some kind of library - even if it's one built especially for that application. There are a number of problems in JavaScript (most of them originating with browser incompatibilities) that any moderately complex application will need to deal with - things like normalised event handling, DOM node selection, sane animation or drag and drop. Solve these problems once so you can get on to the interesting task of building the application. If you can find a library that solves them for you so much the better!
The big four. Dojo, MochiKit, Prototype/Scriptaculous and the Yahoo! UI Library are the top of the pile as far as I'm concerned. They cover the bases effectively and each one offers something interesting that makes it worth studying in its own right. If you plan to evaluate some existing libraries these make an excellent starting point.
Leaky abstractions. Joel Spolsky's essay The Law of Leaky Abstractions is my favourite of all of his online articles. It's basic tenet is that abstractions that are designed to save the programmer time inevitably leak, and if they leak at a lower level of abstraction than the programmer is familiar with they prove almost impossible to debug. Paradoxically, the more time saving abstractions you are using the more you actually have to know.

JavaScript is possibly the leakiest abstraction of them all, thanks to the many different browser environments it runs in. If you rely on a library to abstract away the browser bugs from you you are certain to run in to a bug that you can't fix sooner or later. Don't use libraries as crutches; if you're not prepared to figure out what the library is doing for you you'll end up in a world of pain further down the line.
Community and documentation matter. As with all open-source software, it's a good idea to get a feel for the amount of community activity around a project before you commit to building on it. The big four all have active communities, which means less bugs, more support and a faster rate of improvement. Likewise, good documentation is invaluable.
Libraries should play well with others. Reusable code that excludes other code from being reused has severely limited long-term value. In JavaScript, that means that libraries that mess with Object.prototype or pollute global namespaces should be treated with caution. The Prototype library was a major culprit here, but thankfully has cleaned up its act (at least with regards to altering Object.prototype) in recent releases. It's all very well saying that you'll always be using code built with your core language modifications in mind, but you may well change your tune when you try to incorporate Yahoo! or Google Maps and everything breaks.
Go with the grain. If you take the time to learn it properly, JavaScript is a powerful and surprisingly elegant language. Good JavaScript code takes advantage of its dynamic, functional nature. Libraries that promise to take the pain out of JavaScript by writing the code for you probably aren't as smart as they seem. Abstractions leak.

Dan Webb has a good comparison of the big four on SitePoint, although he did overlook YUI's Animation library which I consider a highlight.

For more accurate coverage of the @media panel itself, take a look at my co-worker Paul Hammond's detailed notes.

Notes from my Yahoo! UI Library talk

Simon Willison — 2006-05-18T13:48:22-00:00

I gave my talk on the Yahoo! User Interface Library here at XTech on Tuesday. There's so much great stuff in the library that cramming it all in to 45 minutes proved impossible, so I ended up focusing on the utilities (dom, event, connection, animation and dragdrop) and providing an overview of the controls at the end.

As with my JavaScript tutorial at ETech I made written notes before starting to compile the slides. The notes are incomplete (they abruptly end half way through the dragdrop section - at which point I decided to concentrate on putting the slides together) but should still be useful for people who haven't looked at the Library in detail:

Preparatory notes for "The Yahoo! User Interface Library"

I've also uploaded the slides as a 240 KB PDF.

Speaking gigs

Simon Willison — 2006-05-01T23:23:21-00:00

I've been doing a fair amount of public speaking recently, based on the principle that the only way to get good at it is to get a lot of practise. My last two talks were a session on Django and Web Application Frameworks at the ACCU 2006 conference and a talk on the Yahoo! Developer Network for NMK's Beers and Innovation series.

I've got a bunch more coming up. Here's my calendar for the next few months:

XTech 2006, May 16th-19th in Amsterdam

I'm chairing the Ajax Developers' Day, during which I'll also be giving a talk about the Yahoo! UI Library. The line-up for the day is pretty impressive, including representatives from a number of high-profile Ajax projects and a keynote by Alex Russell. I'm also giving a talk on Django later in the conference.

London JavaScript Night, May 25th, London

This is a night of talks that are in someway related to the JavaScript language. I'm giving an overview of the most important JavaScript libraries, how they compare and the general problems that they are trying to solve. The organisers are still looking for speakers, so drop them a line if you have something to share.

@media 2006, June 15th-16th, London

I'm on a panel with Cameron Adams, Peter-Paul Koch, Stuart Langridge and Dan Webb talking about the benefits (and drawbacks) of JavaScript libraries.

I'm also hoping to speak at LUG Radio Live in July and d.Construct 2006 in September, topics undecided.

Finally, I'm heading to a couple of geeky social events this week: Pub Standards on Tuesday and London 2.0 RC 5 (the monthly Django/Python/Rails/Everything-else meetup) on Wednesday. Both events are fun, friendly and open to all.

My ETech JavaScript tutorial

Simon Willison — 2006-03-07T05:42:23-00:00

I gave a three hour JavaScript tutorial at ETech this morning, aimed at people with previous programming experience who hadn't yet dived deep in to JavaScript as a programming language. It seemed to go pretty well - some good questions were asked at various points and a few people told me afterwards that they had found it interesting.

I didn't finish the presentation in time to get handouts made up - in fact I was adding the finishing touches 15 minutes before the session began - so I'm posting the slides here. The 111 slides (reduced to low quality JPEGs with Keynote to save on bandwidth) start here: A (Re)-Introduction to JavaScript.

For the sake of completeness, I'm also releasing the notes I made in preparation for the talk (PDF). These are pretty rough and don't match what I actually said very closely but they might be of interest in any case.

The photos in the presentation were all found using Flickr's awesome Creative Commons search feature.

Update: I've converted the notes to a single HTML file with permalinks for each of the sections: A (Re)-Introduction to JavaScript (HTML).

There's a lot of interest in the slides as a single file. The PDF is 50MB thanks to all of the images, but I'll see about getting it hosted on the ETech site. In the mean-time, I've posted higher quality copies of the slides to Flickr.

Yahoo! UI JavaScript treats

Simon Willison — 2006-02-14T14:01:12-00:00

The Yahoo! Developer Network was updated yesterday with a veritable gold-mine of Exciting New Stuff, coinciding with the launch of the brand new Yahoo! User Interface Blog.

Here are some of the highlights:

Mature, extensively tested cross-browser JavaScript libraries for animation, event handling, DOM wrangling, XMLHttpRequest and Drag and Drop.
CSS-skinnable UI controls built on those libraries: Calendar, Slider and TreeView.
A library of documented design patterns for modern web applications.
A description of Yahoo!'s Graded Browser Support policy, which should be of great interest to anyone who cares about Web Standards, semantic HTML, accessibility, progressive enhancement and unobtrusive JavaScript.

The code is all under a BSD Open Source license, which means you can use it freely in your own projects, including for commercial development.

This release represents the culmination of more than 9 months of work by an incredibly talented team. Eric Miraglia provides some behind-the-scenes insight in to the work that has gone in to the code.

I've been playing with this internally quite a bit over the past few months and I keep finding new useful bits and pieces. Here are some of my favourite details:

The Event library normalises the event model across multiple browsers, including workarounds for many of the weird frustrations faced when dealing with Safari. It cleans up events when the page is unloaded, which guards against a whole swathe of memory leak problems in IE. Best of all, if you assign an event to the DOM ID of an element that has not yet been loaded, the library keeps trying to attach the event (polling) until it succeeds or the page finishes loading.
The CustomEvent library implements the publish-subscribe/listener pattern for you to use within your own scripts.
The Dom library's getStyle and setStyle methods include cross-browser support for the CSS 3 opacity property.
The Animation library uses a global Animation Manager object to oversee all animations on the page. This means it can adjust the frame-rate depending on how much stuff is happening - so if you tell a block to move across the screen in 1 second it will happen in one second no matter how much other stuff is going on at the same time.
Here's a surprisingly tricky question: when does a drag start? When the user presses down on the mouse they might be starting a click rather than a drag. The Drag and Drop library defaults to starting a drag after a user either clicks and moves 3 pixels or more, or holds the mouse down for over a second.
The Drag library also makes it easy to have elements "grouped" so that they can only interact with other elements in the same group - see this demo for an example.

And finally, I couldn't resist quoting this bit of code from one of the Animation examples:

var attributes = {
   width: { to: 100 },
   height: { to: 100 },
   fontSize: { from: 100, to: 50, unit: '%' },
   opacity: { to: 0.5 }
}
var anim = new YAHOO.util.Anim('size', attributes, 1,
  YAHOO.util.Easing.easeOut);

A big thanks to the team that put this all together.

Escaping regular expression characters in JavaScript

Simon Willison — 2006-01-20T12:19:13-00:00

JavaScript's support for regular expressions is generally pretty good, but there is one notable omission: an escaping mechanism for literal strings. Say for example you need to create a regular expression that removes a specific string from the end of a string. If you know the string you want to remove when you write the script this is easy:


var newString = oldString.replace(/Remove from end$/, '');

But what if the string to be removed comes from a variable? You'll need to construct a regular expression from the variable, using the RegExp constructor function:


var re = new RegExp(stringToRemove + '$');
var newString = oldString.replace(re, '');

But what if the string you want to remove may contain regular expression metacharacters - characters like $ or . that affect the behaviour of the expression? Languages such as Python provide functions for escaping these characters (see re.escape); with JavaScript you have to write your own.

Here's mine:


RegExp.escape = function(text) {
  if (!arguments.callee.sRE) {
    var specials = [
      '/', '.', '*', '+', '?', '|',
      '(', ')', '[', ']', '{', '}', '\\'
    ];
    arguments.callee.sRE = new RegExp(
      '(\\' + specials.join('|\\') + ')', 'g'
    );
  }
  return text.replace(arguments.callee.sRE, '\\$1');
}

This deals with another common problem in JavaScript: compiling a regular expression once (rather than every time you use it) while keeping it local to a function. argmuments.callee inside a function always refers to the function itself, and since JavaScript functions are objects you can store properties on them. In this case, the first time the function is run it compiles a regular expression and stashes it in the sRE property. On subsequent calls the pre-compiled expression can be reused.

In the above snippet I've added my function as a property of the RegExp constructor. There's no pressing reason to do this other than a desire to keep generic functionality relating to regular expression handling the same place. If you rename the function it will still work as expected, since the use of arguments.callee eliminates any coupling between the function definition and the rest of the code.

JSON and Yahoo!'s JavaScript APIs

Simon Willison — 2005-12-16T00:10:57-00:00

I had the pleasure yesterday of seeing Douglas Crockford speak about JSON, the ultra-simple data interchange format he has been promoting as an alternative to XML. JSON is a subset of JavaScript, based around that language's array, string and object literal syntax.

As of today, JSON is supported as an alternative output format for nearly all of Yahoo!'s Web Service APIs. This is a Really Big Deal, because it makes Yahoo!'s APIs available to JavaScript running anywhere on the web without any of the normal problems caused by XMLHttpRequest's cross domain security policy.

Like JSON itself, the workaround is simple. You can append two arguments to a Yahoo! REST Web Service call:


&output=json&callback=myFunction

The page returned by the service will look like this:


myFunction({ JSON data here });

You just need to define myFunction in your code and it will be called when the script is loaded. To make cross-domain requests, just dynamically create your script tags using the DOM:


var script = document.createElement('script');
script.type = 'text/javascript';
script.src = '...' + '&output=json&callback=myFunction';
document.getElementsByTagName('head')[0].appendChild(script);

In long running apps you'll want to work out some kind of cleanup system to remove script tags from the DOM once they have been executed. More on this technique here.

It should be noted that del.icio.us has had APIs like this for a while.

Canvas demos

Simon Willison — 2005-10-05T16:43:36-00:00

Jesse Andrews (of Book Burro and userscripts.org fame) has built some awesome canvas demos for users of Safari or Firefox 1.5. He has a bar chart and some animated rectangles, but the real gem is the live chart which polls a server using XMLHttpRequest and updates a line graph with live data. He also has some fun mathematical experiments: a cellular automata generator and a neat exploration of Lindenmayer systems (both static and interactive). Read more on his blog.

More fun with the monkey

Simon Willison — 2005-09-17T19:44:56-00:00

Cory Doctorow points to America from the Great Depression to World War II: Color Photographs from the FSA-OWI, 1939-1945, with the following observation:

Unfortunately, the organizational back-end for this is so primitive (especially in comparison with modern image-sharing and organizing sites like Flickr) that it, too, seems to hail from 1939-1945, making the site a real pain to navigate and use.

Thanks to Greasemonkey, great content spoiled by poor navigation is a solvable problem.

americanmemoryfixer.user.js includes the following improvements:

Changes the colour scheme to black-on-white, and the typeface to Verdana.
Removes all table borders.
Adds headings to some pages, and fixes various title tags.
Sets the default gallery view to be a set of thumbnails, rather than a list of names.
Displays a large image (as opposed to a thumbnail) when you view a photograph.

The scariest hack in the script is the way in which subject page titles are passed around. In the current site, if you visit a category page (such as Sharecroppers) the title of the category is not displayed on the page - even though it has been passed as a parameter in the URL. If you click a link (say to the Galley thumbnail page) the information in the URL is lost as well.

My solution was to extract the subject from the URL on that first page, then rewrite the other links to include an additional "&subject=Sharecroppers" parameter. This new parameter is ignored by the CGI scripts that power the site, but my Greasemonkey script watches out for it on subsequent pages and uses it to display a title (and further propagate it to other links on the page). It's not a very robust solution, but it's good enough.

There's plenty of scope for further improvement - if you want to use my script as a starting point, please feel free.

Firefox 1.5 developer highlights

Simon Willison — 2005-09-11T13:46:19-00:00

Firefox 1.5 Beta 1 is out, and is the most exciting browser release in a very long time. It comes with the Gecko 1.8 rendering engine, which includes a ton of interesting new features. New in this version (unless you've been tinkering with the Deer Park series):

SVG
Canvas
CSS3 Columns
JavaScript 1.6

Let's take a look at these in turn.

SVG

The Mozilla SVG project has been quietly chugging along for years now, but until quite recently you needed to download an entirely separate build of Mozilla to try it out. Firefox 1.5 is the first Firefox release to include it and opens up a whole new world of vector graphics to web developers. The Croczilla Examples page is a great place to start - be sure to try the interactive demos near the bottom. There's more about Mozilla's SVG support on the Mozilla Developer Centre.

Canvas

The <canvas> element introduces a modern 2D drawing API to client-side development, with the promise of a 3D API in the future. It was invented by Apple for use in Dashboard widgets, then written up as a specification by the WHATWG to facilitate compatible implementations in other browsers.

The potential for innovation with <canvas> is enormous. Demos are pretty thin on the ground at the moment, but this interactive demo by Darin Fisher hints at the things to come. The Canvas shell is handy for playing with the API. Devmo has the beginnings of a great Canvas tutorial. This thing (originally a Safari 1.3 demo) is pretty too.

The ability for Firefox extensions to render web pages to images using <canvas> should lead to some very cool new extensions in the future.

CSS3 Columns

The ability to create auto-flowing columns (famously achieved on IHT.com using terrifyingly hairy JavaScript) has long been desired for the web - among other things it's a neat way of keeping line widths comfortably readable. The CSS3 Multi-column layout module enables this ability, and Gecko 1.8 implements parts of it in the form of the -moz-column-count, -moz-column-width and -moz-column-gap properties - the -moz- prefix is W3C approved syntax for vendor-specific CSS extensions. You can see columns in action on Robert O'Callahan's blog (Robert implemented column support in Gecko). Devmo's column documentation has further details.

JavaScript 1.6

According to New in JavaScript 1.6, the new features are E4X support, Array extras and "Array and String generics". I can't find anything detailing what the latter means, but the first two are pretty interesting. E4X (aka ECMAScript for XML) is a relatively new specification which promotes XML to a native data type. It lets you do things like this:


var x = <foo><bar>baz</bar></foo>;
alert(x.bar);
x.monkey = "good";
alert(x.toXMLString());

for each (var child in x) {
  alert(x.toXMLString());
}

(Example excerpted from Aaron's post to the Greasemonkey mailing list.)

It isn't available within script tags by default: you have to explicitly turn it on with <script type="text/javascript;e4x=1">, presumably to preserve backwards compatibility with existing pages.

Array extras consist of some new methods on the Array class, the most interesting of which are every(), forEach(), map(), filter() and some(). Each of these takes a function reference (i.e. a callback) as an argument and applies it to each item in the array in some manner. Fans of functional programming techniques will welcome these, and they can easily be enabled in older browsers by conditionally modifying Array.prototype. WebReference.com has a tutorial covering the new array methods.

If anyone knows what "Array and String generics" are, please leave a comment!

The above are just the highlights of Firefox 1.5 from a web developer's perspective. The improvements for regular users are also notable, including re-orderable tabs, caching of browser state for lightning-fast back and forward and lots of cosmetic (and performance) improvements for the Mac version. An unofficial changelog is available courtesy of the Burning Edge.

Exciting times indeed.

Understanding the Greasemonkey vulnerability

Simon Willison — 2005-07-20T03:09:19-00:00

If you have any version of Greasemonkey installed prior to 0.3.5, which was released a few hours ago, or if you are running any of the 0.4 alphas, you need to go and upgrade right now. All versions of Greasemonkey aside from 0.3.5 contain a nasty security hole, which could enable malicious web sites to read any file from your hard drive without you knowing.

Unfortunately, 0.3.5 disables all of the GM_ API functions, without which many of the more interesting user scripts out there simple won't work. This is a temporary measure - the GM_ functions should return in a later release, once the security problem with them has been resolved.

I'm going to explain how the vulnerability works, because it illustrates a number of interesting concepts in both web application security and JavaScript.

Same-origin policy

JavaScript has always enforced a same-origin policy for scripts loaded over the internet. This originally applied to scripting between frames (and iframes): a script loaded from a certain domain is only allowed to access the DOM of other pages loaded from that same domain. The same restriction has been extended to XMLHttpRequest - you are only allowed to make an XMLHttpRequest call back to the domain from which the script was originally loaded.

This policy exists to prevent cross-domain attacks. Say for example you work for a company with an intranet hidden behind the firewall, full of interesting proprietary information. Without the same-origin policy, malicious sites that you visit on the public internet would be able to read information from your intranet, using your browser as the middle-man.

GM_xmlhttpRequest

The GM_xmlhttpRequest API function does not have this restriction - it can load data from any domain. This enables a whole host of interesting user scripts - the most famous of which is probably Book Burro, which shows comparison prices from different online stores for the item you are currently looking at on Amazon, Barnes and Noble and more.

Malicious user scripts could use this feature to steal information from your private intranet, but malicious user scripts could also do all manner of other nasty things - stealing your Hotmail password for example. This is why you should never install a user script from an untrusted source without first reviewing the code.

Restricting API functions to user scripts only

To keep things safe then, it is essential that the GM_ family of API functions can only ever be used by user scripts, not by code running on pages that you have visited. By installing a user script you have declared it trustworthy - but visiting a web page does not carry that contract.

The way the flawed versions of Greasemonkey do that now is simple: the GM_ functions are added to the JavaScript global object (which is the window object), the user scripts for the page are "injected" using dynamically created <script> elements, they run, then the GM_ functions are removed from the global object to prevent scripts on the page from accessing them. This works because Greasemonkey injection and execution happens just before the onload event is fired - which is when most well behaved scripts kick in.

Object.watch()

Here's the clever part: JavaScript 1.5 defines a method of the Object class (which is inherited by all other JavaScript objects) called watch. Watch is extremely powerful: it lets you register some code to be executed when a property on some other object is assigned. This is the key to the Greasemonkey vulnerability - by watching the window object for the point at which Greasemonkey adds the API functions, a malicious script can use those functions at the moment they are attached.

The file:// protocol

Here's the final piece of the puzzle: the file:// protocol in Firefox allows you to view files and directory listings in your browser. Unfortunately, it also allows the GM_xmlhttpRequest function to do the same. It's not at all hard for a malicious script to use the function to load in files at a known location - or even load in directory listings (as HTML), parse them and use them to find all kinds of things scattered around your hard drive.

Solving the problem

The principle problem then is the requirement for "safe" Greasemonkey API functions - that is, functions that can be used by the user scripts but not by code running on a website. Aaron is looking in to this right now - it looks like the solution will require a minor change to be made to many existing scripts, but the trade-off in terms of security is more than worth it. The GM_xmlhttpRequest function will also be modified to disallow file:// URLs.

Until then, 0.3.5 is the only safe version of Greasemonkey.

Tweaking Wikipedia

Simon Willison — 2005-06-06T18:13:44-00:00

Does anyone know why Wikipedia displays a redirected page at the same URL rather than using a proper HTTP redirect? Case in point: Topics in human-computer interaction actually displays the content from List of human-computer interaction topics (that's my next exam topic) - the same content appears at two different URLs. Yuck. Here's a Greasemonkey script to fix it: wikipedia-redirect.user.js.

While I'm at it, Wikipedia's search function is painfully slow. Here's another user script which changes their search button to search the site using Google instead. It also swaps the positions of the "Search" and "Go" buttons, and makes "Search" the default action for the form: wikipedia-googlesearch.user.js.

Stuart's book

Simon Willison — 2005-06-03T13:44:02-00:00

I meant to mention this earlier, but Stuart's book, DHTML Utopia: Modern Web Design Using JavaScript & DOM, has been published. I worked as a technical editor on the book, and I'm proud to have been associated with it. Don't worry about the hairy title (apparently you have to have DHTML in it or bookshops won't know where to put it / people won't know what it's about), the inside is pure gold. In their usual style, SitePoint have posted the first four chapters online for your perusal so you don't have to take my word for it, you can try it out for yourself.

Fixing web applications with Greasemonkey

Simon Willison — 2005-05-24T01:59:40-00:00

In Greasemonkey FUD, I highlighted the importance of Greasemonkey as a tool for fixing interface problems in "enterprise" web applications. DJ Adams has done exactly that for OSS Notes, part of the SAP service portal. His user script ditches the frames in the interface, makes the page title more useful and adds hyperlinks to other note references on the page - significantly improving the user experience in less than 40 lines of code. The improvements are clearly explained in the accompanying screencast.

Spotted via Jeremy on the Greasemonkey mailing list.