Just A Summary :

Work with us

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 03 Jul 2008 06:00:00 -0500

If the last post about our Javascript issues didn’t put you off, then you might be interested to know that we’re hiring. If you’re an experienced, test infected Ruby on Rails programmer with some Javascript and a real world consumer website or two under your belt, and you’re happy to work in Newcastle upon Tyne, then we definitely want to hear from you. I’d probably be interested in at least hearing from you if you’re an experienced dynamic language programmer who has only recently made (or is considering making) the switch to Ruby and Rails. It’s only syntax after all.

The money’s decent, the work is interesting, the people (well, apart from me, obviously) are great, and Newcastle’s a fantastic city. Drop me a line, ping me on AIM/gTalk/Twitter or just send your CV to the jobs@amazing-media.com.

Usability testing (throws) rocks

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 03 Jul 2008 04:55:00 -0500

Usability testing is wonderful. But wow, its humiliating.

I’ve spent the last few weeks working on the Amazingtunes in page player. Amazingtunes is a music site, so we need to play music. However, we don’t like the way that most music sites work; either the music stops as you go from one page to another, or the player is a huge Flash app running in its own window. There has to be a better way. There needs to be a popup window if you want to eliminate stop/start behaviour, but there’s surely no reason not to keep the controls on the main page.

So, we set about writing somthing that did just that. We settled on using Jeroen Wijering’s excellent flvPlayer, which handles the media formats we need and has good Javascript control and communications. This sits in the child window and we use Javascript cross-window communication to have a player controller in the main window that looks something like:

Uploaded with plasq’s Skitch!

This is all done in HTML and and Javascript, the progress bar does the Safari trick of running behind the tune data links, the buttons do their AJAX magic and the whole thing is rather slick, though I say so myself.

At least, we thought it was slick until we pointed the usertesting.com legions at it. Without exception, they ignore the in page player, foreground the popup and use the teeny weeny controls on the flash player. Originally, the popup window didn’t even display any transport controls, it just had a picture of some speakers and some text asking the user not to close it because it was playing the tunes. We added transport controls as a stopgap while we made the in page player work properly.

I sound like I’m whinging don’t I? It’s certainly a blow to the ego to see something we spent so much time and attention on being ignored by our sample users. On at least one occasion, while watching the screencasts I found myself boggling at the things the users did, and if I didn’t shout “Just play some bloody music!” at the screen, then I came worryingly close.

It would be easy to retreat into a state of denial: “They’re not our target users! They’re stupid! They’re American! If they would only magically intuit the way we think they should use the site!”. And maybe it would be comforting to do so, for a while. The right thing to do is to suck it up – take away from those videos the sure and certain knowledge that bits of the site don’t work and do something about it.

We may dislike the ‘popup window for transport controls’ model of controlling music playback, but users are cool with it. And it’s not as if the work we did on making the in page player work is going to be wasted – widget is straightforwardly event driven so it’ll work just as well in the popup window, and the communication protocol will be much simpler. Having the player in its own window means we’ll be able to extend its interface in ways that would be hard when the player had to share window space with the rest of the page. In the end, it’s all good.

But… damn that in page player was sweet. I learned Javascript as I wrote it (mostly by pretending it was Perl with odd syntas) and I’m bloody proud of it. I’ll happily replace it with the next iteration (which I’m already working on), but it’ll be with a pang of remorse all the same.

Announcing Announcements for Ruby

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 29 Jun 2008 04:46:00 -0500

I’ve just pushed a just about usable (but horribly untested) port of Vassili Bykov’s very lovely Smalltalk Announcements framework onto github. It’s a very raw port at the moment (the interface isn’t what you’d call idiomatic ruby yet), but I shall be working on that soon. Documentation (beyond a synopsis in the readme file) is nonexistent, but I reckon that there’s the core of something useful there (I’ve got plans for using it in Typo as the basis of a Wordpressesque plugin architecture and I need it for my Sooper Sekrit Project too…).

Expect some more details on the whats and whys in this blog soon, but if you want to have a play, you’ll find it at http://github.com/pdcawley/announcements. Enjoy.

git is the monads

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 15 May 2008 01:57:00 -0500

When, in the course of learning about Haskell, I reached the point where I thought I understood what Monads were for, I wrote about it. In the comments, Seth Gordon observed that:

There are two kinds of people who try to learn Haskell: the people who give up because they can’t figure out monads, and the people who go on to write tutorials on how to understand monads.

I remembered this today as yet another git tutorial rolled by in my newsreader. Maybe git is the monads of version control.

What other ‘monads’ are there out there? RESTful routing in Rails seems an obvious candidate, but I’m sure there’s more.

Rails tip: Dealing with out of sync migrations

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 07 May 2008 10:09:00 -0500

Sometimes, for one embarrassing reason or another (usually involving chaotic branch merges…) a database migration can get leapfrogged. When this happens, it’s tempting to renumber the leapfrogged migration, but that breaks any servers where the migration didn’t get renumbered. Here’s how I dealt with it recently:

class MaybeOldMigration < ActiveRecord::Migration
  def self.up
    unless old_migration_applied?
      OldMigration.up
    end
  end

  def old_migration_applied?
    # Checks that the schema looks as it should
    # if the old migration got applied
  end
end

Yeah, it’s a hack, but it’s a fairly robust hack.

A quick Javascript formatting tip

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 16 Apr 2008 04:01:00 -0500

IE’s a pain. The particular pain I want to write about is its pickiness about Javascript object literals. Consider the following Javascript object:

{ success: function () {...},
  failure: function () {...},
}

If you’re used to programming in Perl or Ruby, that trailing comma’s perfectly fine, in fact leaving it there is sometimes considered good practice because it makes it easy to extend the hash, just add a new row and leave another trailing comma.

The trouble is, it’s not strictly legal to do that in Javascript. Pretty much every implementation out there will allow it though.

Except IE.

So, I’ve taken a leaf out of Damian Conways Perl Best Practices and writing my object literals as:

{ success: function () {...}
, failure: function () {...} 
}

By sticking the comma at the beginning of the line, I’m never going to make an object that breaks IE, and adding a new line to the hash is straightforward too. Just stick the cursor in front of the }, type my leading comma, space, attribute name, and hit return when I’m finished.

I’ve also started using the same practice pretty much everywhere else that I’ve got a comma separated list of things:

var foo
  , bar
  , baz
  ;
$.each( anArray
      , function () { ... }
      );

It looks weird at first, but trust me, it grows on you.

Update

In the comments, I make reference to tweaking Steve Yegge’s excellent js2-mode to handle leading comma style a little more gracefully. Since then, I’ve made it work and attached a diff to this issue on the project’s issue tracker.

Ads are gone

pdcawley@bofh.org.uk (Piers Cawley) — Tue, 15 Apr 2008 01:33:00 -0500

Back when I was writing the occasional “How do you find me?” article, I would get some weird ads showing up. On one occasion, I commented that the searcher had obviously just typed a homework question into google and expected an answer. All the ads on that page ended up being for sites that would write your essays for you.

“Hmm…” I thought, “That’s not good.”, and set about adding those advertisers to the block list.

Then, in another article, I ranted about those, ah, bastions of democracy (thank you, Tom Lehrer) who indulge in comment spamming and found myself advertising them.

Rolling back the enlightenment which discussed, amongst other things, my Uncles’ marriage after 36 years together (yay!) attracted some pretty dodgy ads too.

To my shame, none of these made me rethink the advertising, I just extended the filter and shook my head at the unpleasantness of it all.

The straw that broke the camel’s back was the advertising around Fat is an economic issue ran utterly counter to the spirit of the of the article, and they were like “whack a mole”. Every time I extended the filter, another ad for another bloody health insurance leech or some other snake oil peddler popped up.

So, screw that, the Google ads are gone – it’s not as if they were making me a huge amount anyway. I’ll continue with the left margin Amazon links though – nobody ever buys anything I recommend, but I like the pictures.

Fat is an economic issue

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 12 Apr 2008 01:41:00 -0500

Terry Pratchett once observed that a character of his was anorexic because every time they looked in a mirror, they saw a fat person. By that measure, I’m anorexic, though I tend to avoid mirrors. By more objective measures, I’m morbidly obese – 6’ tall, 346 pounds; the Body Mass Index calculation is never going to give a good number.

To listen to some sections of the press, I might as well be public enemy number one. A drain on the public purse, a morally bankrupt insult to the eye of the right thinking, Daily Mail reading healthy public. Apparently, I have the self control of an incontinent puppy and spend my time sat on my arse in front of the TV or looking at child porn on the internet, constantly stuffing my face with lard and chocolate.

Mmm… lard and chocolate…

Now, everyone’s entitled to their opinions of the morality of others. Catch me on another day, and I can rant for ages about the ugliness of the way diet and beauty industries stoke insecurities to sell product. Today I want to talk about the cost issue.

Fat people aren’t cheap to keep – we tend to die younger, and in the process of our dying we cost a lot of money. But dying can be an expensive business and however we go, we all do sooner or later. Fat people’s healthcare costs may well be higher than those of healthy people of the same age, but those healthy people could end up waiting ‘til they are 80 before dying of something horribly expensive like cancer, or Alzheimer’s.

When you add up the net lifetime healthcare costs of the average healthy person, and those of the average obese person, the healthy person costs more.

This is analogous to the “Long Tail” idea – the idea that the long tail of low sales of ‘specialist’ books/records/services adds up to being of greater value than the total sales of the top 100 or whatever. Obvious when you think about it, but running counter to traditional thinking in this area.

We’re really good at spotting the big things, but rather less good at spotting the slow accretion of small things. Each drop of water that falls from a stalactite to a stalagmite doesn’t appear to contribute anything. Nevertheless, the stalactite and stalagmite got there somehow.

I find this knowledge rather heartening. Okay, so as a fat git with diabetes, I’m likely to die young, but that’s my business. The pernicious idea that the broader population of tax payers is somehow suffering because of me and that because of that I should be denied access to health care isn’t just morally repugnant, it’s economically illiterate.

References

Lifetime Medical Costs of Obesity: Prevention No Cure for Increasing Health Expenditure from PLoS medicine. The paper describes a mathematical model rather than something which does the arithmetic on a real population, so it’s only as good as the assumptions, but it does seem compelling that the driver for high total healthcare costs over a lifetime is the length of that life. But, as I fat git, I might be expected to say that. Ho hum.

Reinventing the wheel for fun and profit

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 07 Apr 2008 06:05:00 -0500

When they tell you to stop because you’re reinventing the wheel, ignore them and carry on building a better mousetrap.

Code is data, and it always has been

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 07 Apr 2008 00:12:00 -0500

I’m just back from the first Scotland on Rails conference, and a jolly fine conference it was too. Much kudos is due to Alan, Graeme, Abdel and Paul. It was hard to believe that this was the first conference these guys have run and I think all my fellow delegates hope it won’t be the last. As I said in the Pub on Saturday night, I’d had a talk proposal knocked back and, in that situation, it’s terribly easy to find yourself sitting in a session thinking “Bloody hell, my talk would have been better than this!”, but not at this conference.

A phrase that cropped up a couple of times was the old saw that “Data == Code” – it’s very hard to avoid the idea once you start talking about code generation or Domain Specific Pidgins, parsing… I first came across the idea in The Structure And Interpretation of Computer Programs where it’s elegantly phrased as “Data is just dumb code, and code is just smart data”. Traditionally, the idea seems to be attributed to John McCarthy, the inventor of Lisp. But it’s older than that. Way older than that. The idea is actually older than Computer Science. It lies at the core of Turing’s original paper On Computable Numbers, with an Application to the Entscheidungsproblem in which invents computer science on the way to proving that it’s impossible to decide algorithmically whether a given statement of arithmetic is true or false.

In the course of the paper, Turing posits what has become known as the Halting Problem:

Given a description of a program and a finite input, decide whether the program finishes running or will run forever, given that input.

Turing’s proof runs something like this:

Suppose we have a subroutine halts?(code,data) which solves the halting problem. Let’s use that to write something like:

def counter_example(code)
  if halts? code, code
    for (;;)
    end
  else
    return
  end
end

counter_example(File.read(STDIN))

and ask the question “What happens when we run counter_example.rb < counter_example.rb”? If halts? reckons that counter_example would halt, given itself as input, then counter example will enter an infinite loop, but if halts? reckon that it would enter an infinite loop, then it would halt. Which is a contradiction. Which means that there can be no subroutine halts?, which means that maths is hard enough to be interesting and occasionally undecidable.

Look at how the proof works – it’s built around the idea that code can be treated as data. In fact, you could say that the Turing Machine looks like it does because Turing was working backwards from this core idea to describe a sufficiently powerful machine that could obviously treat it’s own description as data. Certainly when you compare the clarity of his proof that the halting problem is undecidable (given the idea of the universal Turing machine) with the contortions required to make mathematics swallow its own tail in similar fashion so that Gödel could prove his Incompleteness Theorem.

So, if you want to know who the idea that code is data is due to, the answer (as is so often the case in our field) is Turing.

Postscript

Incidentally, Turing is also responsible for the first ever bug – his original implementation of a Universal Turing Machine has a couple, one of which is probably a dumb typo (which even I could spot when I read the paper). Another is more subtle, but still fixable. Somewhat delightfully, a young grad student, (Donald W Davies, who invented packet switching) spotted these bugs and told Turing:

I … found a number of quite bad programming errors, in effect, in the specification of the machine that he had written down, and I had worked out how to overcome these. I went along to tell him and I was rather cock-a-hoop … I thought he would say ‘Oh fine, I’ll send along an addendum. But in fact he was very annoyed, and pointed out furiously that really it didn’t matter, the thing was right in principle, and altogether I found him extremely touchy on this subject.

Nearly fifty years later Davies wrote and published a debugged version of the code, which you can find in The Essential Turing. One lesson to draw from the above is that getting annoyed at people pointing out trivial bugs in example code is also at least as old as computer science. Rather splendidly, there’s also a story of the chap who wrote the first ever assembler getting a serious telling off from Turing because the computer’s time was too valuable to waste it on converting symbols into machine code when humans were perfectly capable of doing it themselves. Who knows, maybe Turing’s contention was actually true back in the days of the Manchester Baby…

Tragedy

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 24 Mar 2008 03:32:00 -0500

There’s always a moment, in a perfect tragedy, where you dare to hope that maybe the heroes are going to break the surface tension of the plot and escape. That perfect moment in Romeo and Juliet where, no matter how often you’ve seen it, you hope that this time, Juliet’s message will reach Romeo. Or, when watching Cruel Intentions, you find yourself hoping that the writers have managed to wangle a happy ending.

It never happens of course, and we’d be disappointed if it did. We are taken to the critical point, when everything seems possible, when the characters are pushed to their utmost… and fail. Give me the life and death struggles of two teenagers for whom love is everything and life isn’t worth living without it over the pat solutions of the Dream, or give me “Fortinbras, knee deep in Danes” over the cross dressing, weddings and cruel taunts of Twelfth Night. (Not that I don’t enjoy the comedies).

What brought this one?

Why am I musing on tragedy instead of code?

The explanation is simple: I just watched the last ever episode of The Wire.

If you follow the show yourself, nothing more is needed. If you don’t, why not? Okay, so if you’re in the UK you’re reduced to paying the Murdoch tax, buying the DVDs or watching through Bittorrent (and only Bittorrent is up to date), but you should. All five seasons of The Wire add up to being the best thing I’ve ever seen on television. It’s impossible to describe how good it is without misrepresenting the whole. It’s the kind of campaigning documentary fiction which would make Dickens or Mrs Gaskell proud. It’s a sprawling epic with a huge cast of fascinating and flawed characters. It’s the story of how a cop destroys his career, a junky kicks his habit, a school system fails its pupils, politicians fail their constituents, a newspaper fails its readers and how a policy of prohibition fails a country.

Prohibition and its consequences are shot through the fabric of The Wire. It’s easy to see how drugs destroy addicts. Easy (for liberal old me at least) to see how the money spent in the War on Drugs could be spent more effectively. What’s not so easy, and what The Wire does so well, is to show how “the game” destroys generation after generation of the best and brightest of the urban poor too. Why bother working to get to college, or getting a regular job when selling drugs is so easy and so profitable? If you’re going to jail for selling the stuff in the first place, why scruple to put a bullet in the head of a business rival, witness, or some mope who calls you a coward? How many “mute, inglorious Miltons” end up dead and decaying in a walled up vacant, or stuck behind the barbed wire of the state pen serving out their natural lives with no hope of parole?

And what does putting them away achieve? For every small victory, we’re shown a corresponding fall. New characters slot into the rôles they have vacated and the cycle begins again. It’s a perfect tragedy – the game is still rigged and only the players change. The gods are unmoved by the struggles of poor mortals, the lawyers get richer, incompetence is rewarded and money is siphoned away from the streets into the pockets of rich white men who already have plenty.

Welcome to Baltimore. Have a nice day.

Javascript scoping makes my head hurt

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 20 Mar 2008 11:41:00 -0500

Who came up with the javascript scoping rules? What were they smoking. Here’s some Noddy Perl that demonstrates what I’m on about:

my @subs; 
for my $i (0..4) {
  push @subs, sub { $i }
}

print $subs[0]->(); # => 0;

Here’s the equivalent (or what I thought should be the equivalent) in Javscript:

var subs = [];
for (var i in [0,1,2,3,4]) {
  subs[i] = function () {
    return i;
  }
}
alert subs[0]() // => 4

What’s going on? In Perl, $i is scoped to the for block. Essentially, each time through the loop, a new variable is created, so the generated closures all refer to different $is. In Javascript, i is scoped to the for loop’s containing function. Each of the generated closures refer to the same i. Which means that, to get the same effect as the perl code, you must write:

var subs = [];
for (var shared_i in [0,1,2,3,4]) {
  (function (i) {
    subs[i] = function () {
      return i;
    };
  })(shared_i);
}
subs[0]() // => 0

Dodgy Ruby scoping

I had initially planned to write the example “How it should work” code in Ruby, but it turns out that Ruby’s for has the same problem:

subs = [];
for i in 0..4
  subs << lambda { i }
end
subs[0].call # => 4

Which is one reason why sensible Ruby programmers don’t use for. If I were writing the snippet in ‘real’ Ruby, I’d write:

subs = (0..4).collect { |i|
  lambda { i }
}
subs[0].call # => 0

My conclusion

Javascript is weird. Okay, so you already know this. In so many ways it’s a lovely language, but it does have some annoyingly odd corners.

Baby's first screencast

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 14 Mar 2008 14:29:00 -0500

If you follow the Ruby blogs, you will probably have seen a bunch of programmers attempting to do something akin to Haskell’s maybe, or the ObjectiveC style, message eating null.

Generally, by about the 3rd time you’ve written

if foo.nil? ? nil : foo.bar 
  ...
end

you’re getting pretty tired of it. Especially when foo is a variable you’ve had to introduce solely to hold a value while you check that it’s not nil. The pain really kicks in when you really want to call foo.bar.baz. You can end up writing monstrosities like (tmp = foo.nil? ? nil : foo.bar).nil? ? nil : tmp.baz (actually, if you were to write that in production code, you probably have bigger problems). One option is to just define NilClass#method_missing to behave like its Objective C equivalent, but I’ve never quite had the nerve to find out how that might work. I wanted to write

if maybe { foo.bar.baz }
  ...
end

and have nil behave like an Objective C nil for the duration of the block, but no longer. So I wrote it. Then I thought about how to present it. I wrote the thing test first using rspec and the whole thing just flowed, but writing up a test first development process for a blog entry is painful, so I’ve made a very rough (but blessedly short) screencast of the process instead.

That’s a slightly reduced thumbnail, the movie is substantially more readable. The bottom pane of the window is the output of autotest rerunning the spec every time either the spec or the implementation changes. The top pane alternates between the specs and the implementation. Generally, every time I edit the specs, a test starts failing and every time I edit the implementation it starts passing again. In the (any) real coding run, there were of course false starts, but generally the specs kept me pretty straight.

A word or two of warning: This is a completely unedited, silent, screen cast, there are typos, backtrackings and other embarrassments. I stopped recording once I’d got 4 tests passing, but this is far from release quality (it’s perfectly usable if you know its limitations, but it’s not entirely robust).

Please let me know what you think of this. I’m aiming to make a more polished version, complete with voice over and it would be good to know which bits are confusing and need addressing in more detail in the voice over.

I am not a rock star

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 23 Feb 2008 00:13:00 -0600

I am not a rock star. I am a computer programmer. I think I’m quite a good one.

You are not a rock star either.

387,000 matches to that query. Can we all just… I don’t know… grow up please?

Mutter… grumble… chunter… I’m 40 you know!

Updates

I have it on reliable authority that James O’Kelly is a Ruby on Rails Rockstar that would make a great addition to any team!

Joined up thinking: why your resources want links

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 22 Feb 2008 02:26:00 -0600

Remember the good old days? The days before Google? The days before Altavista? The days when a 14k4bps modem was fast? Did I say good old days?

In those days, the web had to be discoverable ‘cos it sure as hell wasn’t searchable. The big, big enabling technology of the web was the humble <a href='http://somewhereelse.com'>Go somewhere else</a>. Placing the links right there in the body of the document turned out to be exactly the right thing to do.

And it continues to be the right thing to do. Consider the two pieces of YAML below the fold.

Here’s something close to what I’ve been generating at work (We generate JSON, but YAML’s easier to write out by hand):


location: http://site/users/pdcawley/tunes/99
name: Bill Norrie
creator:
  name: Piers Cawley
  location: http://site/users/pdcawley
duration: 360
image: http://s3.amazonaws.com/...
stream: http://s3.amazonaws.com/...

Here’s that same snippet, recast in the URL free style that’s common in ‘RESTful’ APIs:


id: 99
name: Bill Norrie
creator:
  name: Piers Cawley
  id: pdcawley
duration: 360
image: http://s3.amazonaws.com/...
stream: http://s3.amazonaws.com/...

Which of those would you rather have served up to your API client? Obviously, the one with the URLs because then your client has to know so much less. If resources carry links within their body they’re discoverable, just like anything else on the web. A client only needs to know about a few resource directories and maybe a mechanism for searching, and most of the rest should follow from the RESTful principles¹.

If you wanted to write a client to make use of resources in the second form, you’d need to know that a tune’s URL is of the form /users/creator.id/tunes/tune_id, that user urls have the form /users/creator.id and so on. You’d curse the API designer who designed that URL scheme. You’d curse again when, having written the Ruby API, you sat down to write the Javascript version (modern sites need their AJAX, right?) and had to teach it all about the URL scheme all over again.

Don’t Repeat Yourself

If there’s one thing worse than violating the DRY principle, it’s forcing other people to do it. Structuring your APIs representations around IDs rather than URIs is doing just that. The complexity that you dodge by punting URL generation to the user is multiplied by the number of API client implementations (and, arguably, the number of client that don’t get written ‘cos your API’s a PITA).

It’s worse than that though. You already have the code for mapping between resources and URLs. If you’re using Ruby on Rails, your mapper is bidirectional too². Generating URLs within your representations should be a snap³. So, what are you waiting for? Start writing Joined Up APIs.

¹ For values of ‘the rest’ that are concerned with the how of the API. You’re still going to have to document the what and the why. You can get a long way down that road with careful use of <link rel="ServiceDoc" href="/servicedocs/model" /> though.

² Catch me in the right mood and I can rant for ages about the awfulness of RoR’s routing system, but the bidirectional nature of them trumps almost all my complaints. It’s like Jamie Zawinski’s line that Java doesn’t have free. Everything else about the routing system can suck almost as hard as it likes, but two way routing is a win.

³ Not quite the snap it could be in Rails because rails is of the opinion that URL generation is something the controller does. Which is fine as far as it goes until you find yourself writing Tune#to_json without a controller in sight and you can’t change to_json’s signature because it’s a standard method. And you just want to cry. Model.include ActionController::UrlWriter is wrong, but so tempting… In fact I’ve succumbed, just to get the test to pass. I’m in the process of refactoring by introducing a UrlPolicy singleton that will do all that stuff and at least isolate the bits where my models get to play like controllers.

Patterns and principles

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 16 Feb 2008 09:57:00 -0600

Recently I’ve been thinking about the way that patterns on different scales interact with each other. If you read Christopher Alexander’s A Pattern Language, the first pattern in the book is Independent Regions, which are talked about within the context of a World Government, so it seems like a huge pattern. And it is, sort of, but it’s scale invariant – it applies at the level of countries, but it also applies to states, cities, neighbourhoods, streets, houses and arguably even rooms within those houses.

Or maybe it emerges from the patterns that apply at those scales. As Kipling has it:

As the creeper that girdles the tree trunk, the law runneth forward and back;
For the strength of the pack is the wolf, and the strength of the wolf is the pack.

Can we see similar scale invariant patterns in our programming practice? Of course we can, but we tend to call them principles. Programmers reading this will, I hope, be familiar with “The DRY (Don’t Repeat Yourself) Principle”, which I first came across by that name in Hunt and Thomas’s The Pragmatic Programmer. It’s such a fundamental pattern that, it suffuses every pattern in Beck’s Smalltalk Best Practice Patterns, but isn’t actually expressed as a pattern there.

There are other macro patterns, one that I’m starting to appreciate more and more is:

Fail Fast

Fail Fast is the principle that, when something starts going wrong, you shouldn’t cover it up, but raise the issue as quickly as possible, hopefully to a level where it can be dealt with. As a pattern it informs almost every activity:

I want to go and photograph the Angel of the North. I plan to be at the angel half an hour before sunrise to get that magical pre dawn golden light. But the weather forecast says tomorrow will be entirely overcast, so I scratch that plan and decide to shoot some still life stuff in diffuse window light instead.

I’m working on adding something to the work site and I realise that I’m not going to get it done in time, so I take it to the boss immediately. We work out how to reduce the scope of the change so that we’ll still have something useful, but which can/should be extended in a future iteration.

A low level method gets some data it didn’t expect and doesn’t know how to deal with, so it throws an exception and attaches what it knows about the problem – hopefully something up the caller chain will have enough information to deal with the problem.

I’m looking for a new house. I check the details to see make sure there is room for our (huge) dining table, which means the room needs to be at least 18 feet long. If the dining room isn’t big enough, I’m probably not going to like the rest of the house either, so I can reject houses quickly as I’m looking at details.

All reasonably obvious applications of the pattern, I hope you’ll agree.

It gets fun is when Fail Fast affects other patterns. For instance, there’s a pattern for choosing the name of the each block parameters. It says that you should always use the same name, usually each or ea. Many people rebel against the idea: surely it’s better to reflect the parameter’s type or rôle, or something. And they’re right, sort of. However, those are rules for naming method parameters (type suggesting) and temporary variables (rôle suggesting or ‘explaining’). If an each block gets long enough that you want to give the parameter a ‘better’ name, then it’s time give the block a name too. Pull the body of the block out into a method, ideally on the parameter’s class (which spares you the headache of naming the parameter – it’s called self – and, if you’re using ActiveSupport or something like it, you can replace the block with &:method_name.

By naming your block parameters this way, you’re applying the Fail Fast pattern. Your block becomes obviously ugly far sooner than it would if you gave it a more suggestive name, and getting ugly fast is often a good strategy. Similarly, if you’re stuck with an old fashioned for loop, call your iterator i and when the body of the loop gets unwieldy, replace it with a method or function call that takes the counter and (probably) a Collecting Parameter as arguments.

Fail Fast is why I’m using Haml more and more in personal work. Haml recasts HTML in a YAML like structure, doing away with all the line noise involved in closing tags and letting me concentrate on the structure and content of the page. In an ERB template, it’s all too easy to fall into the trap of writing complex logic in view code where it doesn’t belong. In Haml, that gets ugly quickly, which makes me factor the logic into helpers. Then, because generating markup in a helper is a pain in the arse, it’s easier to set things up so that the conditional logic simply selects which candidate partial to render. At the end of the process, the template, helpers and partials are working together, but each element is doing one thing and one thing only, and that makes for more comprehensible code. At least, it makes it more comprehensible to me.

Smalltalk people have been doing this sort of thing forever. A common complaint from new Smalltalkers is that the code editor isn’t very capable compared to, say emacs, or vi, or whichever IDE the newbie is used to. Seasoned Smalltalkers will reply that, if you’ve reached the point where you wish you had a more capable editor, the method you’re working on is probably too big. Limited text editing capability is just another way of failing faster, getting to the point where the code is telling you, loudly, that it needs to be better factored.

When getting to ugly hurts

When a pattern or programming language starts to get ugly fast if you start down a dodgy road, the programmer wins. But sometimes the wrong sort of code gets ugly. When I’m asked why I don’t code in Perl 5 any more (Ruby ‘til 6 is still my motto) I usually reply that “I got fed up of unrolling @_.”)

For those unfortunates who are unfamiliar with Perl 5, Perl subroutines are odd in that they don’t have named parameters. Almost every method ends up beginning like:

sub some_method {
  my $self = shift;
  my($other, $thing) = @_;
  ...
}

There are arguments about whether or not to use shift to pull $self off the front of the parameter array, some folk argue for my($self, $other, $thing) = @_; as the One True Way, but ~~they are heathens and should be shunned~~ it really comes down to taste and local coding standards.

The problem with this style of argument passing is that you have to do it for every bloody method. One or two lines of precious vertical space are always lost to unrolling the argument list. Vertical space is precious. Losing one or two lines of space for every method is fine when your methods are long, but well factored methods are anything but long. When your method bodies are usually 3 or 4 lines long, that repeated chunk of code is adding 25-30% to your line count, and those added lines are almost pure repetition. The temptation was always to let that method get a little bit longer, swallowing the extra complexity rather than waste another few precious lines on doing the same damned thing again. Perl 6’s implicit self and named arguments are, on the face of it at least, only minor improvements, but they’re the sort of improvements that make all the difference.

What did I miss?

I’m sure you’ve got pet examples of this pattern, things that I’ve overlooked or never thought of. Tell us about it – comment here or blog it. Let’s all start failing earlier and winning bigger.

The authentication tarpit

pdcawley@bofh.org.uk (Piers Cawley) — Tue, 29 Jan 2008 05:19:00 -0600

At work, we’re looking at adding the Atom Publishing Protocol in a few places where it makes sense. APP’s got a lot going for it – the spec is a great example of how to design a Resourceful API and is worth reading even if it’s not an immediately good fit for your application.

But…

It’s one of the givens of good application security that you don’t store passwords in clear text and you do your level best not to send them over the wire in cleartext. That way, if someone pinches your user database, they should have their work cut out for them if they want to find out what your password is (because, unless you’re very good or are using something like 1password, you probably use the same password for lots of different websites).

A decent authentication protocol should ensure that the clear text of the password is never sent over the wire and doesn’t need to be stored in the clear on the server. Also, it shouldn’t be subject to reply attacks. One way to do this is to use SSL for authenticated sessions and rely on that protocol’s encryption to solve the problem of the password going over the wire in the clear. Or you could use the standard HTDigest authentication method. The basic trick with systems that don’t send the plain password over the wire works a little like this:

Alice and Bob have agreed a secret password and a hashing algorithm.

Alice wants to prove to Bob that a particular request comes from her, so she comes up with a unique ‘nonce’ string. She then concatenates this string with the agreed secret, and generates a digest string using the agreed hashing algorithm. She attaches the nonce string and the resulting digest to her request. When Bob receives the request, he concatenates the nonce string with the agreed secret, runs it through the hashing algorithm and, if he gets the same digest value as the one attached to the request, then it’s very probable that Alice is the real requester.

There are variations with different protocols, of course, but the general rule is to send a set of inputs and a result that can only derived from the results by someone who knows the agreed secret. It’s the sort of thing your bank does when you phone them: “Can I have you postcode? Surname and initial? What’s the first letter of your password? And the last letter? Your memorable address?” The theory is that only you know how to answer those last 3 questions, but at no point are you required to say “My password is ‘flapdoodle’” loudly and clearly in a crowded restaurant. Also, it ensures that the bank employee doesn’t get to see your whole password either. Where the banks fall down is when they phone you and immediately try to take you through the security questions without doing anything to prove that they are who they claim to be.

The HTDigest protocol has a neat little wrinkle in its hashing algorithm. Instead of generating the digest directly from a combination of the user identifier¹, password and nonce string, it generates an intermediate digest from the user identifier and password, and then uses the same hashing algorithm to calculate a digest from this intermediate result and the nonce string. This means that the server can store the intermediate result instead of the plaintext password.

The problem with implementing the Atom Publishing Protocol is that one of the client apps that we want to support, Nokia’s LifeBlog mobile app, only supports the adaptation of the WSSE UserToken authentication protocol recommended by Marc Pilgrim. There’s lots to like about this protocol, especially the way it allows CGI based servers to take control of authentication without needing access to Apache’s .htaccess or requiring mod_digest to be installed. However, the design of the protocol is such that there’s no way to avoid storing the user’s password in plain text on the server. Which we really, really, really don’t want to have to do.

Mutter. Grumble. Chunter. Bloody WS-* – biting the big one again.

¹ The user identifier is a combination of a username and a ‘realm’, a little like the way that email addresses often take the form username@domain

Martin Fowler's big mouthful

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 18 Jan 2008 17:19:00 -0600

Martin Fowler is writing a book about Domain Specific Languages and, because you could never accuse Martin of a lack of ambition, he’s trying to write it in a reasonably (implementation) language agnostic fashion.

It’s fairly easy to write an implementation language agnostic book about old school DSLs, what used to be called little languages – there’s a fairly well established literature and theory to do with lexing, parsing and interpreting. These are all about algorithms, and algorithms are implementation language neutral by their very nature.

Where Martin has his work cut out for him is trying to talk about what he calls ‘internal DSLs’ and what I’ve been calling ‘pidgins’. These are the sorts of languages where you don’t write a lexer or parser but instead build a family of objects, methods, functions or whatever other bits and pieces your host language provides in order to create a part of your program that, while it is directly interpreted by the host language, feels like it’s written in some new dialect.

The Lisp family of languages can be said to be all about this. A good ‘bottom up’ lisp programmer will shape a language to fit the problem space, essentially building a new lisp which makes it easy to solve the problem at hand. Lisp’s minimal syntax, powerful macros and the way it blurs the boundary between code and data really support this style.

Once you move from Lisp to more ‘syntaxy’ languages, things get hairier. As Martin himself says

Another issue with book code is to beware of using obscure features of the language, where obscure means for my general reader rather than even someone fluent in the language I’m using. [...] this is much harder for a DSL book. Internal DSLs tend to rely on abusing the native syntax in order to get readability. Much of this abuse involves quirky corners of the language. Again I have to balance showing readable DSL code against wallowing in quirk.

He’s dead right. When I’m thinking about writing a pidgin in Ruby for instance, my first thought is usually to start with some kind of tabula rasa object which I can use to instance_eval a block. That lets me start to shape my language by lexically scoping the change:


in_pidgin do
  ...
end

But, though it’s easy to illustrate what I’d do with my tabula rasa, the implementation is somewhat tricky, and the tricks needed are unique to Ruby.

That sort of construct’s not really available to someone trying to write a pidgin in Java or Perl. In Perl, there are other odd corners of the language that can be abused to good effect. Dynamic scoping can let you ‘inject’ methods into a block even though there’s no Perl equivalent to instance_eval, or you can do some quite staggering things with the otherwise really annoying Perl function prototypes. For instance, here’s part of a Jifty definition of a persistent object:


column title => 
       type is 'text',
       label is 'Title',
       default is 'Untitled post';

column body => 
       type is 'text',
       label is 'Content',
       render_as 'Textarea';

Doesn’t look much like Perl does it? But it’s parsed and executed by perl with no source filters or eval STRING in sight. And there’s no unsightly :symbols scattered about the place either come to that.

These things all work by making the language do something unexpected, and generally, the way to do that is by knowing your host language inside out and playing with it. One of Damian Conway’s more inspired moments in recent years was List::Maker, in which the good doctor managed to find a corner of Perl where he could wedge a proper old school, complete with full on parser to build the AST, Little Language right in the heart of Perl without it looking like he was taking a plain old string and interpreting it. So, having found this odd little corner, he proceeded to implement a remarkably neat tool for building complex lists that are beyond the capabilities of Perl’s .. operator.


@odds   = <1..100 : N % 2 != 0 >;

@primes = <3,5..99> : is_prime(N) >;

@available = <1..$max : !allocated{N} >

You may not think that’s all that sexy, but, and trust me on this, it’s just gorgeous. Yet more proof that Damian Conway is an (evil) genius.

Frankly, once you’ve seen the best of the pidgins available in Perl, some of highly praised ‘DSLs’ in Ruby start to look a bit ordinary. Ruby makes a great deal of stuff that a pidgin breeder needs to do really easy. In Perl it’s often rather hard with a huge amount of hoopage to deal with. But some of the things that are hard in Perl are impossible in Ruby.

Anyhoo… coming back to my point. I do find myself wondering if Martin’s bitten off more than he can chew in attempting to write a book that covers implementing pidgins without getting bogged down in the nitty gritty of individual languages. The problem he’s facing is that different languages don’t just have different quirks, they have different idioms too. What reads naturally in the context of a Ruby program will read very weirdly in, say Java or a lisp. Any patterns of implementation beyond broad (but important) strokes like “Play to your host language’s strengths” will surely end up as language specific patterns. Designing and implementing a good pidgin is hard. Doing it effectively means getting down and dirty with your host language and its runtime structures. And that’s not the sort of thing you can cover effectively in a language agnostic book.

Martin, if you’re reading this, good luck. I think you’re going to need it. I look forward to being proved wrong.

Getting to grips with Javascript

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 18 Jan 2008 02:30:00 -0600

I’ve been busily adding AJAX features to the work website, and I got bored of writing Form handlers. I got especially bored of attaching similar form handlers to lots of different forms on a page, so I came up with something I could attach to document.body and then plug in handlers for different form types as I wrote them.

So, I wrote FormSender and set up my event handler like so:

FormSender.onSubmit = function (e) {
  if (canDispatch(e)) {
    YAHOO.util.Event.stopEvent(e);
    YAHOO.util.Connect.setForm(e.target);
    YAHOO.util.initHeader('Accept', 'application/javascript, application/xml');
    YAHOO.util.Connect(e.target.method.toUpperCase(), e.target.action,
    callbackFor(e));
  }
};

jQuery(document.body).each(function () {
  YAHOO.util.Event.addListener(this,  "submit", FormSender.onSubmit);
});

I decided to mark any Ajax dispatchable forms using a class of ‘ajax’, the idea being that it would be a simple thing to check jQuery(e.target).hasClass('ajax'), but there was a snag. We had two sorts of forms on our pages, forms built using form_for(..., :class => 'ajax') and forms built using button_to(..., :class => 'ajax'), and they attached their classes in different places. In the form_for case, the class was on the form tag, but in the button_to case, it was on the generated form’s submit field. One option would be to monkey patch button_to, or roll my own ajax_button_to, but I ended up writing canDispatch like so:

function canDispatch(e) {
  jQuery(e.target).find(':submit').andSelf().hasClass('ajax');
}

This uses jQuery to build a list of the form, and its submit button, and then checks to see if any member of that list has the class ‘ajax’.

So, we can now tell if the source of a submit event is a form we should be doing AJAX dispatch with. The next trick is to work out what needs to be done with the results of sending the form. One option is the Prototype trick of simply evaluating the returned javascript, but it often makes sense to keep the behaviour clientside and just have the server return a datastructure. I decided that the way to do this would be by adding a second class to a form which used a none default handler, and then keep a hash of callback constructors keyed by class. This made callbackFor look like:

function callbackFor(e) {
  var candidates = candidateClasses(e.target);
  for (var i = 0; i < candidates.length; i++) {
    if (FormSender.callbacks[candidates[i]]) {
      return new FormSender.callbacks[candidates[i]](e);
    }
  }
  return new FormSender.callbacks.ajax(e);
}

candidateClasses is, again, a little more complex than I’d like, by virtue of the differences between button_to and form_for differences, but still reasonably straightforward, thanks to jQuery:

function candidateClasses(element) {
  return
  jQuery(element).find(':submit').andSelf() 
    .filter('.ajax').attr('className')
      .replace(/ajax/, '').trim().split(/ +/);
}

JQuery gets the form and its submit button, then selects the tag that has the ‘ajax’ class and pulls out the full className string. The replace gets rid of ‘ajax’, trim chops any useless whitespace off either end, and split(/ +/) turns it into an array of classnames. The replace -> trim -> split pipeline has the feel of something that must already exist in some DOM interface somewhere, but I’m not sure where, so I rolled my own.

Once we have a list of classes it’s easy to just cycle through the candidates until we find one that matches a callback constructor, falling back to the default where nothing matches.

For completeness, I’ll show you my current default handler, which I expect to be extending to deal with a couple more media types and, in the case of the failure handler, more failure statuses.

FormSender.callbacks.ajax = function (e) {
    var form = e.target;
    this.scope = form;
};

FormSender.callbacks.ajax.prototype.success = function (o) {
    switch (o.getResponseHeader['Content-Type'].replace(/;.*/, '')) {
    case 'application/javascript':
    case 'application/x-javascript':
    case 'text/javascript':
        eval(o.responseText);
        break;
    default:
        YAHOO.log("Can't handle AJAX response of type " + o.getResponseHeader['Content-Type']);
    }
};

FormSender.callbacks.ajax.prototype.failure = function (o) {
    switch (o.status) {
    case 401:
        Authenticator.loginThenSubmit(this);
        break;
    default:
        switch (o.getResponseHeader['Content-Type'].replace(/;.*/, '')) {
        case 'application/javascript':
        case 'application/x-javascript':
        case 'text/javascript':
            eval(o.responseText);
            break;
        default:
            YAHOO.log("Can't handle AJAX failure response of type " + o.getResponseHeader['Content-Type']);
        }
    }
};

You’ll notice a reference to Authenticator.loginThenSubmit in the 401 handler, but that’s something I’ll save for another day.

A note on namespacing

Although I’ve been showing the various FormSender helper functions as if they were in the global namespace, in the real code they’re wrapped in a function call:

var FormSender = (function () {
  var candidateClasses = function (element) {...};
  var callbackFor = function (e) {...};
  ...
  var onSubmit = function (e) {...};
  return {onSubmit: onSubmit, callbacks: {}};
})();

I love the (function () {...})() pattern – it’s a great way of keeping your paws out of the global namespace until you really, really need to.

FormSender Benefits

Aside from the obvious benefit of drastically reducing the number of onSubmit event handlers registered with the browser, I found that using FormSender has simplified some of my response handlers. For instance, one form would get a chunk of html back from the server and would use that to replace the div that contained the form. But the new div also contained a form that needed to have Ajax behaviour, so a chunk of the handler code was concerned with reregistering onSubmit handlers for the new form (or forms). No fun. By switching to a single, body level, form handler, that problem simply disappears – so long as the new forms have the right class, they automatically get the appropriate behaviour. Result.

Obviously, FormSender is unobtrusive javascript, which is nice, and its pluggable nature means it’s easy to extend just by writing new response handlers and registering them with the FormSender object.

Future Directions

One obvious extension to FormSender is to pull out the meat of the onSubmit method into the callback object to allow for forms that don’t simply send themselves to the server. Another is to wrap my head around the workings of Javascript’s object model to make it easy to build handlers that don’t duplicate the behaviour of the default handler through the medium of copy and pasting code…

Your comments please?

I’m still very new to Javascript as a programming language and I’m sure I’m doing plenty of boneheaded things here. Please let me know if there’s things I can do to improve this, or point me at any libraries that already cover this ground.

The secret of comedy is...

pdcawley@bofh.org.uk (Piers Cawley) — Tue, 08 Jan 2008 11:46:00 -0600

... timing. You either have it or you don’t.

Does this count as good timing?

Finish up some improvements to the way Typo sweeps cached pages
Announce Typo 5.0
Go down with a horrible cough and cold that leaves you exhausted and incapable of hacking
Discover that the ‘improvements’ in Typo’s cache sweeping can, occasionally, cause it to wipe the entire Typo installation directory
Stagger out of bed. Attempt to fix problem
Release Typo 5.0.1
Discover that the fix doesn’t work
Bleargh!
Let your co-maintainer deal with the fall out before releasing Typo 5.0.2 which does fix the cache sweeper
We hope
Recover enough to write a blog entry

Okay folks, Typo 5.0.2 is out and it appears to be working. I’m running it here, and I’ve had no problems so far. I’ve still got the cold, but it’s nowhere near as horrible as it was (went to bed at 5pm on New Year’s Eve, woke up at 11am on New Years Day – first time I’ve missed the turning of the year in ages).

Typo 5 is out - and more on the future

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 30 Dec 2007 09:10:18 -0600

Right, we’ve cut a Typo 5 gem and it’s on rubyforge and heading to various mirrors I hope. Frédéric’s writing the release notification which will be appearing on Typosphere Real Soon Now.

It’s been a surprisingly tricky process – we’re now requiring Rails 2.0.2 because the workings of view_paths have changed in a way which means we can’t quite make themes with Rails 2.0 and 2.0.2 and working with the edge seems like the more sensible proposition. If you’re on the bleeding edge, you should find that you get the right Rails via svn:externals anyway.

Typo futures

Meanwhile, I’ve been playing with stet and I’ve come to the conclusion that, although there’s mileage to be had in a radically slimmed down approach to the way Typo works, I’m better off simply removing the misfeatures from Typo and building from there – there’s a surprising amount of stuff that needs to be done in a competent blogging engine that Typo gets right – starting again would be throwing the baby out with the bathwater I think.

However, this does mean that if you’re following the Typo SVN trunk, you’ll be seeing a reduction in features in the short term. We’ll be copying the current trunk to a 5-0-stable branch before we start with the featurectomies though, so if you’re just after bugfixes, you’ll be better off there.

Multiblogging

We’re aiming to have multiblogging in the next release, but we’re rethinking the how of it. Right now, the ‘Blog’ object adds a bunch of complexity to code that would be much happier simply assuming that it has the database to itself. So we’re going to look at switching to a database per blog approach, that way our core code can pretty much forget about the complexities of multiblogging, and (at least initially) anyone who wants multiblogging can get there by monkeying with configuration files – of course, we intend to add a web based admin interface once things settle down and we know how things are going to work.

Caching

Caching is always a bugbear in any typo installation. Because we want to be installable on the widest possible range of hosts, we can’t rely on the presence of handy tools like ‘memcached’. Also, some of our users are operating under some fairly severe memory and process constraints, so it makes sense to have the webserve serve static files as much as possible. Meanwhile, tools like Evan Weaver’s Interlock are pointing the way towards seriously effective fragment caching. I shall be looking into implementing something that conforms to the interlock interface, but which can use an arbitrary cache backing store for fragments and maintain a full page cache. It’ll be interesting to find out if this is doable…

Atom Publishing Protocol

ActionWebService is going to go away – it’s already in the ousted branch of the rails SVN repository, and including it in Typo to support the various different admin APIs is getting painful. So, we’re going to preempt it. We won’t be getting rid of the various XMLRPC APIs until the pain becomes too great, but we are going to be concentrating on implementing, and strongly favouring, the Atom Publishing Protocol.

Feeds for everything

In particular, we’ll be adding atom feeds for all sorts of administrative data as a means of enabling people to write external tools for, say, spam protection, comment moderation and notification tasks. Right now, there’s a great deal of computation happening on the server side every time someone, say, comments on a post – in the kind of resource limited environments some people are running Typo in, that’s too much work. Switching to a feed + APP approach should help enormously with resource utilization.

Speaking of resources…

Using the server to render article previews is… suboptimal. Expect to see a javascript based preview system akin to the one I use for comments here.

Rails 2.0 and the Future of Typo

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 16 Dec 2007 10:35:00 -0600

So, if you’ve been watching the Typo tree, you’ll see there’s been a fair amount of activity on it since Rails 2.0 got released. There’s a new default theme replacing the rather creaky ‘azure’, and a fair amount of work on getting our code compatible with the current state of Rails. As we work on this, it becomes apparent that Typo’s code is getting horribly brittle. I have said before that there’s been several places where we’ve zigged before Rails zagged, and we’re paying the price for that. It doesn’t help that our test coverage is distinctly ropy either – and I’m probably guiltier than most for letting things get into that state.

So, our goal is to get what we have cleaned up and working with Rails 2 before releasing Typo 5.0. Once that’s done, that line of code will go into maintenance mode – there are still plenty of bugs to fix and documentation to write, but I’m afraid that extending that base is becoming too much of a chore.

Which is why I have a new path in my local svk repository, //stet. I’m using this for experimental development of a new, slimmed down blogging engine that will be, first and foremost, a capable Atom Publishing Protocol host. Things like spam processing will be removed from the core of the application, but we’ll provide a suite of webservice clients that will consume the ‘unmoderated feedback’ webfeed and use APP to either approve or delete the feedback as appropriate.

Theming (at least initially) will probably be confined to Javascript and CSS changes, and I’m even thinking of exposing the sidebars as Atom collections – certainly I expect that, in the first cut, sidebars will be static – if you want content that looks dynamic you’ll have to do it via javascript.

My initial goal is to slim things down as far as I possibly can – I want to build a blogging engine that can cope with the tight memory constraints of shared hosting by off loading much of the heavy lifting to client boxes. After all, I have far more processing capability available to me on the laptop I’m typing this on than the slice of Site5’s hosting infrastructure that’s actually running the blog. By making things small and static, I also hope to wring good performance numbers out of the tool as well – expect aggressive page caching at the very least.

Another important goal is easy migration of Typo databases. I expect to be writing models and controllers from the ground up, but converting the database should just be a matter of running a migration.

Experimental

Of course, stet’s currently very experimental – about the only thing that’s actually written so far are a couple of routing plugins which should help radically simplify our routes.rb (expect an article’s url to change from /articles/2007/12/16/rails-20-and-the-future-of-type to /article/2007/12/16/rails-20-and-the-future-of-typo, but with a redirect in place to cater for the old style urls). I may have grandish plans for the thing, but I could equally discover that I’m off up a blind alley, in which case you can expect me to return to the current typo codebase with a few more lessons learned.

ActiveResource?

I remain unconvinced by ActiveResource as a technology. I agree with the authors of RESTful Webservices – good webservices are joined up. They take full advantage of what could be described as the defining technology of the world wide web, the URL based hyperlink to knit resources together in a discoverable fashion. An ActiveResource based webservice may well be a good HTTP citizen, but it’s still not really ‘webby’ enough for my taste. Which means the Atom Publishing Protocol will remain my friend for most of the things I hope to do with stet. It may be harder to write a good APP server, but I’m convinced that it’s a much better interface for clients, and you should always favour ease of use over ease of implementation. If nothing else, we’re aiming to have more users than developers. Many more.

Comprehensible sorting in Ruby

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 16 Dec 2007 03:00:51 -0600

Here’s a problem I first came across when I was about 13 and helping do the stock check at the family firm. The parts department kept all their various spare parts racks of parts bins. Each bin was ‘numbered’ with an alphanumeric id. We had printouts of all the bin numbers along with their expected contents and we’d go along the racks counting the bins’ contents and checking them off against the print out. What confused me at the time was the way the printouts were organized. Instead of the obvious ordering, “A1, A2, A3, ..., A99”, the lists were ordered like “A1, A10, A11, ..., A2, A20, A21, ...”. After a bit of thought I realised that the computer was sorting the numeric bits of the bin numbers as if they were just sequences of strange letters. A bit more thought made me realise why, post computerisation, people were starting to use bin numbers like “A01, A02, ...”. Computers were more important than people so, in order to make sorting things easier, just add spurious leading 0s to make the number field a fixed width and Robert’s your parent’s brother.

27 years later and computers are still crap at sorting things in a sensible fashion. Back before Moore’s Law was really kicking in, I suppose it was excusable, but surely we’ve moved past that now.

Over on the labnotes blog, there’s an example of some ruby code that attempts to do ‘human’ sorting:

module Enumerable
  def sensible_sort
    sort_by { |key| key.split(/(\d+)/).map { |v| v =~ /\d/ ? v.to_i : v } }
  end
end

It’s okay, as far as it goes. It certainly solves the parts bin problem I outlined above, but it’s not ideal. For example, you might expect ['-1', '1', '1.02', '1.1'].sensible_sort to leave the order unchanged, but what you actually get is ‘1, 1.02, 1.1, -1’. Not ideal. Let’s rewrite sensible sort as

module Enumerable
  def sensible_sort
    sort_by {|k| k.split(/([-+]?\d+(?:\.\d+)?(?:[-+]?[eE]\d+)?)/).map {|v| Float(v) rescue v}}
  end
end

That ugly regular expression should match a far wider selection of string representations of numbers. Certainly our ‘bad’ list is now sorted correctly.

But what about “a-1”, “a-2”. Using the implementation above, they’d get sorted as “a-2, a-1”, which can’t be right, can it? Let’s extend it a bit more and make sure we only worry about the ’+’ and ’-’ if they’re at the beginning of a line or preceded by whitespace.

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v}}
  end
end

And that works fine, until you find that “B” sorts before “a”. Let’s catch that as well:

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

Yay!

Oh, wait a minute, what about version numbers? How should we sort, say “perl 5.8.0” and “perl 5.10.0”? The 5.8.0 form should definitely come first… Hmm…

How about

module Enumerable
  def sensible_sort
   sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

How far down does this thing go?

I just noticed that ”.1” sorts after “1”. Time for another tweak…

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?(?:\.\d+|\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?))/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

but that doesn’t work with version numbers like ”.8.2”, ”.10.2”...

Time passes… Thorin sits down and sings about gold

I was planning on giving an extension of the regex that caught this issue as well, but I’m afraid I’ve stumped myself – I can’t do it with a single regular expression unless I can use a fixed width lookbehind assertion, but they’re only available in Perl. Of course, it’s still possible to fix it, but doing so will take more thought than I have available to me at this time on a Sunday morning. And all this is before we get onto making sure that “1/2” sorts between “0” and “1”. And phone numbers. After all, “01915551238” is ‘obviously’ the same as “0191 555 1238” and “0191 555-1238”, so they should end up next to each other in the sorted list.

It looks like this is a ‘three pipe problem’ after all. I shall probably return to this…

Can I get a witness?

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 19 Nov 2007 13:17:21 -0600

Worrying about test coverage when you’re doing Test- or Behaviour-driven development is like worrying about the price of fish in Zimbabwe when you’re flying a kite.

Your tests are there to help you discover your interface and to provide you with an ongoing stream of small bugs to fix. If you write them cleanly, and keep them well factored (you are refactoring your tests, aren’t you?) they will help to document your intent too. Ensuring that every code path is exercised might be intellectually satisfying but that satisfaction costs time, and time is money. And that’s before you start worrying about your code’s malleability. Cover the happy path and the edge cases you know how to deal with and move on. If you’ve screwed something up, it will get found during acceptance testing (or out in the wild) and you can write a few more tests to isolate the problem, fix it, and move on.

If you deliberately add a test that passes without requiring you to write another line of code, ask yourself why you bothered with it. The test isn’t isolating a bug or specifying new behaviour. If you’re lucky, it merely confirms something you already know, and if you’re unlucky, you just introduced a bug in your test suite. Better to move on and either pull a new bug off the queue, or turn a feature request into a new bug – “Feature X doesn’t work!” – fix it, refactor and move on to the next. Further down the road, you may discover that the feature you were about to exhaustively test doesn’t even need to be there, or maybe it works differently than you expected. Aren’t you glad you didn’t worry about 100% coverage then? Maybe you will need to revisit the tests and cover more cases in the future. But future you knows more about the problem domain and can do a better job of working out what the behaviour needs to be. And if future you is likely to do a worse job than you right now, you may have bigger problems than the code to worry about.

So... I was wondering

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 17 Nov 2007 12:54:23 -0600

Has anyone written an Atompub client in JavaScript yet?