Just A Summary : Category the-practice-of-programming, everything about the-practice-of-programming

A quick Javascript formatting tip

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 16 Apr 2008 04:01:00 -0500

IE’s a pain. The particular pain I want to write about is its pickiness about Javascript object literals. Consider the following Javascript object:

{ success: function () {...},
  failure: function () {...},
}

If you’re used to programming in Perl or Ruby, that trailing comma’s perfectly fine, in fact leaving it there is sometimes considered good practice because it makes it easy to extend the hash, just add a new row and leave another trailing comma.

The trouble is, it’s not strictly legal to do that in Javascript. Pretty much every implementation out there will allow it though.

Except IE.

So, I’ve taken a leaf out of Damian Conways Perl Best Practices and writing my object literals as:

{ success: function () {...}
, failure: function () {...} 
}

By sticking the comma at the beginning of the line, I’m never going to make an object that breaks IE, and adding a new line to the hash is straightforward too. Just stick the cursor in front of the }, type my leading comma, space, attribute name, and hit return when I’m finished.

I’ve also started using the same practice pretty much everywhere else that I’ve got a comma separated list of things:

var foo
  , bar
  , baz
  ;
$.each( anArray
      , function () { ... }
      );

It looks weird at first, but trust me, it grows on you.

Update

In the comments, I make reference to tweaking Steve Yegge’s excellent js2-mode to handle leading comma style a little more gracefully. Since then, I’ve made it work and attached a diff to this issue on the project’s issue tracker.

Code is data, and it always has been

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 07 Apr 2008 00:12:00 -0500

I’m just back from the first Scotland on Rails conference, and a jolly fine conference it was too. Much kudos is due to Alan, Graeme, Abdel and Paul. It was hard to believe that this was the first conference these guys have run and I think all my fellow delegates hope it won’t be the last. As I said in the Pub on Saturday night, I’d had a talk proposal knocked back and, in that situation, it’s terribly easy to find yourself sitting in a session thinking “Bloody hell, my talk would have been better than this!”, but not at this conference.

A phrase that cropped up a couple of times was the old saw that “Data == Code” – it’s very hard to avoid the idea once you start talking about code generation or Domain Specific Pidgins, parsing… I first came across the idea in The Structure And Interpretation of Computer Programs where it’s elegantly phrased as “Data is just dumb code, and code is just smart data”. Traditionally, the idea seems to be attributed to John McCarthy, the inventor of Lisp. But it’s older than that. Way older than that. The idea is actually older than Computer Science. It lies at the core of Turing’s original paper On Computable Numbers, with an Application to the Entscheidungsproblem in which invents computer science on the way to proving that it’s impossible to decide algorithmically whether a given statement of arithmetic is true or false.

In the course of the paper, Turing posits what has become known as the Halting Problem:

Given a description of a program and a finite input, decide whether the program finishes running or will run forever, given that input.

Turing’s proof runs something like this:

Suppose we have a subroutine halts?(code,data) which solves the halting problem. Let’s use that to write something like:

def counter_example(code)
  if halts? code, code
    for (;;)
    end
  else
    return
  end
end

counter_example(File.read(STDIN))

and ask the question “What happens when we run counter_example.rb < counter_example.rb”? If halts? reckons that counter_example would halt, given itself as input, then counter example will enter an infinite loop, but if halts? reckon that it would enter an infinite loop, then it would halt. Which is a contradiction. Which means that there can be no subroutine halts?, which means that maths is hard enough to be interesting and occasionally undecidable.

Look at how the proof works – it’s built around the idea that code can be treated as data. In fact, you could say that the Turing Machine looks like it does because Turing was working backwards from this core idea to describe a sufficiently powerful machine that could obviously treat it’s own description as data. Certainly when you compare the clarity of his proof that the halting problem is undecidable (given the idea of the universal Turing machine) with the contortions required to make mathematics swallow its own tail in similar fashion so that Gödel could prove his Incompleteness Theorem.

So, if you want to know who the idea that code is data is due to, the answer (as is so often the case in our field) is Turing.

Postscript

Incidentally, Turing is also responsible for the first ever bug – his original implementation of a Universal Turing Machine has a couple, one of which is probably a dumb typo (which even I could spot when I read the paper). Another is more subtle, but still fixable. Somewhat delightfully, a young grad student, (Donald W Davies, who invented packet switching) spotted these bugs and told Turing:

I … found a number of quite bad programming errors, in effect, in the specification of the machine that he had written down, and I had worked out how to overcome these. I went along to tell him and I was rather cock-a-hoop … I thought he would say ‘Oh fine, I’ll send along an addendum. But in fact he was very annoyed, and pointed out furiously that really it didn’t matter, the thing was right in principle, and altogether I found him extremely touchy on this subject.

Nearly fifty years later Davies wrote and published a debugged version of the code, which you can find in The Essential Turing. One lesson to draw from the above is that getting annoyed at people pointing out trivial bugs in example code is also at least as old as computer science. Rather splendidly, there’s also a story of the chap who wrote the first ever assembler getting a serious telling off from Turing because the computer’s time was too valuable to waste it on converting symbols into machine code when humans were perfectly capable of doing it themselves. Who knows, maybe Turing’s contention was actually true back in the days of the Manchester Baby…

Javascript scoping makes my head hurt

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 20 Mar 2008 11:41:00 -0500

Who came up with the javascript scoping rules? What were they smoking. Here’s some Noddy Perl that demonstrates what I’m on about:

my @subs; 
for my $i (0..4) {
  push @subs, sub { $i }
}

print $subs[0]->(); # => 0;

Here’s the equivalent (or what I thought should be the equivalent) in Javscript:

var subs = [];
for (var i in [0,1,2,3,4]) {
  subs[i] = function () {
    return i;
  }
}
alert subs[0]() // => 4

What’s going on? In Perl, $i is scoped to the for block. Essentially, each time through the loop, a new variable is created, so the generated closures all refer to different $is. In Javascript, i is scoped to the for loop’s containing function. Each of the generated closures refer to the same i. Which means that, to get the same effect as the perl code, you must write:

var subs = [];
for (var shared_i in [0,1,2,3,4]) {
  (function (i) {
    subs[i] = function () {
      return i;
    };
  })(shared_i);
}
subs[0]() // => 0

Dodgy Ruby scoping

I had initially planned to write the example “How it should work” code in Ruby, but it turns out that Ruby’s for has the same problem:

subs = [];
for i in 0..4
  subs << lambda { i }
end
subs[0].call # => 4

Which is one reason why sensible Ruby programmers don’t use for. If I were writing the snippet in ‘real’ Ruby, I’d write:

subs = (0..4).collect { |i|
  lambda { i }
}
subs[0].call # => 0

My conclusion

Javascript is weird. Okay, so you already know this. In so many ways it’s a lovely language, but it does have some annoyingly odd corners.

Joined up thinking: why your resources want links

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 22 Feb 2008 02:26:00 -0600

Remember the good old days? The days before Google? The days before Altavista? The days when a 14k4bps modem was fast? Did I say good old days?

In those days, the web had to be discoverable ‘cos it sure as hell wasn’t searchable. The big, big enabling technology of the web was the humble <a href='http://somewhereelse.com'>Go somewhere else</a>. Placing the links right there in the body of the document turned out to be exactly the right thing to do.

And it continues to be the right thing to do. Consider the two pieces of YAML below the fold.

Here’s something close to what I’ve been generating at work (We generate JSON, but YAML’s easier to write out by hand):


location: http://site/users/pdcawley/tunes/99
name: Bill Norrie
creator:
  name: Piers Cawley
  location: http://site/users/pdcawley
duration: 360
image: http://s3.amazonaws.com/...
stream: http://s3.amazonaws.com/...

Here’s that same snippet, recast in the URL free style that’s common in ‘RESTful’ APIs:


id: 99
name: Bill Norrie
creator:
  name: Piers Cawley
  id: pdcawley
duration: 360
image: http://s3.amazonaws.com/...
stream: http://s3.amazonaws.com/...

Which of those would you rather have served up to your API client? Obviously, the one with the URLs because then your client has to know so much less. If resources carry links within their body they’re discoverable, just like anything else on the web. A client only needs to know about a few resource directories and maybe a mechanism for searching, and most of the rest should follow from the RESTful principles¹.

If you wanted to write a client to make use of resources in the second form, you’d need to know that a tune’s URL is of the form /users/creator.id/tunes/tune_id, that user urls have the form /users/creator.id and so on. You’d curse the API designer who designed that URL scheme. You’d curse again when, having written the Ruby API, you sat down to write the Javascript version (modern sites need their AJAX, right?) and had to teach it all about the URL scheme all over again.

Don’t Repeat Yourself

If there’s one thing worse than violating the DRY principle, it’s forcing other people to do it. Structuring your APIs representations around IDs rather than URIs is doing just that. The complexity that you dodge by punting URL generation to the user is multiplied by the number of API client implementations (and, arguably, the number of client that don’t get written ‘cos your API’s a PITA).

It’s worse than that though. You already have the code for mapping between resources and URLs. If you’re using Ruby on Rails, your mapper is bidirectional too². Generating URLs within your representations should be a snap³. So, what are you waiting for? Start writing Joined Up APIs.

¹ For values of ‘the rest’ that are concerned with the how of the API. You’re still going to have to document the what and the why. You can get a long way down that road with careful use of <link rel="ServiceDoc" href="/servicedocs/model" /> though.

² Catch me in the right mood and I can rant for ages about the awfulness of RoR’s routing system, but the bidirectional nature of them trumps almost all my complaints. It’s like Jamie Zawinski’s line that Java doesn’t have free. Everything else about the routing system can suck almost as hard as it likes, but two way routing is a win.

³ Not quite the snap it could be in Rails because rails is of the opinion that URL generation is something the controller does. Which is fine as far as it goes until you find yourself writing Tune#to_json without a controller in sight and you can’t change to_json’s signature because it’s a standard method. And you just want to cry. Model.include ActionController::UrlWriter is wrong, but so tempting… In fact I’ve succumbed, just to get the test to pass. I’m in the process of refactoring by introducing a UrlPolicy singleton that will do all that stuff and at least isolate the bits where my models get to play like controllers.

Martin Fowler's big mouthful

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 18 Jan 2008 17:19:00 -0600

Martin Fowler is writing a book about Domain Specific Languages and, because you could never accuse Martin of a lack of ambition, he’s trying to write it in a reasonably (implementation) language agnostic fashion.

It’s fairly easy to write an implementation language agnostic book about old school DSLs, what used to be called little languages – there’s a fairly well established literature and theory to do with lexing, parsing and interpreting. These are all about algorithms, and algorithms are implementation language neutral by their very nature.

Where Martin has his work cut out for him is trying to talk about what he calls ‘internal DSLs’ and what I’ve been calling ‘pidgins’. These are the sorts of languages where you don’t write a lexer or parser but instead build a family of objects, methods, functions or whatever other bits and pieces your host language provides in order to create a part of your program that, while it is directly interpreted by the host language, feels like it’s written in some new dialect.

The Lisp family of languages can be said to be all about this. A good ‘bottom up’ lisp programmer will shape a language to fit the problem space, essentially building a new lisp which makes it easy to solve the problem at hand. Lisp’s minimal syntax, powerful macros and the way it blurs the boundary between code and data really support this style.

Once you move from Lisp to more ‘syntaxy’ languages, things get hairier. As Martin himself says

Another issue with book code is to beware of using obscure features of the language, where obscure means for my general reader rather than even someone fluent in the language I’m using. [...] this is much harder for a DSL book. Internal DSLs tend to rely on abusing the native syntax in order to get readability. Much of this abuse involves quirky corners of the language. Again I have to balance showing readable DSL code against wallowing in quirk.

He’s dead right. When I’m thinking about writing a pidgin in Ruby for instance, my first thought is usually to start with some kind of tabula rasa object which I can use to instance_eval a block. That lets me start to shape my language by lexically scoping the change:


in_pidgin do
  ...
end

But, though it’s easy to illustrate what I’d do with my tabula rasa, the implementation is somewhat tricky, and the tricks needed are unique to Ruby.

That sort of construct’s not really available to someone trying to write a pidgin in Java or Perl. In Perl, there are other odd corners of the language that can be abused to good effect. Dynamic scoping can let you ‘inject’ methods into a block even though there’s no Perl equivalent to instance_eval, or you can do some quite staggering things with the otherwise really annoying Perl function prototypes. For instance, here’s part of a Jifty definition of a persistent object:


column title => 
       type is 'text',
       label is 'Title',
       default is 'Untitled post';

column body => 
       type is 'text',
       label is 'Content',
       render_as 'Textarea';

Doesn’t look much like Perl does it? But it’s parsed and executed by perl with no source filters or eval STRING in sight. And there’s no unsightly :symbols scattered about the place either come to that.

These things all work by making the language do something unexpected, and generally, the way to do that is by knowing your host language inside out and playing with it. One of Damian Conway’s more inspired moments in recent years was List::Maker, in which the good doctor managed to find a corner of Perl where he could wedge a proper old school, complete with full on parser to build the AST, Little Language right in the heart of Perl without it looking like he was taking a plain old string and interpreting it. So, having found this odd little corner, he proceeded to implement a remarkably neat tool for building complex lists that are beyond the capabilities of Perl’s .. operator.


@odds   = <1..100 : N % 2 != 0 >;

@primes = <3,5..99> : is_prime(N) >;

@available = <1..$max : !allocated{N} >

You may not think that’s all that sexy, but, and trust me on this, it’s just gorgeous. Yet more proof that Damian Conway is an (evil) genius.

Frankly, once you’ve seen the best of the pidgins available in Perl, some of highly praised ‘DSLs’ in Ruby start to look a bit ordinary. Ruby makes a great deal of stuff that a pidgin breeder needs to do really easy. In Perl it’s often rather hard with a huge amount of hoopage to deal with. But some of the things that are hard in Perl are impossible in Ruby.

Anyhoo… coming back to my point. I do find myself wondering if Martin’s bitten off more than he can chew in attempting to write a book that covers implementing pidgins without getting bogged down in the nitty gritty of individual languages. The problem he’s facing is that different languages don’t just have different quirks, they have different idioms too. What reads naturally in the context of a Ruby program will read very weirdly in, say Java or a lisp. Any patterns of implementation beyond broad (but important) strokes like “Play to your host language’s strengths” will surely end up as language specific patterns. Designing and implementing a good pidgin is hard. Doing it effectively means getting down and dirty with your host language and its runtime structures. And that’s not the sort of thing you can cover effectively in a language agnostic book.

Martin, if you’re reading this, good luck. I think you’re going to need it. I look forward to being proved wrong.

Getting to grips with Javascript

pdcawley@bofh.org.uk (Piers Cawley) — Fri, 18 Jan 2008 02:30:00 -0600

I’ve been busily adding AJAX features to the work website, and I got bored of writing Form handlers. I got especially bored of attaching similar form handlers to lots of different forms on a page, so I came up with something I could attach to document.body and then plug in handlers for different form types as I wrote them.

So, I wrote FormSender and set up my event handler like so:

FormSender.onSubmit = function (e) {
  if (canDispatch(e)) {
    YAHOO.util.Event.stopEvent(e);
    YAHOO.util.Connect.setForm(e.target);
    YAHOO.util.initHeader('Accept', 'application/javascript, application/xml');
    YAHOO.util.Connect(e.target.method.toUpperCase(), e.target.action,
    callbackFor(e));
  }
};

jQuery(document.body).each(function () {
  YAHOO.util.Event.addListener(this,  "submit", FormSender.onSubmit);
});

I decided to mark any Ajax dispatchable forms using a class of ‘ajax’, the idea being that it would be a simple thing to check jQuery(e.target).hasClass('ajax'), but there was a snag. We had two sorts of forms on our pages, forms built using form_for(..., :class => 'ajax') and forms built using button_to(..., :class => 'ajax'), and they attached their classes in different places. In the form_for case, the class was on the form tag, but in the button_to case, it was on the generated form’s submit field. One option would be to monkey patch button_to, or roll my own ajax_button_to, but I ended up writing canDispatch like so:

function canDispatch(e) {
  jQuery(e.target).find(':submit').andSelf().hasClass('ajax');
}

This uses jQuery to build a list of the form, and its submit button, and then checks to see if any member of that list has the class ‘ajax’.

So, we can now tell if the source of a submit event is a form we should be doing AJAX dispatch with. The next trick is to work out what needs to be done with the results of sending the form. One option is the Prototype trick of simply evaluating the returned javascript, but it often makes sense to keep the behaviour clientside and just have the server return a datastructure. I decided that the way to do this would be by adding a second class to a form which used a none default handler, and then keep a hash of callback constructors keyed by class. This made callbackFor look like:

function callbackFor(e) {
  var candidates = candidateClasses(e.target);
  for (var i = 0; i < candidates.length; i++) {
    if (FormSender.callbacks[candidates[i]]) {
      return new FormSender.callbacks[candidates[i]](e);
    }
  }
  return new FormSender.callbacks.ajax(e);
}

candidateClasses is, again, a little more complex than I’d like, by virtue of the differences between button_to and form_for differences, but still reasonably straightforward, thanks to jQuery:

function candidateClasses(element) {
  return
  jQuery(element).find(':submit').andSelf() 
    .filter('.ajax').attr('className')
      .replace(/ajax/, '').trim().split(/ +/);
}

JQuery gets the form and its submit button, then selects the tag that has the ‘ajax’ class and pulls out the full className string. The replace gets rid of ‘ajax’, trim chops any useless whitespace off either end, and split(/ +/) turns it into an array of classnames. The replace -> trim -> split pipeline has the feel of something that must already exist in some DOM interface somewhere, but I’m not sure where, so I rolled my own.

Once we have a list of classes it’s easy to just cycle through the candidates until we find one that matches a callback constructor, falling back to the default where nothing matches.

For completeness, I’ll show you my current default handler, which I expect to be extending to deal with a couple more media types and, in the case of the failure handler, more failure statuses.

FormSender.callbacks.ajax = function (e) {
    var form = e.target;
    this.scope = form;
};

FormSender.callbacks.ajax.prototype.success = function (o) {
    switch (o.getResponseHeader['Content-Type'].replace(/;.*/, '')) {
    case 'application/javascript':
    case 'application/x-javascript':
    case 'text/javascript':
        eval(o.responseText);
        break;
    default:
        YAHOO.log("Can't handle AJAX response of type " + o.getResponseHeader['Content-Type']);
    }
};

FormSender.callbacks.ajax.prototype.failure = function (o) {
    switch (o.status) {
    case 401:
        Authenticator.loginThenSubmit(this);
        break;
    default:
        switch (o.getResponseHeader['Content-Type'].replace(/;.*/, '')) {
        case 'application/javascript':
        case 'application/x-javascript':
        case 'text/javascript':
            eval(o.responseText);
            break;
        default:
            YAHOO.log("Can't handle AJAX failure response of type " + o.getResponseHeader['Content-Type']);
        }
    }
};

You’ll notice a reference to Authenticator.loginThenSubmit in the 401 handler, but that’s something I’ll save for another day.

A note on namespacing

Although I’ve been showing the various FormSender helper functions as if they were in the global namespace, in the real code they’re wrapped in a function call:

var FormSender = (function () {
  var candidateClasses = function (element) {...};
  var callbackFor = function (e) {...};
  ...
  var onSubmit = function (e) {...};
  return {onSubmit: onSubmit, callbacks: {}};
})();

I love the (function () {...})() pattern – it’s a great way of keeping your paws out of the global namespace until you really, really need to.

FormSender Benefits

Aside from the obvious benefit of drastically reducing the number of onSubmit event handlers registered with the browser, I found that using FormSender has simplified some of my response handlers. For instance, one form would get a chunk of html back from the server and would use that to replace the div that contained the form. But the new div also contained a form that needed to have Ajax behaviour, so a chunk of the handler code was concerned with reregistering onSubmit handlers for the new form (or forms). No fun. By switching to a single, body level, form handler, that problem simply disappears – so long as the new forms have the right class, they automatically get the appropriate behaviour. Result.

Obviously, FormSender is unobtrusive javascript, which is nice, and its pluggable nature means it’s easy to extend just by writing new response handlers and registering them with the FormSender object.

Future Directions

One obvious extension to FormSender is to pull out the meat of the onSubmit method into the callback object to allow for forms that don’t simply send themselves to the server. Another is to wrap my head around the workings of Javascript’s object model to make it easy to build handlers that don’t duplicate the behaviour of the default handler through the medium of copy and pasting code…

Your comments please?

I’m still very new to Javascript as a programming language and I’m sure I’m doing plenty of boneheaded things here. Please let me know if there’s things I can do to improve this, or point me at any libraries that already cover this ground.

Comprehensible sorting in Ruby

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 16 Dec 2007 03:00:51 -0600

Here’s a problem I first came across when I was about 13 and helping do the stock check at the family firm. The parts department kept all their various spare parts racks of parts bins. Each bin was ‘numbered’ with an alphanumeric id. We had printouts of all the bin numbers along with their expected contents and we’d go along the racks counting the bins’ contents and checking them off against the print out. What confused me at the time was the way the printouts were organized. Instead of the obvious ordering, “A1, A2, A3, ..., A99”, the lists were ordered like “A1, A10, A11, ..., A2, A20, A21, ...”. After a bit of thought I realised that the computer was sorting the numeric bits of the bin numbers as if they were just sequences of strange letters. A bit more thought made me realise why, post computerisation, people were starting to use bin numbers like “A01, A02, ...”. Computers were more important than people so, in order to make sorting things easier, just add spurious leading 0s to make the number field a fixed width and Robert’s your parent’s brother.

27 years later and computers are still crap at sorting things in a sensible fashion. Back before Moore’s Law was really kicking in, I suppose it was excusable, but surely we’ve moved past that now.

Over on the labnotes blog, there’s an example of some ruby code that attempts to do ‘human’ sorting:

module Enumerable
  def sensible_sort
    sort_by { |key| key.split(/(\d+)/).map { |v| v =~ /\d/ ? v.to_i : v } }
  end
end

It’s okay, as far as it goes. It certainly solves the parts bin problem I outlined above, but it’s not ideal. For example, you might expect ['-1', '1', '1.02', '1.1'].sensible_sort to leave the order unchanged, but what you actually get is ‘1, 1.02, 1.1, -1’. Not ideal. Let’s rewrite sensible sort as

module Enumerable
  def sensible_sort
    sort_by {|k| k.split(/([-+]?\d+(?:\.\d+)?(?:[-+]?[eE]\d+)?)/).map {|v| Float(v) rescue v}}
  end
end

That ugly regular expression should match a far wider selection of string representations of numbers. Certainly our ‘bad’ list is now sorted correctly.

But what about “a-1”, “a-2”. Using the implementation above, they’d get sorted as “a-2, a-1”, which can’t be right, can it? Let’s extend it a bit more and make sure we only worry about the ’+’ and ’-’ if they’re at the beginning of a line or preceded by whitespace.

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v}}
  end
end

And that works fine, until you find that “B” sorts before “a”. Let’s catch that as well:

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+)?(?:[eE]\d+)?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

Yay!

Oh, wait a minute, what about version numbers? How should we sort, say “perl 5.8.0” and “perl 5.10.0”? The 5.8.0 form should definitely come first… Hmm…

How about

module Enumerable
  def sensible_sort
   sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?)/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

How far down does this thing go?

I just noticed that ”.1” sorts after “1”. Time for another tweak…

module Enumerable
  def sensible_sort
    sort_by {|k| k.to_s.split(/((?:(?:^|\s)[-+])?(?:\.\d+|\d+(?:\.\d+?(?:[eE]\d+)?(?:$|(?![eE\.])))?))/ms).map {|v| Float(v) rescue v.downcase}}
  end
end

but that doesn’t work with version numbers like ”.8.2”, ”.10.2”...

Time passes… Thorin sits down and sings about gold

I was planning on giving an extension of the regex that caught this issue as well, but I’m afraid I’ve stumped myself – I can’t do it with a single regular expression unless I can use a fixed width lookbehind assertion, but they’re only available in Perl. Of course, it’s still possible to fix it, but doing so will take more thought than I have available to me at this time on a Sunday morning. And all this is before we get onto making sure that “1/2” sorts between “0” and “1”. And phone numbers. After all, “01915551238” is ‘obviously’ the same as “0191 555 1238” and “0191 555-1238”, so they should end up next to each other in the sorted list.

It looks like this is a ‘three pipe problem’ after all. I shall probably return to this…

Can I get a witness?

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 19 Nov 2007 13:17:21 -0600

Worrying about test coverage when you’re doing Test- or Behaviour-driven development is like worrying about the price of fish in Zimbabwe when you’re flying a kite.

Your tests are there to help you discover your interface and to provide you with an ongoing stream of small bugs to fix. If you write them cleanly, and keep them well factored (you are refactoring your tests, aren’t you?) they will help to document your intent too. Ensuring that every code path is exercised might be intellectually satisfying but that satisfaction costs time, and time is money. And that’s before you start worrying about your code’s malleability. Cover the happy path and the edge cases you know how to deal with and move on. If you’ve screwed something up, it will get found during acceptance testing (or out in the wild) and you can write a few more tests to isolate the problem, fix it, and move on.

If you deliberately add a test that passes without requiring you to write another line of code, ask yourself why you bothered with it. The test isn’t isolating a bug or specifying new behaviour. If you’re lucky, it merely confirms something you already know, and if you’re unlucky, you just introduced a bug in your test suite. Better to move on and either pull a new bug off the queue, or turn a feature request into a new bug – “Feature X doesn’t work!” – fix it, refactor and move on to the next. Further down the road, you may discover that the feature you were about to exhaustively test doesn’t even need to be there, or maybe it works differently than you expected. Aren’t you glad you didn’t worry about 100% coverage then? Maybe you will need to revisit the tests and cover more cases in the future. But future you knows more about the problem domain and can do a better job of working out what the behaviour needs to be. And if future you is likely to do a worse job than you right now, you may have bigger problems than the code to worry about.

Today's Noun is: Reticence

pdcawley@bofh.org.uk (Piers Cawley) — Thu, 27 Sep 2007 18:30:29 -0500

What does the OED say reticence is?

Reticence: Maintenance of silence; avoidance of saying too much or of speaking freely; disposition to say little.

Pretty straightforward. When I chose reticence as one of my five nouns for programmers it was another reminder that objects are not the same as datastructures. Well designed objects keep their cards (instance variables) close to their chest. Client code tells objects what to do, it doesn’t ask them to kiss and tell. In Smalltalk Best Practice Patterns, (you don’t have a copy yet? Are you mad?) Kent Beck recommends that you put your accessor methods in the private protocol unless you have a very good reason for indicating that the accessors should be used by clients by putting the accessors in, say, the accessing protocol or some other, more suitably named, protocol. (In Smalltalk, all methods are public, but you can and should organize them into protocols/categories, either by choosing some existing protocol, or coming up with a new protocol name). In less flexible OO languages, you should probably at least mark your accessors as protected unless you have the aforementioned very good reason.

What about parameter objects then?

One obvious ‘exception’ to the rule of reticence is the parameter object. Say you have an unwieldy method that takes a bunch of arguments and fills the screen with its implementation. Being a conscientious programmer, you want to apply the ‘composed method’ pattern to the method so you’ll still end up with a screen’s worth of code, possibly more, but the individual methods will be much more focussed in what they achieve. What stops you is that bunch of parameters which are used through out the method. So, you introduce a parameter object. Bundle up the parameters into a single object, replace each parameter variable in the body of the code with an accessor call on the parameter object, and have at it. You can extract methods easily and you’ll only ever have to pass a single object.

It’s tempting not to bother creating a ‘real’ object, it’s very easy to just use an options hash, a la Ruby on Rails. And, because the options hash is a common pattern through your code, you can start adding helper methods like with_options, and you’ll get an awful lot of leverage out of it.

However, what’s often missed in discussions of the parameter object pattern is that it’s a waystation, not a destination. It may be handy, but if you persist in treating it as a datastructure, you’re missing out on the good stuff. Now you have a parameter object, you can start to move behaviour onto the parameter object and before you know it you’ll end up with a real object doing real, testable (mockable?), work.

Homework

Find a method where you’re using an options hash. Try using that hash to build a parameter object and then apply the principle of reticence. What happens to it? Where does the behaviour go? If, like me, you’ve come to OO from procedural coding styles, making your objects reticent is not a natural thing to do, it can feel extremely odd. But the more you practice it, the better you’ll get at thinking in a truly object oriented fashion. Sometimes an options hash really is the way to go, but not as often as you’d think.

Try it, maybe you’ll be converted.

My head hurts

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 23 Sep 2007 14:09:24 -0500

During DHH’s keynote at RailsConf Europe it was apparent that there’s a great deal to like in edge rails, so I thought I’d have a crack at getting Typo up on it.

Ow.

I’d expected the pain points to be related to routing, but it seems that the rails routing system is approaching the level of the Excel calculation engine – nobody dares touch it for fear of breaking things, so typo’s custom routes seemed to work quite happily. There were a few things that have been deprecated, pluginized or moved out of the set of modules that’s automatically included when you do a rake rails:freeze:edge, but they were pretty easy to sort – the deprecation messages are a good deal more informative now than they were last time went deprecation squashing. There’s a surprising amount of stuff that’s been removed without any deprecation warnings though, which isn’t very sporting. DHH said there would likely be a 1.2.4 release (possibly a day before 2.0) with a bunch more deprecation warnings covering everything that’s actually going away, so if you’re thinking of moving a maturish app to Rails 2.0 it might make sense to wait for 1.2.4, install that, squash warnings, and move on up to 2.0.

The real pain comes from themes. Typo’s themes rely on Rails internals working in a particular way, but they don’t work like that any more. In theory, the internals appear to be more theme friendly, related to allowing plugins to include views. The problem is, that it’s possible to change Typo’s theme without restarting the server, and the new themish internals don’t expect anything to change until the server’s restarted.

So, I’ve been playing with plugins. The most promising approach appears to be that of the themer plugin, which gets pretty close to doing what we need, and does it in a way that seems like it should work with both 1.2.3 and Edge Rails. It does appear to be making some radically different assumptions about the structure of the themes directory, but the basic framework is good and I should be able to make things work by making our current them object conform to Themer::Base’s interface and duck type my way to the sunny uplands of Edge Rails compatibility.

Which will be nice.

I like the themer approach a lot. Instead of monkeying about in the guts of rails, it monkeys about in front of Rails. It overrides render so that you can pass it a theme/lookup object. If it sees a lookup object, it uses that to rewrite the rest of the render arguments into a form that will render the right thing using the standard implementation of render. In a work project I’ve taken a similar approach to handling polymorphic routes for things like:

map.resources :pictures do |pics|
  pics.resources.comments
end

map.resources :users do |users|
  users.resources.comments
end

I ended up with a to_params method defined on my Comment model, and stuck an extended url_for in front of the default Rails version, which looks something like:

def url_for_with_to_params(*arguments)
  if arguments[0].respond_to?(:to_params)
    with_options(arguments.shift.to_params) do |mapper|
      mapper.url_for_without_to_params(*arguments)
    end
  else
    url_for_without_to_params(*arguments)
  end
end
alias_method_chain :url_for, :to_params

Which is so much neater than the last time I attacked this particular problem (see the acts_as_resource plugin).

One of the nice things about Rails is that, although it’s opinionated and somewhat liberal with the syntactic vinegar for things the core team don’t think is the Right Way, they’re pretty good at leaving the door open for people like me who have other opinions. Both the themer plugin and my as yet unpluginized extension of url_for work by using existing capabilities in new ways and, because those capabilities are documented we can expect them to continue to work over multiple versions of Rails. Plugins that achieve similar effects by monkeying with Rails’s internal interfaces are hostages to fortune. Internal interfaces are free to change at any time, even between point releases, so a plugin can be left high and dry with surprising rapidity. Just ask the Rails Engines folk.

A tiny ruby niggle

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 09 Sep 2007 02:51:00 -0500

You know what? I’m starting to miss compulsory semicolons as statement terminators in Ruby.

“What?” I hear you say. “But not needing semicolons is one of Ruby’s cardinal virtues! Are you mad?”

I don’t think so, but maybe you’ll disagree after I explain further.

Here’s a piece of code that I might write if semicolons were the only way of terminating a statement:

Category.should_receive(:find_by_permalink)
  .with('foo')
  .and_return(mock_category);

Or how about a complex find query

def find_tags_for(tag_maker, order = 'count')
  klass = tag_maker.class
  find :all
    , :select => 'tags.*, count(tags.id) count'
    , :group => Tag.sql_grouping
    , :joins => 
        "LEFT JOIN taggings ON "
      + "      tags.id = taggings.tag_id "
      + "LEFT JOIN bookmarks ON "
      + "      bookmarks.id = taggings.taggable_id "
      + "  AND taggings.taggable_type = 'Bookmark' "
      + "LEFT JOIN #{klass.table_name} ON "
      + "      #{klass.table_name}.id = bookmarks.#{klass.to_s.underscore}_id"
    , :conditions => conditions_for(tag_holder)
    , :order => (order == 'count') 
        ? 'count(tags.id) desc, tags.name' 
        : "tags.name"
    , :readonly => true
    ;
end

I first came across the idea of the leading comma in Damian Conway’s excellent Perl Best Practices. The idea is that, by leading with the comma it’s very easy to add a new argument to an argument list or hash specification without having to remember to stick a comma on the end of the preceding line if it was at the end, and also, the leading comma makes it very plain that the line is a continuation of its predecessor in some way.

To make the examples work in Ruby, you have to add a \ to the end of each line that has a continuation, so the first example has to be written:

Category.should_receive(:find_by_permalink) \
  .with('foo')                              \
  .and_return(mock_category);

Lining up the \s helps to stop them disappearing, but it’s an awful faff.

What tends to happen (in the rails source especially) is that ruby programmers simply don’t break their lines up. A quick search of the rails source finds plenty of lines more than 160 characters long.

Of course, some will argue that it doesn’t matter, that the old 80 column limit is a silly hangover from the days of steam when the only way to interact with your code was through an 80 column, green phosphor terminal. They have a point. An arbitrary line limit is silly, and we should get over it, especially in source code. However, unless you’re going to go around with every window open to its maximum width, lines will wrap, and they won’t do it nicely, or respect the indentation conventions of your language. Long lines are murder in diffs too, finding the point of difference is so much easier when your eye doesn’t have to scan an epic line.

It’s a shame there’s no way of forcing ruby’s parser to require semicolons as statement terminators for those programmers like me who think that the restriction that a statement must end with a semicolon is worth the freedom to break lines where we like without needing to escape every line break. It’s a shame too that popular tools like Textmate are so clumsy when it comes to dealing with line breaks. I would attempt to hold Emacs up as a paragon in this respect, but its Ruby mode tends to get a wee bit bemused once you start breaking lines, so that’s no good.

Domino theory

It’s amazing how far reaching seemingly simple language design decisions can be isn’t it? Just getting rid of the need to terminate statements with a semicolon has an enormous effect on they way code in ruby looks. I’m just not sure that they look better.

Maybe Smalltalk got it really right – they chose to use the most valuable syntactic character of all, the space, to denote sending a message. That freed up the . for use as a statement (sentence?) terminator. Then that freed up ; for use in one of Smalltalk’s most distinctive patterns – the cascade. Where a Rails programmer might write:

form_for(@comment) do |f|
  f.input(:author)
  f.input(:title)
  f.input(:body)
end

A Smalltalk programmer might eliminate the need for a temporary variable by doing:

Comment>>printOn: html

  (html formFor: self)
    input: #author;
    input: #title;
    input: #body.

All those input: ... messages get sent to the result of html formFor: self. Once you get the hang of it, it’s a really sweet bit of syntax.

Incidentally, there’s been some discussion on the squeak mailing lists of a companion to the cascade, which would use a ;; as a sort of ‘pipe’. The idea is to be able to replace code like:

((self collect: [:each | each wordCount) 
    inject: 0 into: [:total :each| total + each]) 
        printOn: aStream.

with

self collect: [:each | wordCount]
    ;; inject: 0 into: [:total :each | total + each]
        ;; printOn: aStream.

(NB: Please ignore what those code snippets do, because that’s gruesome. Concentrate on how they do it).

Nobody’s quite proposed going as far as Haskell does with its Monads, which can be thought of as a magical land where the meaning of the semicolon changes according to what sort of Monad you’re in. (In an IO monad for instance, the semicolon imposes an evaluation order. In some other monad, the semicolon could just as easily denote a backtracking point). Then again, there’s nothing to stop the dedicated Smalltalker implementing something Monadish – every Smalltalk class can specify how its methods should be compiled after all…

In conclusion…

I’m not sure I’ve got a real conclusion for all this. I’m mostly musing. However, I do think it’s useful to think carefully about restrictions and what they free us to do as programmers. Lispers will wax lyrical about the way that their language’s pared down syntax lets them do amazing things with macros. Smalltalkers will defend to the death the idea that the only way to do anything is to send messages to objects. Pythonistas love their syntactic whitespace. Haskellers love their static typing (admittedly, they have an incredibly flexible notation for expressing type that leaves most other programming languages standing).

And any English speaker with ears will know that a poem like Dylan Thomas’s Do Not Go Gentle Into That Good Night gains much of its power from it’s form, the villanelle, one of the most restricted forms of poetry there is. Two lines repeating through the poem and a staggering number of rhymes to find:

Do not go gentle into that good night,
Old age should burn and rave at close of day;
Rage, rage against the dying of the light.
Though wise men at their end know dark is right,
Because their words had forked no lightning they
Do not go gentle into that good night.

Good men, the last wave by, crying how bright
Their frail deeds might have danced in a green bay,
Rage, rage against the dying of the light.

Wild men who caught and sang the sun in flight,
And learn, too late, they grieved it on its way,
Do not go gentle into that good night.

Grave men, near death, who see with blinding sight
Blind eyes could blaze like meteors and be gay,
Rage, rage against the dying of the light.

And you, my father, there on the sad height,
Curse, bless me now with your fierce tears, I pray.
Do not go gentle into that good night.
Rage, rage against the dying of the light.

If that’s not making a virtue of a restriction, I don’t know what is.

Today's Noun Is: Notation

pdcawley@bofh.org.uk (Piers Cawley) — Tue, 28 Aug 2007 16:20:00 -0500

Remember back when I wrote about metaprogramming and programming being the same thing?

Well, Libraries, Domain Specific Languages, Domain Agnostic Languages, Pidgins, YAML, .INI files and the like are all the same thing. They’re all notation.

That’s the thing to remember. Good notation makes life easy. Bad (or inappropriate) notation makes it hard (or, in some cases, fun). There’s no point striving to write a pidgin or DSL if your problem can be solved easily enough with a well factored set of loosely coupled classes. After all, even the neatest and cleanest of domain specific languages has to be learned. Meanwhile a class library that’s respects the language of your problem domain can be easier to learn because the notation with which the library is implemented is plain old YFL and your team already knows that (and if it doesn’t, you have bigger problems). Learning the class library is like learning about the problem domain and vice versa. Learning one supports learning the other.

I’m of the opinion that DSLs and pidgins are at their most when their domains are generic. So ActiveRecord’s most useful language like interfaces are related to general problems like expressing the relationships between objects or describing how to validate an object. Backus-Naur Form, Perl compatible regular expressions, parser combinators are all useful for solving the generic parsing and lexing problems that crop up everywhere. Data serialization notations like YAML, FASTA, XML, CSV, JSON and all the others are useful when you want a generic way to pass data between systems. All of these notations are narrow in their focus, but broad in the sense that the things that they focus on are ubiquitous.

The most useful notation for your specific task is almost always a well factored set of classes or functions (depends on your language of choice) which reflects the language of your problem domain. Different parts of your program might well make use of a wide range of ‘donor’ notations, but it’s a rare problem that’s going to require to you implement a full blown language in order to solve it.

Don’t worry about whether something’s a DSL, pidgin or a library. What it is is notation. A better question is is “does this notation help me solve my problem?”. If the answer is no, find better notation. Or keep plugging on with what you’ve got, but don’t fool yourself about your reasons for using the notation – not everyone’s here for the hunting after all.

Today's noun is: Reification

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 22 Aug 2007 18:22:00 -0500

Reification: The mental conversion of a person or abstract concept into a thing. Also, depersonalization, esp. such as Marx thought was due to capitalist industrialization in which the worker is considered as the quantifiable labour factor in production or as a commodity. – OED

In the sense that the OED has it, I’m not what you could call a fan of reification. At work, we have a rule that anybody who starts talking about ‘resources’ when they mean ‘people’ gets a (verbal) slap.

However, in OO circles (or maybe just in my head), reification is a good thing. It’s the process of taking something abstract and turning it into a ‘real’ object. Usually, the word gets used for big things like turning an intractable method into an object as a step on the way to refactoring that method. I tend to use it in a slightly broader sense. For me, reification is the process of turning something (a method or a data structure usually) into a full blown object with its own behaviour.

Back when I was working on Pixie (a cunning, but weird, object persistence tool written in Perl) we had a data structure which was used for keeping track of managed objects. It started life as a hash. Everything was fine at first, but over time we ended up with more and more code being repeated across the codebase that was concerned with manipulating the cache hash. So, we replaced the hash with a new object and pulled all the repeated code into methods on that object, which gave us cleaner code to extend, and a strong feeling that we should have turned the cache into an object much earlier in the game. (By leaving it so long, we had a lot more code to move about, some of it in fairly obscure places; tracking down the last bit took a while.)

Data structures like hashes and arrays are really useful in languages that have them. The catch is, they have this habit of acquiring code. When this starts to happen, it’s time to reify – to replace the hash with a task specific object. In Ruby, it’s easy enough to inherit from Hash, but Hash comes with a pile of methods that probably aren’t relevant to your particular need. Generally it’s better to delegate. The first cut doesn’t have to be that complicated, just decorate the hash with a new class and initialize an instance of the class at the point where you had just made the hash.

Once that’s done, you can go through your code and move the bits that treat the hash as a data structure onto your new class. As you gather all the common behaviour to the new class, you’ll start to see places where you can improve code quality by merging common behaviours, replacing complex conditionals with polymorphism (you’ll probably have to introduce a factory method if you do that) and pulling hash keys out into instance variables.

Stalled reification

Reifying your data structure isn’t an end in itself, it’s a step along the way as you refactor your code.

There’s an example of a stalled reification to be found in ActionController::Routing::Resources. Consider the implementations of map_resource and map_singleton_resource, which are the worker methods used whenever you do a map.resource or map.resources in your routes.rb.

def map_resource(entities, options = {}, &block)
  resource = Resource.new(entities, options)

  with_options :controller => resource.controller do |map|
    map_collection_actions(map, resource)
    map_default_collection_actions(map, resource)
    map_new_actions(map, resource)
    map_member_actions(map, resource)

    if block_given?
      with_options(:path_prefix => resource.nesting_path_prefix, &block)
    end
  end
end

def map_singleton_resource(entities, options = {}, &block)
  resource = SingletonResource.new(entities, options)

  with_options :controller => resource.controller do |map|
    map_collection_actions(map, resource)
    map_default_singleton_actions(map, resource)
    map_new_actions(map, resource)
    map_member_actions(map, resource)

    if block_given?
      with_options(:path_prefix => resource.nesting_path_prefix, &block)
    end
  end
end

There’s a lot of repetition there. The only differences are the classes of the resource object, name of the second function called in the with_options block. If we take a look at, map_collection_actions things start to look even fishier. Here’s map_collection_actions, for example:

def map_collection_actions(map, resource)
  resource.collection_methods.each do |method, actions|
    actions.each do |action|
      action_options = action_options_for(action, resource, method)
      map.named_route("#{resource.name_prefix}#{action}_#{resource.plural}", "#{resource.path};#{action}", action_options)
      map.named_route("formatted_#{resource.name_prefix}#{action}_#{resource.plural}", "#{resource.path}.:format;#{action}", action_options)
    end
  end
end

resource.collection_methods.each? Let’s see what happens if we the various map_foo_actions methods into methods on ActionController::Resources::Resource. While we’re about it, we can rename map_default_collection_actions to map_default_actions on Resource, and map_default_singleton_actions to map_default_actions on SingletonResource, which inherits from Resource. map_collection_actions becomes:

def map_collection_actions(map)
  collection_methods.each do |method, actions|
    actions.each do |action|
      map.with_options(action_options_for(action, method)) do |m|
        m.named_route("#{name_prefix}#{action}_#{plural}",
                      "#{path};#{action}")
        m.named_route("formatted_#{name_prefix}#{action}_#{plural},
                      "#{path}.:format;#{action}")
      end
    end
  end
end

(we move action_options_for onto resource as well, of course).

Once we’ve moved the various mapping helpers onto the resource classes, we can revisit map_resource and map_singleton_resource


def map_resource(entities, options={}, &block)
  resource = Resource.new(entities, options)

  with_options(:controller => resource_controller) do |map|
    resource.map_collection_actions(map)
    resource.map_default_actions(map)
    resource.map_new_actions(map)
    resource.map_member_actions(map)
  end

  if block_given?
    with_options(:path_prefix => resource.nesting_path_prefix, &block)
  end
end

def map_singleton_resource(entities, options={}, &block)
  resource = SingletonResource.new(entities, options)

  with_options(:controller => resource.controller) do |map|
    resource.map_collection_actions(map)
    resource.map_default_actions(map)
    resource.map_new_actions(map)
    resource.map_member_actions(map)
  end

  if block_given?
    with_options(:path_prefix => resource.nesting_path_prefix, &block)
  end
end

And now, we no longer have two method bodies that look very similar, apart from the resource class, we have to methods that look identical apart from the resource class. So, if we pull out the common bits and put them onto Resource, like so:


class ActionController::Resources::Resource
  def install_routes_in(map, &block)
    map.with_options(:controller => controller) do |m|
      map_collection_actions(m)
      map_default_actions(m)
      map_new_actions(m)
      map_member_actions(m)
    end

    if block_given?
      map.with_options(:path_prefix => nesting_path_prefix, &block)
    end
  end
end

Then map_resource and map_singleton_resource become


def map_resource(entities, options = {}, &block)
  Resource.new(entities, options).install_routes_in(self)
end

def map_singleton_resource(entities, options = {}, &block)
  SingletonResource.new(entities, options).install_routes_in(self)
end

Where’s the benefit?

Apart from making the active_record/lib/resources.rb a bit shorter (a laudable result in itself), where’s the benefit here?

From my own experience of implementing datestamped_resource, a routing plugin that we use in Typo, it makes the life of anyone writing a resource like routing helper for Rails a great deal easier. With datestamped_resource I ended up subclassing ActionController::Resources::Resource, doing the refactoring I’ve outlined here, but leaving the original Rails methods where they were and just implementing the ‘moved’ methods on DatestampedResource (well, not quite, map_collection_actions is pretty different from the default Resource implementation, but the other actions are pretty much the same.

In another project I’m working on, I’m trying to retain meaningful urls with (potentially) deep resource nesting, and it’d be really handy to have an inflected_resource route helper. The problem with using a meaningful to_param on your models is, avoiding permalinks that share a name with your actions. You could set up validations so that, say, ‘new’ is an illegal permalinks, but it’s clumsy.

However, if you arrange things so that your URLs are inflected, you can always tell that a URL that begins /resource/new will be a particular resource, with the permalink ‘new’, and /resources/new will be the virtual new resource.

If the resource system is factored as I outlined, this is almost trivial, you can introduce a InflectedResource subclass of Resource


class InflectedResource < Resource
  def member_path
    @new_path ||= #{path_prefix}/#{singular}/:id
  end
end

and you’re pretty much done. Admittedly, something like that (plus a small amount of copy and paste) would work with the current system, but then we’re looking at 3 substantially identical methods in ActionController::Resources and if it wasn’t time to refactor before, it’d definitely be time to refactor then.

Conclusions

Reification shouldn’t be something you do every day, but nor should it be something you do once a flood. Take a look at some of your projects and some of the places where you’re using hashes. Are those really hashes, or would they benefit from having some behaviour of their own? You can track down stalled reification by looking for anaemic classes; classes which have a lot of accessors but very little behaviour. Once you’ve found an anaemic class, look for all the places that instances of it get used. Try moving some of the client code into methods on your anaemic class. Do that a few times and you’ll end up with a real object.

If you’re fussy about never putting HTML in your models, you could end up with a mediating builder/presenter object as well, but until you start wanting to render the same structured info in different formats, I’d suggest biting the bullet and living with HTML in the model as a lesser evil than structural code. Your mileage may vary.

Reading Beautiful Code

pdcawley@bofh.org.uk (Piers Cawley) — Tue, 21 Aug 2007 18:49:57 -0500

I’m reading Beautiful Code and it’s very good indeed. However, you have to feel sorry for Tim Bray – his chapter, “Finding Things” is excellent, as you’d expect. The only problem is, he’s following Jon Bentley, author of that perennial classic, Programming Pearls. It’s been a while since I (re)read anything by Bentley and I’d forgotten how good his prose was.

Which isn’t to say that Tim Bray is a bad writer. Far from it. It’s just that Jon Bentley is a very good one indeed.

Apart from that, Beautiful Code is a cracking book so far, and the royalties go to Amnesty International. You should buy a copy.

Five nouns for programmers

pdcawley@bofh.org.uk (Piers Cawley) — Mon, 20 Aug 2007 18:45:00 -0500

Reification
Notation
Reticence
Lucidity
Intent

Learn ‘em. Live by them.

What did I miss? Which ones don’t belong on the list. What am I talking about?

Domain Specific Pidgin

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 08 Aug 2007 02:43:00 -0500

So, I’m busily writing an article about implementing an embedded little language in Ruby. It’s not something that’s going to need an entirely new parser, it borrows Ruby’s grammar/syntax but does some pretty language like things to the semantics and ends up feeling far more like at declarative language than the usual Ruby imperative OO style.

Because I tend to chromatic’s view of many ruby programmers’ ability to cry ~~“Wolf!”~~ “DSL!”, I don’t want to claim that it’s a full blown Domain Specific Language, but it’s sufficiently language like that ‘API’ doesn’t seem to fit as a description either.

Then it hit me… it’s a pidgin.

A pidgin can be thought of as a mashup of two languages, taking vocabulary from both its parents and its grammar (usually simplified) from one parent. Historically, pidgins have arisen to help with trade and colonization; grammars have tended to be lifted and simplified from the ‘native’ language and then spiked with words from the colonizing language with a leavening of native words where they make sense. All quite politically incorrect nowadays, but they served their purpose. Pidgins are, by their nature, domain specific; fine if you wanted to talk trade or order your coolies about, but not what you’d write poetry in. Poetry tended to get written in the creoles that evolved from some pidgins. A creole is a general purpose language, with a grammar of its own; they seem to evolve from pidgin, getting invented by the kids of parents who speak that pidgin.

In my little language, much of the vocabulary pinched from my problem domain’s language and the grammar and some terminology is lifted from ruby. Casting the problem domain as the colonial power and ruby as the native language, it’s obvious that I’ve invented a pidgin language.

Let’s embrace that term. We’re not writing domain specific languages, we’re writing pidgins. ActiveRecord’s family of class methods isn’t a DSL, it’s database pidgin. RSpec is a testing pidgin, Parsec is a parsing pidgin, so are any number of APIs that make their host language feel like a new one.

Updates

In the comments, Aristotle Pagaltzis points out that Parse::RecDescent isn’t a parsing pidgin because it uses a full blown parser (itself, obviously) to parse any grammar declarations.

Doing the fixture thing

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 05 Aug 2007 02:39:00 -0500

Fixtures suck! Mocks rock! Don’t you dare let your tests touch the database!

Well… yes… I suppose. Except, mocking can be a complete pain in the arse too (made slightly less of a pain in the arse if you use the null object options) – it’s awfully easy to end up with huge long setup methods that spend all their time faking out a mock object and about two lines testing what you need.

I’m sure someone’ll be along to argue that this is evidence of lazy design on my part. Well, “Yah boo sucks!” to the lot of ‘em. Fixtures can be exceedingly useful.

Spooky action at a distance

The biggest problem I have with Rails’ current implementation of fixtures is… oh, where to start… it’s probably the action at a distance aspect of them. Your test is here, but your fixture is defined some way over there, in a random collection of yaml files.

Then there’s the problem of remembering which fixtures you need to have loaded for a particular test, and it all starts getting horrible very quickly.

One way of addressing this is to use fixture scenarios and fixture scenario builder, which works around the problem of remembering what to load and means you don’t have to write your fixtures in YAML. What it doesn’t work around is the action at a distance issue (but I’m betting it wouldn’t be too hard to repurpose the fixture scenario builder to let you declare the fixtures you need for a particular test/spec in the same place as you do the testing).

At work though, we came up with another way of making it easy to build your fixtures right in the test file. Our current approach is still a little bit clunky, and it’s nowhere near the point where I can turn it into a library, but I think it’s worth discussing anyway.

Exemplars

The question to ask is, what do you use a fixture for? Most of the time, what a test needs is a mostly generic instance of your class which will pass the validations, and which has maybe a couple of attributes set to particular values.

Let’s say you have a user class. As is common with such things, your user has a username, an email address and a password. As is so often the case, the usernames and email addresses must be unique, and the password must not be blank. Let’s say you’re working on a tagging system (isn’t everyone). Here’s the sort of specification you might write:

context "A taggable object" do
  setup do
    # makes a taggable object and a couple of users
  end

  it "Should aggregate taggings" do
    @first_user.tag(@taggable, "tag")
    @second_user.tag(@taggable, "tag")

    @taggable.save!; @taggable.reload

    @taggable.should have(1).tags
  end
end

In this context, the only thing you need from those user objects is that they behave like @users, are distinct and valid. You could simply set up a couple of users your users.yml fixture, but the approach we took at work went something like this:

class User
  class << self
    @@exemplar_count = 0
    def exemplar(overrides = {})
      @@exemplar_count += 1
      with_options(:username => "user#{@@exemplar_count}", 
                    :email => "user#{@@exemplar_count}", 
                    :password => "fredisabadpassword") do |maker|
        maker.new(overrides)
      end
    end

    def create_exemplar(overrides = {})
      returning(exemplar(overrides)) {|user| user.save}
    end

    def create_exemplar!(overrides = {})
      returning(exemplar(overrides)) {|user| user.save!}
    end
  end
end

This lets us write setup code like:

setup do
  @first_user = User.create_exemplar!
  @second_user = User.create_exemplar!
  @taggable = ...
end

In tests where you need an exemplar with a specific property, you can write Model.exemplar(:tested_attribute => specific_value) – the way the overrides work means you only have to describe the ‘interesting’ bits and the obscuring dust involved in simply building a valid object is swept under the carpet.

Homework

If you’re familiar with the Object Mother pattern, this might seem a little familiar, with the wrinkle that, instead of having a factory class, we just push the exemplar builder directly onto the model class.

If you start implementing exemplars yourself, you’ll probably spot a good deal of repetitive coding. I’ve not extracted a library yet because I’ve not quite come up with an interface that I like. Can you come up with a good way of doing it? Can you implement it?

What did I miss?

Cheat all you want, but don't get caught

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 20 Jun 2007 17:16:00 -0500

As far as I can tell, one of the Smalltalk optimizers’ mottoes is “Cheat all you want, but don’t get caught”.

Well, this morning, I caught Squeak with its hand in the till.

One way I attempt to bootstrap myself towards understanding of code is to try and make it better, if that makes sense. So, I’d run SLint over the OmniBrowser package and was trying to shorten a method. One thing that struck me as rather ugly was a piece of code that ran like this:

|selection|
...
selection := OBChoiceRequest prompt: nil labels: usersNames values: users.
selection ifNotNil: [selection browse].

So I thought I’d tweak ifNotNil: so that, if its receiver isn’t nil, it will pass itself as an argument into the block, which will let me rewrite that ugly code with:

(OBChoiceRequest 
    prompt: nil
    labels: usersNames
    values: users) ifNotNil: [:selection| selection browse].

So, I went to have a look at the implementation of ifNotNil: and found that it was already doing exactly what I was after.

At this point, I had a slight premonition of danger, so I brought up a workspace and tried to print the result of running 10 ifNotNil: [:i| i + 10] and got a compiler error, complaining that ifNotNil: takes a 0 argument block. Which isn’t what the implementation of ifNotNil: thinks.

I’d caught Squeak’s optimizer cheating.

What appears to happen is that Squeak catches conditional code and rewrites it before passing it to the compiler. The rewritten code uses VM level primitives where possible. I needed to fix it so that it would only rewrite any calls to ifNotNil: with a zero argument block.

It took a while, but my local image now optimizes ifNotNil: correctly (the ifNotNil:ifNil: and ifNil:ifNotNil: forms are another matter though, but I shall live. Now, if I can just work out where to submit the changeset to…

I have mixed feelings about this. On the one hand, I’ve just changed something in the workings of Squeak, on the other, it’s not been quite as easy as I’d expected it to be. It seems that, if you go poking around in methods that are defined on ProtoObject, don’t be surprised if changing things doesn’t quite do what you expect.

Maybe I should have just written a ifNotNilDo: taking a single argument block, but that just felt ugly… Ho hum.

How I learned to stop worrying and love aggressive mockery

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 06 Jun 2007 05:21:00 -0500

There’s something enormously liberating about writing an RSpec description that starts like:

describe ArticlesController, "feeds:" do
  before do
    @the_mock = mock('everything', :null_object => true)
    ActiveRecord::Base.stub!(:find).and_return(@the_mock)
  end

  it "/articles.atom -> atom feed" do
    get :index, :format => 'atom'
    response.should render_template('_atom_feed')
  end

  ...
end

The :null_object flag to rspec’s mock function is remarkably potent.The resulting mock will return itself from any method call that hasn’t got some other expectation set up. When I’m testing that my index methods render the appropriate views for the format, I don’t care that all the various variables have been set up correctly – I’ve already tested that in another description – I just want to get to the point where I’m about to render a template.

What this does is decouple my tests. I can change the way that the index method fetches its stuff from the database and I’m only going to have to change the innards of the specs that test that.

I tend to think of this as being analogous to the object oriented pattern of trying to write your methods at a single level of abstraction. Within a given spec, I should only be setting up expectations that are directly related to what I’m testing.

Metaprogramming == Programming

pdcawley@bofh.org.uk (Piers Cawley) — Sun, 20 May 2007 11:21:00 -0500

While I’m tilting a windmills, I should just like to tell all those people who bang on about ‘metaprogramming’.

It’s all just programming.

That is all.

Domain Agnostic Languages

pdcawley@bofh.org.uk (Piers Cawley) — Sat, 19 May 2007 18:52:00 -0500

Windmill tilting time again I’m afraid. Blame chromatic and David A. Black.

What is it that characterizes domain specific languages? Before you trot out something like “Programs written in them read like domain experts talk”, take a look at some examples of code written in domain specific languages:

/(?ms)"((?>[^\\"]+)?(?>\\.[^\\"]*)*)/

S3
R$*                     $: < $1 >
R$+ < $* >                 < $2 >
R< $* > $+                 < $1 >
R<>                     $@ < @ >
R< $+ >                 $: $1
R@ $+ , $+        @ $1 : $2
R@ $+ : $+         $@ <@ $1> : $2
R$+ @ $+                $: $1 <@ $2>
R$+ < $+ @ $+ >            $1 $2 <@ $3>
R$+ <@ $+ >             $@ $1 <@ $2>

>gi|2501594|sp|Q57997|Y577_METJA PROTEIN MJ0577
MSVMYKKILYPTDFSETAEIALKHVKAFKTLKAEEVILLHVIDEREIKKRDIFSLLLGVAGLNKSVEEFE
NELKNKLTEEAKNKMENIKKELEDVGFKVKDIIVVGIPHEEIVKIAEDEGVDIIIMGSHGKTNLKEILLG
SVTENVIKKSNKPVLVVKRKNS

If you’re reading this on the front page, try and work out what the ones you recognise do before you dip below the fold…

... Right, that’s scared the dilettantes off.

What do they do?

The first snippet is a regular expression. It should be an efficient matcher for double quoted strings which can span multiple lines and have backslash escaped internal quote marks. It’s not exactly easy to read because it’s optimized to reduce backtracking in failing cases. A simpler, but slower form would look like /(?ms)"((?:\\.|[^\\"])*)"/. I say ‘should’ because I wrote it from first principles without recourse to the manual except to check on the syntax of (?>...) – I’ve been using regular expressions for so long now, they’re getting pretty instinctive¹. Every regular expression is a program written in a language whose domain is concisely expressing the rules to match strings. So long as what you’re matching doesn’t require you to worry about balancing bracket like things, a regex should suit you fine.

The second snippet is a sample of sendmail.cf ruleset, this is the one that canonicalizes email addresses into a form that the rest of the rulesets can work on. Once upon a time I could have taken you through it line by line and explain what it did, but the world has changed since then. Sendmail’s domain was routing email back in the days when there were a mllion and one different email address formats and networks. Who uses addresses like kremvax!ivan@ucbvax.UUCP, pdcawley@uk.ac.nott.cs or (if you’re feeling sufficiently evil) pdcawley@uk.org.bofh.org.uk these days? Nowadays all email addresses look pretty much the same and routing is substantially less complicated, so tools like Postfix, Exim and qmail are far, far easier to configure because the domain has shrunk to the point where you can get away with a mere configuration file.

The last one’s a BLAST query² which can be run against the genome of your choice. Again, if you’re a geneticist (or, more likely a geneticist’s computer), this makes perfect sense. If you’re not a geneticist however, it’s the next best thing to gobbledegook.

Searching for the odd one out…

What characterises Domain Specific Languages isn’t their readability, it’s their narrow focus on their domain. The sendmail config file’s the odd one out here because it turns out that you can implement a Universal Turing Machine in a sendmail.cf file file. While Turing completeness might be fun to prove, it can be problematic in a Domain Specific Language. Ask James Duncan Davidson about accidentally making Ant Turing complete some time – there’s a reason he’s run away to be a photographer you know.

Domain Agnostic Languages

General purpose languages like Smalltalk, Lisp, Perl, Ruby, Python or *spit* Java aren’t domain specific languages in the same sense (unless you reckon that ‘writing general applications’ is narrow enough to be their domain). Instead they’re domain agnostic. General purpose languages support programming ‘in the language of the domain’. So ActiveRecord’s combination of well designed class methods let you describe the relationships between database tables by describing the relationships between objects. Meanwhile in Perl, Jifty has similar capabilities and Seaside meanwhile morphs programming for the web into something that feels remarkably like programming for the desktop. Paul Graham would probably tell you that Arc does the same thing with Lisp (he was also the inspiration for Seaside).

The more malleable of these languages let the cunning programmer embed full blown DSLs, complete with their own syntax and semantics within them. Lisp’s macros are the most obvious example of this, but Smalltalk has some pretty amazing tricks up its sleeve too – check out Scratch or Etoys. As I understand it, you can have a different compiler for every browser window if you want. Compared to these languages, Ruby, Python, Perl et al speak the language of the domain with more or less thick accents.

Do I even have a point?

Well, sort of. It’s easy and tempting to rail against the way everything and its brother is a DSL now, but that ship has probably already sailed. We can hope that as the underlying idea of the DSL – the problem shapes the tool – gains wider currency people will incorporate it in their practice and become better at their craft as a result. We old farts will continue to chunter about the young folks of today not realising what it was like when we had to whittle our own bits and write a DSL to build the parser that would let us build the DSL we really wanted. But maybe it’s good that people get starry eyed. It’s a step down the road; even if they never take the next one, at least they’ve moved in a good direction.

The moment you take a chunk of code and give it a name, you’re in the business of language design. Use your power wisely and the programming zeitgeist will beat a path to your door – especially if you have DHH’s seemingly instinctive marketing nous. Use it less wisely and you’re welcome to your ball of mud.

Updates

With his usual aplomb (and in the very first comment), chromatic pointed out a glaring cock up in the original wording of this post so I’ve tightened it up a bit.

¹ I owe the vast bulk of my understanding of the guts of regex engines to reading the first edition of Mastering Regular Expressions, Jeffrey Friedl’s masterly explanation of the subject. I understand it’s only got better with subsequent editions³.

² Oops, no it’s not. It’s “protein sequence in fasta format in which amino acids are represented by 1 letter codes. You can use it as a BLAST query but the options are endless. It is like a calling a comma separated text file as an Excel input only.” Thanks to Darked for pointing that one out to me in the comments. Remember kids, Google + a merely cursory knowledge of a domain will only get you so far. I’d maintain that fasta format can still be thought of as a domain specific language, but it has more in common with XML or HTML than, say SQL.

³ I just noticed the first edition was published in 1997 and I bought it as soon as it came out. I feel old.

Finally learning JavaScript

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 16 May 2007 17:17:00 -0500

For years I’ve managed to dodge learning JavaScript. People have told me it’s a fine language with some dodge implementations. Friends have built an entire business model in the language, and I’ve continued to treat as if it were The Sound of Music (I’m 39 now and I have managed to completely avoid watching that film).

Except, today I finally had a technical need that I couldn’t dodge by writing another .rjs file, so with the help of the Prototype docs, a helpful article or two and healthy dose of cribbing from others, I was off.

What do you know, it really is a fine language. I doubt I’m writing idiomatic code though – I’m leaning heavily on closures and private member functions, and I haven’t written a single for loop yet, but it seems to accommodate my style.

Now, if I can just get my head round the innards of keyboard events I shall be a great deal happier.

Why I love Smalltalk

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 02 May 2007 18:41:00 -0500

From Extreme UI Testing:

Method wrappers. Easily possible in VW and Squeak since you can subclass CompiledMethod and much with the method dictionaries [...] Niall uses a strategy pattern to automatically wrap threads that get forked off by looking up the stack to see if the code is under test (and if it is, make sure to install a wrapper)

So what if the language doesn’t give me the callbacks I need, I’ll just rejig the system classes until it does. Until Avi needed them for Seaside, Smalltalk didn’t have continuations, but it’s malleable enough that he could implement them.

Malleability rocks (even though rocks aren’t exactly malleable.)

That was fun

pdcawley@bofh.org.uk (Piers Cawley) — Wed, 25 Apr 2007 01:57:00 -0500

On Monday I was down in Brighton for a Brighton Coding Dojo where I had a crack at doing Kata four in Smalltalk.

It took a while to find the balance, but once we got going I think it went well.

We stuck in what seemed like the strangest places though. At one point, I had a method that did almost exactly what I wanted for a new method I was writing so I called up the method in the browser, changed the selector and the few bits that needed fixing up and accepted the changes.

Uproar! “Wha? What’ve you done to testGetMnT?”

“It’s still there, look.” I said, pulling testGetMnT up in the browser.

“But…!”

People were impressed by OmniBrowser’s refactoring tools and slightly boggled by the sheer number of instance methods on Object.

Because we dived straight in, people got a wee bit stuck on the syntax as well. Next time I do something like this, I’ll spend more time walking through what’s going on in each line of code, until people get a bit more secure, and I’ll start handing the keyboard off to other pairs way sooner. Once I did that, it became far more apparent which bits were sticking points.

The session certainly confirmed my opinion that you can read all you like about Smalltalk, but you won’t really get it until you see it in motion.

So, once I have some tuits of the appropriate shape, I’m planning on making a longish screencast of me running through Kata four, with commentary. It won’t be an exemplary example of a Smalltalk user getting the very best out of the toolset; I’m very much a beginner myself, but I hope it’ll give you a feel for why you should at least try Smalltalk for yourself. I also hope that any experienced Smalltalkers watching the screencast will be able to give me some tips on better ways of using the tool.