Just A Summary

There’s something satisfying about reaching the point when you can’t decompose an object any further and all your methods are tiny and do one thing – it’s especially gratifying when you learn something new in the process. Sadly, it doesn’t happen as often as I’d like, there’s usually annoying bits and pieces where you have to placate the language in some fashion that breaks the flow of what you’re writing.

As I get a better handle on the way MooseX::Declare has changed Perl, I’m finding I have to do much less in the way of placation.

Here’s an example. For context, I’m writing a traffic shaping tool. The basic client interface needs to look something like:

$policy->current_weight # => a percentage between 0 and 100

Not much of an interface really. The requirements state that we should be able to specify weights with 15 minute granularity for every day of the week. Our problem becomes one of mapping from a time to a number between 0 and 7 (days) * 24 (hours) * 4 (quarters) - 1 and looking up the weight in an array.

Here’s my first cut:

use MooseX::Declare;
class WeightVector {
    use DateTime;
    
    has vector => (
        isa => 'Array[Int]',
        is => 'ro',
        required => 1,
    );
    
    method current_weight {
        my $now = DateTime->now;
        my $offset = ($now->wday_0 * 7 * 24 + $now->hours) * 4 + int($now->minutes / 15);
        return $self->vector->[$offset];
    }
}

Which is, I suppose, perfectly respectable. However, current_weight isn’t filling me with delight. First it finds the current time, then it converts the time into an offset, then it uses the offset to lookup the weight in the vector. Let’s introduce a method to find the weight at a specific time"¹":#decomp-motivation, the relevant code becomes:

method current_weight {
    $self->weight_at(DateTime->now);
}

method weight_at (DateTime $time) {
    my $offset = ($now->wday_0 * 7 * 24 + $now->hours) * 4 + int($now->minutes / 15);
    return $self->vector->[$offset];
}

And again, we could rest here, but again, we’re doing two things. We’re converting from a time to an offset, then we’re looking up the value in the vector. Type conversions tend to happen again and again, so it’s good if we can specify them separately. We could write a time_to_offset helper method, but we’re in Mooseland now; there’s a better way. Let’s introduce a formal Moose type and define a coercion for it. Here’s the type definition stanza of the code. I’ve taken the opportunity to add types which do bounds checking for the vector as well, while I’m about it.²

use MooseX::Declare;
class WeightVector {
    use DateTime;
    use Moose::Autobox;
    use MooseX::Types -declare => [qw(SlotOffset VectorOfWeights PercentageInt)];
    use MooseX::Types::Moose qw(ArrayRef Int);
    
    use constant TOTAL_SLOTS = 7 * 24 * 4;
    
    BEGIN {
        subtype PercentageInt,
            as Int,
            where { 0 <= $_ && $_ <= 100 },
            message { "$_ does not is not an integeter between 0 and 100" };
    
        subtype VectorOfWeights,
            as ArrayRef[PercentageInt],
            where { $_->length == TOTAL_SLOTS }
            message { "Vector must have ".TOTAL_SLOTS." entries, not ".$_->length };
    
        subtype SlotOffset,
            as Int,
            where { 0 <= $_ && $_ < TOTAL_SLOTS };
    
        class_type 'DateTime';
    
        coerce SlotOffset,
            from 'DateTime',
            via { ($_->wday_0 * 7 * 24 + $_->hours) * 4 + int($_->minutes / 15) };
     
        # Let's allow clients not to care about using DateTime by allowing 
        # them to simply pass the results of calling 'time()' - It's not like it's
        # still 1970...
    
        coerce SlotOffset
            from subtype(as => Int, where { $_ > TOTAL_SLOTS }
            via { to_SlotOffset(DateTime->from_epoch(epoch => $_) };
    }
    
    has vector => (
        is => 'ro',
        isa => VectorOfWeights,
        required => 1,
    }
    
    ...

Now, if we were programming in plain old Moose, we could rewrite weight_at like so:

sub weight_at {
    my $self = shift;
    my $offset = to_SlotOffset(shift);
    $self->vector->[$offset]
}

Which would be pretty sweet, but we’re using MooseX::Declare; there’s an even better way:

method weight_at (SlotOffset $offset does coerce) {
    $self->vector->[$offset];
}

Sweet!

We could stop there, but I had an insight. What we’ve got here is basically a wrapper around a delegation to our vector, and Moose’s new native types feature let us express the delegation to the vector quite neatly, like so:

has vector => (
    isa => 'VectorOfWeights',
    is => 'ro',
    required => 1,
    traits => ['Array'],
    handles {
        weight_at => 'get',
    },
);

...

around weight_at (SlotOffset $offset does coerce) {
    $self->$orig($offset);
}

This could be overkill when vector is a simple ArrayRef as we have here, but the pattern of delegating declaratively in the attribute definition and then munging arguments in an around handler is applicable to more than just argument transformation. A typical delegation pattern involves having the delegating object passing itself in as an argument to the method delegated to. The nature of Moose’s handles declarations makes that impossible to do within the attribute declaration, but it’s easy to fix with an around helper:

around delegated_method (Any @args) {
    $self->$orig($self, @args);
}

(If you’re wrapping more than one method in this fashion, you should probably consider using a plain old Moose style around handler, which lets you wrap multiple methods with around @delegated_methods => sub {...}

So, at the end of all that, and after we’ve extracted our Type declarations into WeightVector::Types, we have:

use MooseX::Declare;
class WeightVector {
    use WeightVector::Types qw(VectorOfWeights PercentageInt SlotOffset);
    
    has vector => (
        isa => 'VectorOfWeights',
        is => 'ro',
        required => 1,
        traits => ['Array'],
        handles {
            weight_at => 'get',
        },
    );
    
    method current_weight {
        $self->weight_at(time());
    }
    
    around weight_at (SlotOffset $offset does coerce) {
        $self->$orig($offset);
    }
}

And we’ve pushed all knowledge of DateTime off onto our type declarations and gained a boatload of handy bounds checking. We’ve also got a new tool for handling tricky delegation setups in the handles/around combo.

Notes

Motivation

I realise that this looks like a radical decomposition of the class with very little motivation, but it was driven by tests and by some other requirements that I’ve removed from the body of the post. In particular, the type coercions were driven by the need to build particular vectors for testing, a key method being:

method set_weight (PercentageInt $weight, 
                   SlotOffset $from does coerce,
                   SlotOffset $to does coerce)
{
    ...
}

Type coercion is wonderful

Generally, I’m not a fan of static typing. I’m from the “duck type all the way” school of programming,³ so most of my method declarations have no type declarations. But type declarations, especially ones that coerce, make so much sense on methods that make up the public protocol of a class. I only use type declarations on internal methods when I need a narrower coercion, or if I’m using MooseX::Multimethods, which I still haven’t used for anything but exploration.

Updates

Thanks to Chris Dolan for spotting that I’d got the SlotOffset coercion completely wrong. The real code’s doing the right thing, but that’s what comes of recreating code from memory.

¹ This was actually motivated by trying to write tests to verify that the weights were correctly set.

² I’m declaring these in a BEGIN block of the class itself mostly for explanatory purposes – there’s a good case for moving them out into a separate file and pulling it in with use.

³ Except during my periodic attempts to learn Haskell. I’ve learned Haskell at least three times now.

In Ruby, when you’re doing division on integers, things can get a little counter intuitive. For instance, 6/4 is obviously 1. At least, it is until you decide that you’d rather have numbers that behave a little more like ‘real’ numbers and you do require 'mathn', a module in the Ruby standard library (ie, it comes as part of ruby). Then you enter a whole new world of rational maths, where 6 / 4 returns 3/2.

Several very fine and useful Ruby gems rely on the workings of mathn, including ruby-units, which is a spiffy tool for avoiding problems when one team is working in kilometres and the other in miles and it’s no fun at all when your space probe is suddenly incommunicado.

Other fine and dandy Ruby gems include ultrasphinx and webrat. Both of these two (and no doubt others) rely on the the fact that 302/100 == 3.

Hmm… can you see my problem?

Please, if you’re working on a gem that you intend to publish widely, then adopt the practice of never trusting that dividing an integer by another integer will return a third integer. You’re not even making yourself a hostage to some other gem, you’re making yourself a hostage to the standard library. Always do (an_integer / another).to_i and your code will be so much more robust.

I’ve got a pull request and lighthouse ticket in for webrat and, once I’ve hit ‘publish’ on this post, I shall be doing the same thing for ultrasphinx, but I’m sure there are other gems out there with the same problems. Please people, check your assumptions.

Back in (crikey) February, I gave a talk at the London Perl Mongers’ technical meeting about Moose for Ruby Programmers and wrote it up here. Mike Whittaker was in the front row of the audience with his iPhone and, a couple of minutes in, started a voice recording and gave me a copy.

So… finally… I’ve taken the time I should have been using to write another article for The H and wrestled the slides and the audio into something like sync and uploaded the results to Vimeo for your viewing pleasure.

An introduction to MooseX::Declare from Piers Cawley on Vimeo.

I got a bit of stick on IRC last night for some of the choices I’d made when I was writing Test::Class::Sugar, in particular because one of the prerequisites is chromatic’s handy and opinionated Modern::Perl module. The ‘controversial’ aspect of Modern::Perl is that, when you use it, your code won’t run on any Perl before Perl 5, version 10.

The thing is, I don’t care about older perls any more. Version 10 features like ~~ and // are too convenient to fart about writing circumlocutions just to run on a version of Perl that I have no intention of ever using again.

Actually, that’s not quite true, those version 10 features are too convenient for me to fart about working around their absence. If you think that a module I write is useful enough that you want it to work on version 8, then of course I’ll accept your patch. But don’t be surprised if, when I start adding new features, I break the backward compatibility.

Also, on the happy day that version 10.1 escapes the pumpking patch, I’ll be setting that as my minimum perl version even if chromatic doesn’t bump the version number in Modern::Perl.

It’s all about the virtues

Your context is different from mine. I’m writing in Perl again for my own amusement more than anything. There are developments in modern Perl – tools like Moose and Devel::Declare – that I think are exciting and important. The Announcements project I started was as much about playing with the new tools as it was about trying to write something of wider utility. Test::Class::Sugar arose as a direct result of attempting to write Announcements and the desire to write test classes without hoopage. My principle drive then, is impatience to get Test::Class::Sugar to the point where I can get back to writing Announcements.

But then laziness and hubris kick in. So the code needs some polish. The parser and the code generator need to be disentangled, I need to get Adrian Howard to apply the little patch I had to make to Test::Class. Laziness demands I document it.

Impatience tells me to lean on the features of modern perl – that way I can get back to being a user of the new library as quickly as possible. Laziness tells me that I’m not going to need backwards compatibilty. Hubris tells me my work is good enough that someone who does will like it enough to send me a patch.

Everybody wins.

In which your correspondent does magical battle with the guts of Perl and emerges bloodied, but unbowed with a useful principle to code by.

Skip to the conclusion if you’re uncomfortable with the guts of the Perl runtime

Test::Class had me tearing my hair out earlier. There I was, happily transforming

test something {
  ok 1;
};

into something very like¹:

*test_something = 
  Sub::Name::subname(
      'test_something' 
    => sub : Test { ok 1 }
  );

through the magic of Devel::Declare, but Test::Class didn’t seem to be playing fair. Instead of letting my tests run happily, it was complaining that it:

cannot test anonymous subs – you probably loaded a Test::Class too late (after the CHECK block was run). See ‘A NOTE ON LOADING TEST CLASSES’ in perldoc Test::Class for more details

The thing is, I wasn’t loading Test::Class too late. The problem is that, at the point I applied the Test attribute to my sub, the sub didn’t have a name and, because of the constraints you’re operating under when you’re using Devel::Declare to do code transformation, there was no obvious way to give it a name in time.

Incompatible magics

The trouble is, Test::Class does what it does through the magic of compile time code attributes, and, further, it relies on the fact that if a perl subroutine that gets inserted into the symbol table like this:

sub has_a_name {...}

Then, when you get hold of a reference to that code by other means (say, in the subroutine that handles the setting of an attribute, that code ref knows its own name. However, if a subroutine that ends up in the symbol table like this:

*anonymous_ref = sub {...};

Doesn’t know its name, unless you take advantage of the Sub::Name module.

So, in my generated code, I was giving my coderef a name, but it was happening to late. At the point that Test::Class::Test method was seeing the coderef, the coderef was anonymous.

My magic and Test::Class’s magic were incompatible.

The thing is, both sorts of magic are really just sugar for some pretty mundane donkey work. Test::Class does what it does through attributes because no flesh and blood programmer in their right mind would want to write something like this every time they wanted to write a test method:

sub test_something {
  ...
}
__PACKAGE__->mark_as_special_method('test_something', 'test', '3');

In fact, mark_as_special_method doesn’t even exist as its own subroutine. The code that marks a method as special is just part of the body of the Test attribute handler.

Conclusion

Which brings me neatly to my conclusion.

When you’re designing a module that does anything magical, consider starting with a mundane core API that handles the business side of things. Then layer your magic on top of that API. Then document the API and the magic. Obviously the magic bits go up front in the docs, and the API goes in its own section (or even podfile) down at the bottom, where only eejits like me, who want their magic to work slightly different to yours, will bother reading it.

Obviously, I’m motivated by an issue I’m having with a particular module from CPAN, but the principle of separating the magic and the mundane is applicable everywhere. It’s called Separation of Concerns, or The Single Responsiblity Pattern. I call it a Just Story.

You’ll find the pattern in well designed websites that are using unobtrusive javascript to wave an AJAX wand over the site. You’ll see it woven through books like The Structure and Interpretation of Computer Programmers – where it’s called an Abstraction Barrier.

Patches sent

It turns out to be very easy to add mark_as_special_method (though I actually wrote it as ‘add_testinfo’ in the patch) to Test::Class. It’s about as straightforward an Extract Method refactoring as I’ve ever done – even without automated tools, I managed not to fuck it up. There’s a patch in Adrian Howard’s inbox, and I’m hopeful that it’ll be applied soon.

¹ Not exactly – that’s the result of calling the shadowed test subroutine which was the result of the code transformation.

Check out the osfameron fork of Devel::Declare for the beginnings of some decent documentation which explains what’s going on.

One aspect of coming back to Perl for ‘recreational’ programming is that if, like me, you’ve declared war on @_ and boilerplate code, then testing can be somewhat trying. The Perl testing framework that best fits my head is Test::Class, which is an excellent Perlish implementation of xUnit style testing. If you’re unfamiliar with the, library, Ovid is writing what’s shaping up to be an excellent series of introductory articles about it at http://www.modernperlbooks.com/.

The problem I’m having with Test::Class at the moment is that I can’t write:

use MooseX::Declare
class Test::Person
  extends Test::Class
{
  use Test::Most;

  method class_under_test {'Person'}
  
  method startup : Test(startup => 1) {
    use_ok $test->class_under_test
  }

  ...
}

Test::Class is doing too much in its initialization phase, and relies too heavily on code attributes, for it to play well with MooseX::Declare. Drat.

On reflection though, this might be a good thing, because maybe MooseX::Declare isn’t really what’s needed. What I’d like to write is something like:

use ...;

testclass Test::Person
  exercises Person
{
  startup class under test should be usable (1 test) {
    use_ok $test->class_under_test
  }
}

And have the library ‘…’ expand the testclass declaration into something that looks like the first code snippet. After all, if MooseX::Declare can work without source filters, it should be possible to come up with something nicely declarative for specifying test classes.

Obviously, there’s nothing on CPAN that does this yet though. So I went fossicking through MooseX::Declare to see how it works¹ and discovered thing of Lovecraftian beauty that is…

Devel::Declare

Devel::Declare is possibly the most hostilely documented library I’ve ever come across. Its documentation only begins to make sense when you already understand enough about how it works that you don’t really need the docs. What it does is to let you declare your own Perl keywords. You could, for instance use it to introduce given/when into versions of Perl that don’t have it yet. You declare your keywords and associate them with parsers. When, during its compilation phase, perl hits one of your keywords in the right context, it hands off to your parser which can then do what the hell it likes in the way of code transformation, before handing control back to Perl, which then parses the transformed code as if that was what was there all along.

So, to want to transform that testclass syntax I just pulled out of my ass into a real Test::Class package, I just need to write an appropriate parser and code generator, perform the appropriate Devel::Declare incantations, and I’m laughing.

Making progress

So far, I’ve got to the point where I have a working testclass keyword, but nothing yet for the ‘inner’ bits (setup, test, teardown, etc). I can write:

testclass Test::Person
  exercises Person
{
  ...
}

testclass Test::Person::Employee
  extends Test::Person
  exercises Person
{
  ...
}

and, as I write this, I’m realising that the syntax I’d cooked up for using extra test helper modules:

testclass AnotherTest
  helpers -More, -Exception, Carp
{
  # use Test::More;
  # use Test::Exception;
  # use Carp;

  ...
}

would probably read better as:

testclass AnotherTest
  +uses Carp
{
  # use Test::Most;
  # use Carp;

  ...
}

and also that I want this:

testclass exercises Person {
  ...
}

to build me a Test::Person class.

What’s still blowing my mind about Devel::Declare’s possibilities is that I’m no longer constrained to writing a Domain Specific Pidgin which works by building a tower of proxy objects and weird evaluation contexts to produce something that’s legal code in the host language, but which has the feel of another language. With Devel::Declare, I control the horizontal and the vertical until I choose to hand control back to Perl. Right now that means my error reporting is disgracefully bad, but it also means that I can roll a syntax that makes sense without worrying about how I’m going to get perl to parse it.

One of the things I find frustrating about writing RSpec specifications is that describe and it both want to be first class keywords – it feels like you should be able to write:

describe SomeClass, "in some context"
  before each
    # set things up
  end
 
  it "should do something or another"
    ...
  end
end

But, because RSpec works by taking advantage of Ruby’s block magic, you have to write:

describe SomeClass, "in some context" do
  before :each do 
    # set things up
  end

  it "should do something or another" do 
    ...
  end
end

I definitely prefer the version without the extraneous dos and the gratuitious : before each in the before declaration. Does anyone feel like writing devel/declare.rb?

Show us the code!

If you want to see the current state of my Test::Class::Sugar art, the place to look is http://www.github.com/pdcawley/test-class-sugar. At the time of writing it relies on http://www.github.com/rafl/devel-declare and doesn’t have anything so useful as documentation, a Makefile.PL or even any tests beyond the collection of code samples that is t/initial.t. Expect all those when and if I push it to CPAN.

Caveats

Yes, I know that this sort of metasyntactic abstraction is trivial in a Lisp. I just happen to like syntax, okay?

Update 20090312

use Test::Class::Sugar

testclass exercises Foo 
    +uses -Warn
{
    ...
}

Now generates

{
  package Test::Foo;
  use base qw/Test::Class/;
  use Test::Most;
  use Test::Warn

  ...
}

So that’s one hurdle jumped. And I now know how to write the various method helpers and, when I get the appropriately shaped tuits, I shall actually write the damned things.

Then all I have to do is document it.

And write up a proposal about it for YAPC::Europe.

Update 20090314

I now know what a plan looks like:

test with multiple assertions << 3 {...}

And, more importantly, I’ve implemented, and documented everything and am almost good to cut a 0.001 distribution. I need a few ducks up on CPAN, but once that’s done, we’re good and I can get on with parameterizing some of the assumptions that are hard coded at the moment.

¹ Something I swore blind I wasn’t going to do in my London.pm presentation. Seems my word isn’t to be trusted…

Here’s something interesting. This is the test directory from the very first version of Perl, released in 1987 and described by Larry as ‘a “replacement” for awk and sed’. Day one of Perl, and we already have the bare bones of the Test Anything Protocol and a prototype for Test::Harness in the TEST file.

If we truffle about in the various other branches we find other useful milestones for module developers:

• 5.000, in 1994 came with the first iteration of h2xs which could be used to generate the basic boilerplate for a perl module. Even today, with more sophisticated module starters available, you won’t find a CPAN module of repute that doesn’t follow the basic directory structure laid down in this utility. ExtUtils::MakeMaker generates a Makefile with a test target which runs all the tests in the t subdirectory
• 5.002, in 1996, h2xs starts stubbing test.pl
• 5.003_12, late 1996, first version of CPAN in the Perl distribution. From day one, CPAN would not install a module if any tests failed, unless you forced it.

Meanwhile, Ruby:

• Has only recently embraced a language test suite
• Appears to have no standard layout for gem distributions
• Doesn’t run tests as part of the installation process for a new gem

Is it any wonder that chromatic gets a little cranky when sweeping claims are made about how spiffy Ruby’s testing culture is?

There are those who claim that CPAN is Perl’s shining glory, but it’s not really the collection of servers, it’s the toolchain that enables it, and that toolchain can exist because so many libraries follow a pretty minimal set of conventions.

I’d love to see the Ruby community settle on a similar, single, set of conventions for the way things should work. Start with a guarantee of a either a top level Rakefile or setup.rb with build, test and install tasks. Make rubygems run the tests before installation, if the target is available, and halt the installation if they fail. Make it easy to send reports of test failures to module authors (look at the Perl CPAN and CPAN Testers sites, and their associated tooling for ideas).

I know… I should STFU And Write Some Code.

Update

Further investigation shows that gem install -t whatever does run the tests as part of the installation process. The capability is there, but it’s turned off by default. How depressing.

It doesn’t matter what language you program in, there’s one very important thing that you should do if you want to get better at your craft.

Read. The. Fucking. Manual.

Seriously. Tutorials are good. Blog articles can be good. Code snippets are probably better than a poke in the eye with a fecally decorated sharp stick.

But the language reference? That’s gold. Read the docs. Read the source. Whatever you do, get the fundamentals down.

Let’s say you’re writing a ruby library. Something you want and expect others to use. Here’s how to reduce those others to incandescent rage:

require 'rexml'

include REXML

module MyShinyModule
  ...
end

No. Not no how. Not never. You just crapped in my namespace and stomped on who knows how many of my own classes. Now, include is a fine and dandy thing, and it certainly has its uses, but using it at the top level of your library files is not one of them.

Ahem.

Can anyone guess which library has been wasting my morning?

One of the people who make me think that Perl is still worth knowing is chromatic, the ex-editor of O’Reilly’s perl.com. He’s one of the core team who are working to bring the Perl 6 Christmas, a vector of test infection, an extreme programming practitioner and, right now at least, no longer on Giles Bowkett’s christmas card list.

When I wrote the Healthcheck: Perl for (oh ghod, they’ve rebranded) The H, I mentioned that O’Reilly hadn’t announced Perl: The Good Parts yet. This was an allusion to something chromatic had said when I interviewed him for the article – at the time he was pitching that very book, but it seems he got knocked back.

So, he’s writing a book and a blog on Modern Perl, and a jolly good blog it is too. It reads like a manifesto in places, but it’s a good manifesto, so that’s okay. You should read him. If you’re a publisher, you should definitely talk to him. If you’re a ruby programmer, don’t be in the least bit surprised if he’s rude about the ruby community…