Just A Summary

... timing. You either have it or you don’t.

Does this count as good timing?

1. Finish up some improvements to the way Typo sweeps cached pages
2. Announce Typo 5.0
3. Go down with a horrible cough and cold that leaves you exhausted and incapable of hacking
4. Discover that the ‘improvements’ in Typo’s cache sweeping can, occasionally, cause it to wipe the entire Typo installation directory
5. Stagger out of bed. Attempt to fix problem
6. Release Typo 5.0.1
7. Discover that the fix doesn’t work
8. Bleargh!
9. Let your co-maintainer deal with the fall out before releasing Typo 5.0.2 which does fix the cache sweeper
10. We hope
11. Recover enough to write a blog entry

Okay folks, Typo 5.0.2 is out and it appears to be working. I’m running it here, and I’ve had no problems so far. I’ve still got the cold, but it’s nowhere near as horrible as it was (went to bed at 5pm on New Year’s Eve, woke up at 11am on New Years Day – first time I’ve missed the turning of the year in ages).

Right, we’ve cut a Typo 5 gem and it’s on rubyforge and heading to various mirrors I hope. Frédéric’s writing the release notification which will be appearing on Typosphere Real Soon Now.

It’s been a surprisingly tricky process – we’re now requiring Rails 2.0.2 because the workings of view_paths have changed in a way which means we can’t quite make themes with Rails 2.0 and 2.0.2 and working with the edge seems like the more sensible proposition. If you’re on the bleeding edge, you should find that you get the right Rails via svn:externals anyway.

Typo futures

Meanwhile, I’ve been playing with stet and I’ve come to the conclusion that, although there’s mileage to be had in a radically slimmed down approach to the way Typo works, I’m better off simply removing the misfeatures from Typo and building from there – there’s a surprising amount of stuff that needs to be done in a competent blogging engine that Typo gets right – starting again would be throwing the baby out with the bathwater I think.

However, this does mean that if you’re following the Typo SVN trunk, you’ll be seeing a reduction in features in the short term. We’ll be copying the current trunk to a 5-0-stable branch before we start with the featurectomies though, so if you’re just after bugfixes, you’ll be better off there.

Multiblogging

We’re aiming to have multiblogging in the next release, but we’re rethinking the how of it. Right now, the ‘Blog’ object adds a bunch of complexity to code that would be much happier simply assuming that it has the database to itself. So we’re going to look at switching to a database per blog approach, that way our core code can pretty much forget about the complexities of multiblogging, and (at least initially) anyone who wants multiblogging can get there by monkeying with configuration files – of course, we intend to add a web based admin interface once things settle down and we know how things are going to work.

Caching

Caching is always a bugbear in any typo installation. Because we want to be installable on the widest possible range of hosts, we can’t rely on the presence of handy tools like ‘memcached’. Also, some of our users are operating under some fairly severe memory and process constraints, so it makes sense to have the webserve serve static files as much as possible. Meanwhile, tools like Evan Weaver’s Interlock are pointing the way towards seriously effective fragment caching. I shall be looking into implementing something that conforms to the interlock interface, but which can use an arbitrary cache backing store for fragments and maintain a full page cache. It’ll be interesting to find out if this is doable…

Atom Publishing Protocol

ActionWebService is going to go away – it’s already in the ousted branch of the rails SVN repository, and including it in Typo to support the various different admin APIs is getting painful. So, we’re going to preempt it. We won’t be getting rid of the various XMLRPC APIs until the pain becomes too great, but we are going to be concentrating on implementing, and strongly favouring, the Atom Publishing Protocol.

Feeds for everything

In particular, we’ll be adding atom feeds for all sorts of administrative data as a means of enabling people to write external tools for, say, spam protection, comment moderation and notification tasks. Right now, there’s a great deal of computation happening on the server side every time someone, say, comments on a post – in the kind of resource limited environments some people are running Typo in, that’s too much work. Switching to a feed + APP approach should help enormously with resource utilization.

Speaking of resources…

Using the server to render article previews is… suboptimal. Expect to see a javascript based preview system akin to the one I use for comments here.

So, if you’ve been watching the Typo tree, you’ll see there’s been a fair amount of activity on it since Rails 2.0 got released. There’s a new default theme replacing the rather creaky ‘azure’, and a fair amount of work on getting our code compatible with the current state of Rails. As we work on this, it becomes apparent that Typo’s code is getting horribly brittle. I have said before that there’s been several places where we’ve zigged before Rails zagged, and we’re paying the price for that. It doesn’t help that our test coverage is distinctly ropy either – and I’m probably guiltier than most for letting things get into that state.

So, our goal is to get what we have cleaned up and working with Rails 2 before releasing Typo 5.0. Once that’s done, that line of code will go into maintenance mode – there are still plenty of bugs to fix and documentation to write, but I’m afraid that extending that base is becoming too much of a chore.

Which is why I have a new path in my local svk repository, //stet. I’m using this for experimental development of a new, slimmed down blogging engine that will be, first and foremost, a capable Atom Publishing Protocol host. Things like spam processing will be removed from the core of the application, but we’ll provide a suite of webservice clients that will consume the ‘unmoderated feedback’ webfeed and use APP to either approve or delete the feedback as appropriate.

Theming (at least initially) will probably be confined to Javascript and CSS changes, and I’m even thinking of exposing the sidebars as Atom collections – certainly I expect that, in the first cut, sidebars will be static – if you want content that looks dynamic you’ll have to do it via javascript.

My initial goal is to slim things down as far as I possibly can – I want to build a blogging engine that can cope with the tight memory constraints of shared hosting by off loading much of the heavy lifting to client boxes. After all, I have far more processing capability available to me on the laptop I’m typing this on than the slice of Site5’s hosting infrastructure that’s actually running the blog. By making things small and static, I also hope to wring good performance numbers out of the tool as well – expect aggressive page caching at the very least.

Another important goal is easy migration of Typo databases. I expect to be writing models and controllers from the ground up, but converting the database should just be a matter of running a migration.

Experimental

Of course, stet’s currently very experimental – about the only thing that’s actually written so far are a couple of routing plugins which should help radically simplify our routes.rb (expect an article’s url to change from /articles/2007/12/16/rails-20-and-the-future-of-type to /article/2007/12/16/rails-20-and-the-future-of-typo, but with a redirect in place to cater for the old style urls). I may have grandish plans for the thing, but I could equally discover that I’m off up a blind alley, in which case you can expect me to return to the current typo codebase with a few more lessons learned.

ActiveResource?

I remain unconvinced by ActiveResource as a technology. I agree with the authors of RESTful Webservices – good webservices are joined up. They take full advantage of what could be described as the defining technology of the world wide web, the URL based hyperlink to knit resources together in a discoverable fashion. An ActiveResource based webservice may well be a good HTTP citizen, but it’s still not really ‘webby’ enough for my taste. Which means the Atom Publishing Protocol will remain my friend for most of the things I hope to do with stet. It may be harder to write a good APP server, but I’m convinced that it’s a much better interface for clients, and you should always favour ease of use over ease of implementation. If nothing else, we’re aiming to have more users than developers. Many more.

Whee! Sat in my home directory is a version of typo that appears to work with Rails 2.0. I ended up giving up on the themer approach which proved to be very hard to get up and running transparently – things kept disappearing off around infinite loops, which is no fun at all, let me tell you.

So, ended up cheating and ended up monkeying with @@view_paths directly, which is almost certainly against the rules, but has the cardinal virtue of working.

I’m sanguine about it because the way themes worked on rails 1.2.x also involved fiddling with undocumented methods which aren’t there in edge rails.

So, it now remains to walk around the thing, kick the tyres, write a few more tests and generally get myself to the point where I’m happy that it all works, and merge my local branch with the main line. I’m going to wait until 2.0.0 proper has been released, but now that I’ve done the work I expect to be merging it sharpish.

During DHH’s keynote at RailsConf Europe it was apparent that there’s a great deal to like in edge rails, so I thought I’d have a crack at getting Typo up on it.

Ow.

I’d expected the pain points to be related to routing, but it seems that the rails routing system is approaching the level of the Excel calculation engine – nobody dares touch it for fear of breaking things, so typo’s custom routes seemed to work quite happily. There were a few things that have been deprecated, pluginized or moved out of the set of modules that’s automatically included when you do a rake rails:freeze:edge, but they were pretty easy to sort – the deprecation messages are a good deal more informative now than they were last time went deprecation squashing. There’s a surprising amount of stuff that’s been removed without any deprecation warnings though, which isn’t very sporting. DHH said there would likely be a 1.2.4 release (possibly a day before 2.0) with a bunch more deprecation warnings covering everything that’s actually going away, so if you’re thinking of moving a maturish app to Rails 2.0 it might make sense to wait for 1.2.4, install that, squash warnings, and move on up to 2.0.

The real pain comes from themes. Typo’s themes rely on Rails internals working in a particular way, but they don’t work like that any more. In theory, the internals appear to be more theme friendly, related to allowing plugins to include views. The problem is, that it’s possible to change Typo’s theme without restarting the server, and the new themish internals don’t expect anything to change until the server’s restarted.

So, I’ve been playing with plugins. The most promising approach appears to be that of the themer plugin, which gets pretty close to doing what we need, and does it in a way that seems like it should work with both 1.2.3 and Edge Rails. It does appear to be making some radically different assumptions about the structure of the themes directory, but the basic framework is good and I should be able to make things work by making our current them object conform to Themer::Base’s interface and duck type my way to the sunny uplands of Edge Rails compatibility.

Which will be nice.

I like the themer approach a lot. Instead of monkeying about in the guts of rails, it monkeys about in front of Rails. It overrides render so that you can pass it a theme/lookup object. If it sees a lookup object, it uses that to rewrite the rest of the render arguments into a form that will render the right thing using the standard implementation of render. In a work project I’ve taken a similar approach to handling polymorphic routes for things like:

map.resources :pictures do |pics|
  pics.resources.comments
end

map.resources :users do |users|
  users.resources.comments
end

I ended up with a to_params method defined on my Comment model, and stuck an extended url_for in front of the default Rails version, which looks something like:

def url_for_with_to_params(*arguments)
  if arguments[0].respond_to?(:to_params)
    with_options(arguments.shift.to_params) do |mapper|
      mapper.url_for_without_to_params(*arguments)
    end
  else
    url_for_without_to_params(*arguments)
  end
end
alias_method_chain :url_for, :to_params

Which is so much neater than the last time I attacked this particular problem (see the acts_as_resource plugin).

One of the nice things about Rails is that, although it’s opinionated and somewhat liberal with the syntactic vinegar for things the core team don’t think is the Right Way, they’re pretty good at leaving the door open for people like me who have other opinions. Both the themer plugin and my as yet unpluginized extension of url_for work by using existing capabilities in new ways and, because those capabilities are documented we can expect them to continue to work over multiple versions of Rails. Plugins that achieve similar effects by monkeying with Rails’s internal interfaces are hostages to fortune. Internal interfaces are free to change at any time, even between point releases, so a plugin can be left high and dry with surprising rapidity. Just ask the Rails Engines folk.

If you’ve been running on the Typo edge recently, you’ll be all too painfully aware that there have been issues with the cache being flushed at the wrong times and not flushed at all at others. Which is not a happy state of affairs.

However, I’ve recently got back to working on Typo and, after a few warmup refactorings and some missteps, I think caching is working properly now.

There’s lots more work to do (there always is) but I reckon that you could do worse than bump your installation to r1513, which is what I’m running here.

There’s something enormously liberating about writing an RSpec description that starts like:

describe ArticlesController, "feeds:" do
  before do
    @the_mock = mock('everything', :null_object => true)
    ActiveRecord::Base.stub!(:find).and_return(@the_mock)
  end

  it "/articles.atom -> atom feed" do
    get :index, :format => 'atom'
    response.should render_template('_atom_feed')
  end

  ...
end

The :null_object flag to rspec’s mock function is remarkably potent.The resulting mock will return itself from any method call that hasn’t got some other expectation set up. When I’m testing that my index methods render the appropriate views for the format, I don’t care that all the various variables have been set up correctly – I’ve already tested that in another description – I just want to get to the point where I’m about to render a template.

What this does is decouple my tests. I can change the way that the index method fetches its stuff from the database and I’m only going to have to change the innards of the specs that test that.

I tend to think of this as being analogous to the object oriented pattern of trying to write your methods at a single level of abstraction. Within a given spec, I should only be setting up expectations that are directly related to what I’m testing.

So, I just pushed the first step of what I’m thinking of as the Great Typo Controller Reorganization to the typo repository and updated things here. It’s always scary when I do that – local testing’s all very well, but running on a live site is a different matter.

I’ll be talking about the changes in a later post, there’s a couple of things in the recent change that I’m rather pleased with, as well as a Rails plugin that I hope will be widely useful, but which I need to document.

Wow:

$ rake spec
...

156 examples, no failures

$ ./script/spec spec
...

156 examples, 2 failures

$ rake
[unit tests, all pass]
[functional tests, all pass]
[specs...]

156 examples, 2 failures

For extra points, the 2 failures from running the plain rake are not the same as the failures from running ./script/spec spec. And if I run ./script/spec spec after a full rake run, I get a host of extra failures.

I wonder what I’m doing to so comprehensively screw up test isolation.

Ho hum.

It’s definitely fun to commit a major rework of something that’s been bugging you to SVN. It’s slightly less fun to check it out in your production server and have it fall over until you remember you to retweak the environment.rb file.

Anyhow, we’re now running on Typo 4.1.1+r1438. Typo 4.1.1 got released last night and, pretty much as soon as it had cleared the gates I checked in a major rework of the Article and Feedback state mechanisms.

What state mechanisms?

When you’re modelling blog posts, and comments, it’s very easy to think of them in terms of simple status flags. An article is published, or its a draft, so we have a #published boolean and we’re done.

Except, what about an article that you want to publish at a specific time? And how do we work out when to send notifications and trackback pings? Should you send the pings again when an article is first published then withdrawn, then published again?

It’s complicated, and you can end up with a rats nest of complex conditional code.

So, you work out that an article can be in one of several states. We went with :new, :draft, :just_published, :published, :publication_pending, :just_withdrawn and :withdrawn, and we used the State Pattern to handle this. All our tangled conditionals were replaced with simple delegations to the state object. There’s still some conditional code, but there’s a great deal less of it now.

Until recently, we implemented the states using Rails’s composed_of helper, but it’s not really suited to the task – you know your code isn’t happy when you find yourself calling a class ContentState::Factory and overriding its new method. It was also really hard to divine the workings of the state machine.

So, during a discussion of Coding without ifs I found myself sketching a way of setting up a state field declaratively and thought to myself “Hey, I could implement that for Typo and make things a good deal clearer…”

So I did.

Now, if you look at the top of article.rb, you’ll find:

  has_state(:state,
            :valid_states  => [:new, :draft,
                               :publication_pending, :just_published, :published,
                               :just_withdrawn, :withdrawn],
            :initial_state =>  :new,
            :handles       => [:withdraw,
                               :post_trigger,
                               :after_save, :send_pings, :send_notifications,
                               :published_at=, :published=, :just_published?])

  include States

And it sets up the delegations (as well as a set of #new?, #draft?, ... predicates based on the names of the valid states) for you. I pulled the States module out into a separate article/states.rb file, but given a simple set of states there’s no reason not to declare them inline.

I’m really pleased with it. I’ll probably do a bit more work on the interface (I want it to be more language like), add a has_strategy declaration (which I intend to use for our feedback spam checking system) and then extract a plugin from it. If you’re interested in doing something similar in your own code right now, the thing you need to pinch is lib/stateful.rb in the Typo distribution, and the only documentation is the source. Check app/models/feedback/states.rb and app/models/article/states.rb for examples of states written to this interface.