Slow tests

Robert Collins of Bazaar, recently blogged about the cost of slow tests. I couldn’t agree more.

Launchpad has a very slow test suite. It takes upwards of five hours to get a change into our “known good” branch. It’s been a source of pain for the team for years, and now that Launchpad is Free Software, it’s becoming a pain to new contributors.

We’ve tried a few things to deal with it, none of them great. First up, we’ve got a tool for running the test suite on EC2 instances. It’s nice, but it’s still too slow. I’ve written a “faster tests” spec outlining some of the options we have; but specs never got software written. Likewise, there are a bunch of bugs flagged as build-infrastructure, many of which address the slow test sped. We also have a rotating “Build Engineer” position within the team to address those bugs.

It’s probably not enough though. From watching Rob’s blog posts, and the emails to the Bazaar mailing list, I think that what Launchpad really needs is someone with a passion for solving the problem (i.e. making Launchpad hacking fun) who is willing to lead the way, and a full commitment from the rest of the team for getting the runtime down and keeping it there.