A Modest Success

We use Bazaar for version control and we use Launchpad for hosting code and doing code review. Here’s an example one.

Code review is quite often fairly boring. My guess is that one in ten actually provoke interesting changes. Most of our branches make small changes, all of our changes are covered with tests, and we have confidence in our test suite and our ability & willingness to make follow-up improvements. Thus, rather than seeing code review as our last line of defence for ensuring code quality, it’s an opportunity to learn and to perhaps accelerate the process by spotting problems earlier.

Once a merge proposal is approved, magic happens.

In this case, “magic” is a Jenkins instance that uses tarmac to poll Launchpad to figure out which merge proposal are approved and thus ready to land.  It runs our test suite, and if it passes, lands the change. If it fails, it posts the failure to the merge proposal and our IRC channel and marks it as needing more work.

It’s important here that our test suite is fast, that it’s reliable, and that it gives useful error messages when things fail.

When the change is to a dependency, then we run the tests of the things that depend on it with the latest change.

After a successful change to trunk, we have a Jenkins job that triggers an automatic deployment to our staging server.

All of this takes less than five minutes.

Once it’s on staging, our “QA” website lists it as a revision that needs to be manually verified. What we do there isn’t really QA, but instead making sure that if we roll it out to production, we won’t break anything. As with code review, we are confident in our ability to fix things later.   

We tend to do these in batches, as our webops are insanely busy. Once we’ve got a group of changes that are safe to deploy to production, we ping the webops, who then (I think) run a single command that deploys to production very quickly. Sometimes because of interrupts, it can take twenty minutes to an hour to get someone to do this.

Completion

At this stage, we return to our definition of done and check to see if the change actually provides the value we thought. 

Because automatic packaging is an inherently fuzzy problem, we run a barrage of real world data through the new system to see if the recent changes actually help. This also generates interesting qualitative data that gives us hints on where we might want to work next.

Principles

**
**

This is the end result of a lot of work, mostly by James Westby with me cheerleading from the sidelines. We’ve iterated through it a lot, making things faster and more reliable, generally by going from polling to events, by cutting out unnecessary review steps, and by making visibility improvements for when things are broken.

Underlying it all are a few principles that we have found to be either true or useful:

• a thing is not done until users are happily using it
• more strongly, a thing is not valuable until users are happily using it, until then it is wasted effort 
• we should be able deploy any change in the main code or a dependency within minutes at any time, every time
• all deployments must be from trunk, all revisions of trunk must be tested and known-good
• we can avoid a lot of needless effort by asking ourselves why we are doing what we doing, and by being explicit about our assumptions
• regressions are intolerable, everything else is tolerable, because we can, will and do fix it