Daily Releases to Production

During the course of 2007 the DevCreek team released 10 versions of the DevCreek Eclipse plug-in and 81 updates to the devcreek.com website. These weren’t just developer integrations or even nightly automated builds to test platforms that the customer could theoretical browse, each of these were released to the public. Often the code written in the afternoon, such as adding a new feature to the website, would be in use by our members and available to the general public that same evening. This approach has been uncommon so far at Intelliware, and I wouldn’t be surprised if 91 releases tops the accumulated releases for all the company’s other projects during the year.

What is valuable about releasing so frequently? When does agile become merely rabid? One of the pressures of agile is to reduce the delay between a customer recognizing a need and supplying an implementation to the users of the system. If 2 months is better than a year, why not aim for the shortest time possible? Of course the delay in delivering functionality involves far more than the period between implementation and deployment, but one nice effect of reducing that period is the potential to reply immediately to feedback from users. Knowing code written that afternoon will be in use that evening also forces clarity regarding the essence of a feature and the minimal useful expression of it, and helps developers forge a sense of responsibility for their work.

But you are only working on a poxy little website? Fair enough, the project is one of the smaller ones at the company with only 2,000 tests to its credit. It does face a number of technical challenges though, for example all these releases took place while preserving or extending upon a published open web API used by various versions of the Eclipse and other clients, or that we have a not-insignificant 100 million records of data in an extensible format feeding into a score of graphical reports.

In practice the more significant limits on the frequency of releases for projects is related to the lack of control over production deployments and the rate at which the business can generate requirements. We already tend to push on those, but for new projects or more agile customer organizations we should at least entertain the idea of increasing the rate of releases.

But what if I mess up, that doesn’t leave much time for ‘user testing’ to find it? Yes not much time at all and you have to accept that errors will creep through. Just like normal. At least you can fix them in the following day’s release.

Our process is not too distant from other agile projects that we’ve seen; we just push for the line each time. The build process is actually weaker than on the last few projects I’ve been involved with, involving a few steps rather than the all-in-one make-and-deploy-a-new-release command. We do have a test platform that is a close duplicate of the production machine, with one wrinkle: we keep it up to date with the data from production. This gives us more confidence regarding data migration and strange data cases. If our 15 minutes of manual smoke testing on the test platform works out fine then we feel confident of pushing it to production and walking away. We use the same packaged code on the production that was exercised on the test machine. The only tweak on the production machine is that we keep the old releases on the machine allowing us to roll-back the last release quickly and with little risk in case of problems rather that fussing around rebuilding the earlier release from version control then pushing that through the process.

One further change was triggered by the one time that we were over taken by fear and rolled back a release. An issue with releasing at the end of the day is that you are often tired and if things go wrong you do not want to spend the night fixing the problem. Remember that in contrast to the once-a-quarter release procedure you have not specially prepared for this release, it is simply another day at work. The night has not been kept clear of events just in case things fall apart, and you have not fortified yourself at lunch with the energy needed to work through the night or weekend. In our case as the new release started up on the production machine it seemed that some of the user activity numbers were unexpectedly jumping up every minute or so. It wasn’t clear if that was an issue or not, so we just pulled the plug on the release, flipped back to the previous version of the software on the machine and went home. To provide a simple measure of whether the new release starts up fine we now take a screen shot of the recent activity on the system before the upgrade. Once the new release has started we compare the numbers with the intent that if anything looks fishy we can flip back to the previous release and investigate and fix the problem with the new release on the morrow when we are fresh.

What about the mismatch in story size with daily releases? How can you release a story before all of its tasks are completed? Decomposing week long stories into half-day or day long tasks for implementation is often source of debate in the early days of a team. For us, smaller teams often seem to shrink the size of stories such that they are closer to a day or two in size rather than a week or two, so the distance to bridge wasn’t so great for us. Taking a suitably critical judgment of the task line-up for a story, one can commonly see how tasks can build upon each other, how the first task might only handle one of many cases, or starts with a basic UI that is built upon over the next few releases.

Confusing users with partly built features is a concern, though we haven’t experienced any significant fall out from it so far. We don’t use “under construction” warnings to draw attention to work in progress. Of course everything should we work, even if the early releases cover only a subset of cases. On a related note, we only post a list of new features as part of a weekly notice rather than following each release to ease the burden on our users.

This posting only touches on some of the issues, for example what is the impact on the various rhythms that XP and related agile development process are built upon, or regarding concerns related to the activities of larger teams. There are many reasons to disregard daily releases for your current project, but our hope is that come the next time around merely recognizing that daily releases are possible might make you more confident pushing on the boundaries of your process a little more than you are use to.

PS – As final bait for the sceptic, the 82nd version of the website was released while writing this blog entry 😉

It's only fair to share...
Share on FacebookGoogle+Tweet about this on TwitterShare on LinkedIn

Leave a Reply