A couple of years ago I mentioned the approach of daily deployments to the production DevCreek.com server, in which we racked up ninety or so releases in a year without introducing any problems to the running service. This was enabled through developer created stories and full control over the production environment, while being supported with only a primitive build and deployment mechanism.
A far more impressive case of pushing the rate of production deployments is described in this blog article by Timothy Fitz:
As I understand it, their continuous build system is triggered approximately 70 times a day and runs through 15,000 tests, across 40 test machines. If successful, then typically the software will be pushed to their production machines. This is applied to a few of the machines in their production cluster first and only if their behaviour matches the operational norms over a few minutes of work will it then be released to the remaining machines in their production environment. In case of problem the release is automatically rolled back. All together this can result in fifty deployments to production every day.
This seems a splendid example of what is possible when you keep on pushing and pushing with build and deployment automation. It’s is also another case of being able to make use of sampling and statistical approaches once code starts running across many machines in a heavily trafficked online environment.
This is the first time I have read any the author’s blog entries and found many further excellent posts on his site.
For example, the following stood out from: Continual Automation
When we spin up new engineers we target a working sandbox by lunch and a typo fix live in production by end of day. On the first day of their employment. This is one of our best ways to demonstrate our cultural differences from most development shops….We didn’t get here overnight. We didn’t get here in a few weeks. We didn’t get here by funding a project. We got here by a culture. That culture overridingly said, if you did it twice it’s time to start automating. Every time you repeat a task, make progress on automating it….
It is very easy to become disillusioned with the incremental tool development approach when faced by the continual pressure of being on the story and release treadmill, so I found it reassuring that it is still possible to accomplish something of great value in this environment. Admittedly this seems to be a product company rather bespoke development firm, still it gives hope for what is possible.
This kind of work highlights to me just how many opportunities are still open for us to grow our technical practices and improve our tools and automation. It seems we have barely started the journey.