Bringing order to a chaotic release schedule

A novel approach to managing releases.

I’m currently involved on one of three distinct projects affecting a single website and we’ve run into a bit of a bottleneck when it comes to our release schedule.

For a bit of background our setup is Team Foundation Server for source control with a per-project branching structure, we have a continuous integration and deployment set-up using Team City and Octopus Deploy. However, we are lacking the level of automated acceptance tests required to remove the need for extensive manual testing.

Short release cycles have meant we’ve been forced to frequently rollback a patchwork of changesets that had entered UAT whilst we were testing another feature ready for release, effectively changing the state of the code we are testing. Often we are forced to do rollbacks, pre-release to ensure that we aren’t about to release untested code. The problem is these rollbacks strictly speaking invalidate all prior testing, but that’s another matter.

In the long term, we are looking at changing our workflow to automate the release process and testing process as much as possible. However, this is not a problem that we can fix overnight. What’s more we still need to keep up with a fast-paced release schedule for the next few months whilst we improve this tooling.

We looked at a bunch of short term technology focused solutions, that were either flawed in our current situation or add more technical complexity of their own. Instead, after a short discussion, there was only one solution we all agreed would help fix this problem. Almost straight out of Lord of the Flies, we have the check-in chicken. Yes, you heard that right.

How it works

The holder of the chicken manages check-ins for their next release, let's call them the Chicken Master, all must bow before him, etc.

  1. No chicken, no check-in, only the chicken master may check-in freely.
  2. Once a chicken masters' check-ins are tested and approved for release, the chicken is passed to the next chicken master. At this point, any release documentation is created and that version released.
  3. The chicken master has the right to approve or veto someone else's check-in, adding it to their release. It's up to the chicken master and requestor to ensure these changes are safe to check-in in with the current changeset(s) (This is equivalent to pull request in Git terms).

There is one caveat, under certain circumstances, the chicken master must relinquish the chicken, in doing so they must:

  1. Revert all changes back to the previous release-ready/live version of the code base.
  2. Bestow the check-in chicken on the next chicken master.

The idea behind the caveat is that in the event of a release having to be delayed or in the case of emergency we can quickly roll out a hotfix for an issue. It's trivial to roll back the current releases changeset(s), the hotfix can be tested and released, and the previous chicken master can then revert their rollback.

We introduced the chicken to ensure that our team better communicate their intentions, hopefully, solving some of our recent issues. It sounds farcical, but it's our team's way of making light of a problem we are having and can't solve comprehensively. A fun and novel temporary solution, until we have the resource to remove the pain points of our release process. Communication is the only way of allowing our team to navigate through fast release cycles because it’s a clear indicator when and where it will fail.

***

This is post is part of my "30-day writing challenge".

Update: 8 April 2016

I shared this on the challenge's slack channel and @jamesdoc pointed out that GDS have a similar process, with the @badgerofdeploy. The chicken is jealous of the badger's status.