Reducing Continuous Delivery Impedance – Part 2: Solution Complexity

This is my second post in a series I’m writing about impedance to doing continuous delivery and how to overcome it.  Part 1 about Infrastructure challenges can be found here.  I also promised to write about complexity in continuous delivery in this earlier post about delivery pipelines.

I’m defining “a solution” as the software application or multiple applications under current development than need to work together (and hence be tested in an integrated manner) before release to production.

In my experience, continuous delivery pipelines work extremely well when you have a simple solution with the following convenient characteristics:

  1. All code and configuration is stored in one version control repository (e.g. Git)
  2. The full solution can be deployed all the way to production without needing to test it in conjunction with other applications / components under development
  3. You are using a 3rd party PaaS (treated as a black box, like HerokuGoogle App Engine, or AWS Elastic BeanStalk)
  4. The build is quick to run i.e. less than 5 minutes
  5. The automated tests are quick to run, i.e. minutes
  6. The automated test coverage is sufficient that the risks associated of releasing software can be understood to be lower in value than the benefits of releasing.

The first 3 characteristics are what I am calling “Solution Complexity” and what I want to discuss this post.

Here is a nice simple depiction of an application ticking all the above boxes.


Developers can make changes in one place, know that their change will be fully tested and know that when deployed into the production platform, their application should behave exactly as expected.  (I’ve squashed the continuous delivery (CD) pipeline into just one box, but inside it I’d expect to see a succession of code deployments, and automated quality gates like this.)


But what about when our solution is more complex?

What about if we fail to meet the first characteristic and our code is in multiple places and possibly not all in version control?  This definitely a common problem I’ve seen, in particular for configuration and data loading scripts.  However, this isn’t particularly difficult to solve from a technical perspective (more on the people-side in a future post!).  Get everything managed by a version control tool like Git.

Depending on the SCM tool you use, it may not be appropriate to feel obliged to use one repository.  If you do use multiple, most continuous integration tools (e.g. Jenkins) can be set up in such a way as to support handling builds that consume from multiple repositories.  If you are using Git, you can even handle this complexity within your version control repository e.g. by using sub-modules.


What about if your solution includes multiple applications like the following?


Suddenly our beautiful pipeline metaphor is broken and we have a network of pipelines that need to converge (analogous to fan in in electronics).  This is far from a rarity and I would say it is overwhelmingly the norm.  This certainly makes things more difficult and we now have to carefully consider how our plumbing is going to work.  We need to build what I call an “integrated pipeline”.

Designing an integrated pipeline is all about determining the “points of integration” aka POI i.e. the first time that testing involves the combination two or more components.  At this point, you need to record the versions of each component so that they are kept consistent for the rest of the pipeline.  If you fail to do this, earlier quality gates in the pipeline are invalidated.

In the below example, Applications A and B have their own CD pipelines where they will be deployed to independent test environments and face a succession of independent quality gates.  Whenever a version of Application A or B gets to the end of its respective pipeline, instead of going into production, it moves into the Integrated Pipeline and creates a new integrated or composite build number.  After this “POI” the applications progress towards production in the same pipeline and can only move in sync.  In the diagram, version A4 of Application A and version B7 of B have made it into integration build I8.  If integration build I8 makes it through the pipeline it will be worthy to progress to production.

intDepending on the tool you use for orchestration, there are different solutions for achieving the above.  Fundamentally it doesn’t have to be particularly complicated.  You are simply aggregating version numbers in which can easily be stored together in a text document in any format you like (YAMLPOMJSON etc).

Some people reading this may by now be boiling up inside ready to scream “MICRO SERVICES” at their screens.  Micro services are by design independently deploy-able services.  The independence is achieved by ensuring that they fulfill and expect to consume strict contract APIs so that integration with other services can be managed and components can be upgraded independently.  A convention like SemVer can be adopted to manage change to contract compatibility.  I’ve for a while had this tagged in my head as the eBay way or Amazon way of doing this but micro services are now gaining a lot of attention.  If you are implementing micro services and achieving this independence between pipelines, that’s great.  Personally on the one micro services solution I’ve worked on so far, we still opted for an integrated pipeline that operated on an integrated build and produce predictable upgrades to production (we are looking to relax that at some point in the future).

Depending on how you are implementing your automated deployment, you may have deployment automation scripts that live separately to your application code.  Obviously we want to use consistent version of these through out deployments to different environments in the pipeline.  Therefore I strongly advise managing these scripts as a component in the same manner.

What about if you are not using a PaaS?  In my experience, this represents the vast majority of solutions I’ve worked on.  If you are not deploying into a fully managed container, you have to care about the version of the environment that you are deploying into.  The great thing about treating infrastructure as code (assuming you overcome that associated impedance) is that you can treat it like an application, give it a pipeline and feed it into the integrated pipeline (probably at a POI very early).  Effectively you are creating your own platform and performing continuous delivery on that.  Obviously the further your production environment is from being a version-able component like this, the great the manual effort to keep environments in sync.



Coming soon: more sources of impedance to doing continuous delivery: Software packages, Organisation size, Organisation structure, etc.


(Thanks to Tom Kuhlmann for the graphic symbols.)


Reducing Continuous Delivery Impedance – Part 1: Infrastructure

As I’ve stated previously, I believe continuous delivery should be viewed as a practice that all IT teams should follow.  As I’ve stressed, doing continuous delivery shouldn’t be viewed as a binary capability that you can or cannot do, it should be viewed as the focus of a set of principles to guide practices.

In this series of posts, I’m going to summarise the main challenges to achieving high performing continuous delivery that I’ve been sorry to see but also motivated to try to overcome first-hand over roughly the last 10 years.

I borrowed the electronic engineering term “impedance” for two reasons.  Firstly it basically means resistance which can be defined as:

“refusal to accept to comply with something”, “the use of force to oppose something”, and “the stopping effect exerted by one thing on another”.

All apt ways of characterising the things that get in the way of continuous delivery.  Secondly (in electronics) impedance it is actually resistance experienced to alternating current (AC, i.e. what comes out of mains sockets) which seems appropriate given continuous delivery is cyclical.  Also given the current mania for all of a sudden calling everything digital, analogue things appeal to me.

So how can infrastructure impede doing continuous delivery?

A foundation of Continuous Delivery is the ability to automate, not just application code deployments, but as much as you can, i.e. as far down the stack as possible.  It is unfortunately common in my experience that infrastructure solutions are just not conducive to this.

I believe that the public cloud presents the best experience possible here and is extremely conducive to automation.  For example in AWSCloudFormation allows you to provision your whole (virtual) data centres from scratch.  Heat does the same in Open Stack clouds.  When using the CloudFoundry PaaS, Bosh can be used with AWS and also VMWare underpinned data centres to do the same thing.  Many solutions are becoming readily available.

These are incredibly empowering as they allow you to create environments from version control and scale up and down the number and types of environment you have dynamically during the software release lifecycle (pipeline).  They allow the infrastructure to be treated as code and subjected to software engineering and quality processes.  This puts this in the hands of anyone capable of writing code (which of course can be tempered by governance over the adoption of changes – for example via pull requests in the GitHub).  Fundamentally this significantly helps break down the barrier between Development and Operations teams.

Unfortunately, the fantastic world of public cloud is not available to many, who instead have to accept something on a spectrum from a less sophisticated cloud which whilst offering some self service features e.g. the ability to create new virtual machines but not things like network changes, all the way do to the dreaded manual request for a manually created physical server to be selected, purchased ordered, purchased, delivered, racked, stacked etc. (which can run into months of lead time).  Private cloud is an extremely overloaded term.  I’m open to accepting that some organisations could deliver it at the same level of sophistication and service that public provides, but from my experience so far, it often a long way off and this bleak description is unfortunately familiar and can even be mis-named some type of internal cloud service.

Continuous delivery can be exponentially harder to do the less your infrastructure solution resembles public cloud.  Getting the servers you need can be the first problem with long lead times.  Sometimes I’ve even seen the need to make business cases to convince someone (who seems to have the job of minimizing the number of servers used) that your reasons for asking for servers is genuine.  Obviously this wastes a huge amount of time and effort.

Even once servers have been created, it is not uncommon to struggle to gain access to them. With a public cloud, you are fully in control and can adopt a virtual private networking solution native to the IaaS provider, or implement your own.  I’m not saying this doesn’t need to be done responsibly and with due diligence, but it significantly simplifies things both technically (Software Driven Networking is just a given in the cloud) and also from a people perspective (more on than in a future post).

Once you’ve gained access, you may not be empowered with super-user access and hence unable to implement automation.  You are forced to treat them as “pets” as “opposed to cows”, i.e. it is difficult to destroy and replace them and far more tempting to manually tweak them, creating snowflakes.  Without IaaS, it isn’t uncommon to have to tolerate glaring differences between non-production and production servers, for example not even the same operating system.  It isn’t easy to build and trust a continuous delivery pipeline with that level of inconsistency.

It’s fair to say that automated configuration management tools like Puppet and Chef can make big steps towards improving consistency between even physical servers (although its not unheard of for infrastructure/security etc. teams to actually outlaw these).  But without the mindset that servers are fully disposable, it is all too easy to log on to servers and for integrity to slip (creating a smell).

As I’ve said here there are some opportunities to do some pretty smart things locally on your development workstation.  But even when doing that, if those processes are unable to have any resemblance to what get used downstream, their benefits are limited.

With the above in mind, I highly recommend starting small by allowing development and test to use public cloud and understand first-hand how powerful it can be.  Demonstrate the benefits achieved there as early as possible and start the cultural acceptance of cloud.

Coming soon:  more sources of impedance to doing continuous delivery: Solution Complexity, Software packages, Organisation size, Organisation structure, etc.

Update: here is a video of me talking about this and other sources of impedance: