Reducing Continuous Delivery Impedance – Part 3: COTS Software Packages

This is my third post in a series I’m writing about impedance to doing continuous delivery.  Click these links for earlier parts about the trials of Infrastructure and Solution Complexity.

Software packages, specifically Commercial-Off-The-Shelf Software (COTS) are software solutions purchased from a 3rd party vendor.  The idea is that you are purchasing something already delivering your needs for less money than it would cost you to build it.  There are also lots of other stated benefits.  I’m not certainly not challenging the use of COTS products altogether, or claiming that they are all the same.  However, many of those I’ve worked with over the years have created particularly strong impedance to doing continuous delivery.

In my experience there is nothing like custom code applications when it comes to great compatibility for doing continuous delivery.  For example Java:

  • Starting with early stages of the pipeline you’ve got Maven (or Ant or Grade), Findbugs, Checkstyle, Sonar, JUnit, Mokito, etc. all excellent. (Equivalents are readily available for C#, JavaScript, HTML, CSS, Ruby, Python, Perl, PHP, you name it…)
  • Moving on to deployment, you can literally take your pick: building native OS packages (e.g. RPMs), self-sufficient Java applications (e.g. Spring Boot), Configuration Management like Chef or Puppet, Jenkins and Shell scripts, I could go on…

This isn’t a new thing either, C and even Fortran are great and even Main Frames coupled with a proficient use of something like Endevor are very build-able, releasable and deploy-able.

Sadly with COTS products things are often not so simple…


Let’s start at the beginning of the pipeline.  COTS products can be hard to version control.  Many COTS products seem to consider a development repository part of their package.  – How generous of them to take version control / SCM off our hands?  Not really, how inconvenient, they rarely do they do a good job.  The staples of a good SCM are often absent:

  • Offline / distributed working – many COTS products require access to a shared development database.  This can be bad for productivity (slow), it may even be bad for flexible working (if people can’t get remote access)
  • Revision tracking – many COTS products may store limited metadata e.g. who changed what, but may limit your ability to do things like make an association to an ALM ticket e.g. in Jira
  • Branching  – usually not even a concept leaving you to fork irreconcilably (admittedly arguably branching isn’t really in the spirit of continuous delivery…)
  • Baselines – very rarely a concept allowing no reasonable way of promoting something other than the latest state of the development repository.

Usually to achieve the above sound SCM, you need to work hard to export your configurable items into flat files so that you can then store them in a sound SCM system (e.g. Git).  Generally this will take some kind of export / check-in tool that you may have to write yourself and thus take your own responsibility for ensuring integrity between SCM development repositories.

So let’s say we’ve got our COTS products moving parts flattened into versions control and under our control.  How do we handle merge?  The answer is usually simple: you can’t – any modification of these fragile, sensitive XML or in practical terms binary files outside of the IDE is strictly prohibited by the COTS vendor and even if tried is unlikely to lead to anything that will work (even if you get into clever schema informed XML merging).

Even if you aren’t branching, it may be perfectly reasonable for two people to end up simultaneously updating the same files.  This creates a problem that you need to try to avoid multiple people simultaneous changing the same file.  We must turn to our SCM system or perhaps a hybrid of the COTS IDE and the SCM system to implement pessimistic locking which even if achieved successfully has a negative impact on productivity.

So we’ve got a process of developers pushing consistent changes into version control.  The next challenge may be how to build.  Whilst the COTS product may export flat files, often this is just so that you can import the files into another development repository and you may not be able to build without re-importing your flat files back into another repository.  Worst case of all is when you have to import manually and then trigger a build manually within the IDE.  I’m not saying it isn’t possible to automate these things, I have definitely done it, but when it comes to having to involve AutoIT (screen scraping), you’ve got to feel somewhat frustrated/humiliated/sick/sad.

It’s not uncommon for COTS products to be slow to build.  I’m not saying the problem is exclusive to them, but with a custom application is usually much easier to break the build-up into smaller components to build separately.

Once we have a build, the task of deploying may not be easy.  Many COTS products expect fully manual, mouse-driven deployments and you basically end up reverse engineering to automate deployments.  Deployments can also be slow complex, slow and unpredictable.

Finally environments for COTS products can be tricky.  Usually a custom application will rely on standard middle-ware, or better still a PaaS.  COTS products can involve following hundreds if not thousands of pages of manual steps in installation manuals leading to a huge amount of work with your configuration management tool like Puppet or Chef.  Worse still there may be manual steps e.g. a configuration wizard that are very difficult to automate and again you are left to reverse engineering to remove the manual steps.

Like I said earlier, I’m not rejecting COTS products altogether, but I’d like to be very clear that they can often in many different ways create impedance for doing continuous delivery.  It is very rare that given enough effort these limitations cannot be overcome, but it may actually be equally rare (sadly) that organisations invest the necessary effort.