Calling DevOps teams an antipattern is an antipattern

Calling DevOps teams an anti-pattern IS an anti-pattern because needing no DevOps a team is the destination but not necessarily the journey.

In Star Wars terms, achieving the Continuous Delivery Capability without a supporting DevOps team is Jedi. Making tangible improvements with the support of a dedicated team could be the rebel alliance, but is definitely not the dark side!

Enterprises should feel an urgency to try the Continuous Delivery practice and must feel empowered to experiment on how to best to adopt it in their own organisation.  All organisations whilst united by the need to adopt the Continuous Delivery practice, have a variety of different cultures, skills, history, priorities, accountability, politics etc.  Therefore it is logical to expect a variety of approaches to be effective towards achieving Continuous Delivery.

I don’t disagree with the some of the arguments against teams, but I just feel that DevOps teams should be understood as having traps to avoid, rather than being a trap to avoid.  In the wrong hands anything powerful usually has the ability to do at least as much bad as good.  Heed the cautions and make sure your’s aren’t the wrong hands!

What DevOps strives to achieve

To summarise, DevOps aspires to achieve highly responsive and reliable IT functions that are globally optimised to support and deliver the Business objectives.  In other words: “Deliver what IT is needed, when it’s needed and consistently until the need disappears or changes.”  In more other words: “Restore faith in IT by going the difficult last mile beyond agile development to achieving agile releasing of software.”

What DevOps typically entails in practice

The prominent pattern for doing DevOps is to adopt the Continuous Delivery practice.  DevOps and Continuous Delivery are heavily underpinned by effective solutions for the following core processes:

  • Version Control
  • Configuration and Release Management
  • Environment Management (development, test, and production)
  • Automation and orchestration throughout the software delivery life cycle of at least:
    • code release
    • quality assurance
    • infrastructure.

Take all of the cross-team collaboration / peace and love you like, if the above let you down, you’ll not achieve continuous delivery and worse you’ll have problems.

What traditional Development teams typically achieve

We’ve all heard the statement: “Developers want Change and operators want Stability.”  Surely based on this, at least the developers are ready to embrace the changes required to implement Continuous Delivery?

Unfortunately it’s not this type of change!  It is very common for developers to feel pressured to focus almost exclusively on delivering functionality, even at the expense of neglecting all non-functional “ilities” .  I’ve seen agile actually exacerbate this. When Product Owners (usually aka functionality champions) are permanently based in the development team, the focus on working features, features, features can go too far.

Unfortunately regular demos, valuable as they are, can sometimes also negatively reinforce this functionality addiction.  Watching Chef re-build an environment from version control appeals to a fairly select group of people (of which I am included).  But we are significantly outnumbered by people who prefer to watch the shadows move on fancy new sliders in a UI (or other functional things -not my bag).  The typical demo audience can put development teams off tasks which don’t demo well, for example automating server creation.

I’m not criticising agile or arguing that the above problems aren’t solvable.  These are just example causes of functionality changes dominating at the expense of developing the requisites for Continuous Delivery.  In fact tasks related to Continuous Delivery are often left on the backlog directly below a block of non-functional requirements (if anyone even took the time to determine the relative priorities that far down the list).

What traditional Operations teams typically achieve

A common occurrence is that Operations teams dislike change.  It is absolutely standard for Operations to feel accountable for maximising two things:

  • Stability
  • Efficiency

Stability is threatened by all change.  Not just changes to the code and configuration of supported systems (as requested by Development), changes to people, processes and tools.

One compounding unpleasant phenomena I’ve seen once or twice is a tendency to scape-goat old bad processes that were invented by much blamed predecessors in favour of taking risks on improving, thus avoiding ownership and accountability.  This may lead to prevailing problems like heavily manual deployment processes or worse processes being badged unfit-for-purpose and completely ignored (as we saw with the Change Process in the Phoenix project novel!).

Efficiency is often achieved by cutting back on anything outside core areas of accountability. Continuous Delivery is becoming easier to justify, but making the strategic benefits of automated deployments sound more appealing that the tactical benefits of saving money by hiring fewer people will never get easy.

Efficiency-over-emphasis for Operations parallels the functionality-over-emphasis for Development and the result is to grease the tarmac on which Continuous Delivery needs traction.

Why would you want a DevOps team?

The examples above highlight why neither Development or Operations are never going to have an easy ride implementing Continuous Delivery.  We can also understand from these why both Development and Operations teams so often get to consider themselves locally successful without feeling the need to address the end-to-end cross-team effectiveness and the promise of DevOps.

A major contributing factor for a general lack of progress towards Continuous Delivery is the lack of clarity over ownership.  The core concerns listed above don’t usually fit naturally into either team and as a result get left festering (and failing) in the gap between.  I’ve read enough about DevOps to know that to be taken seriously, you generally need a Deming quote.  I found this appropriate:

“Hold everybody accountable? Ridiculous!”, W. Edwards Deming 

Finding a person accountable for each of the awkward core processes is necessary.  Assuming that this person needs dedicated (or shared) resources to succeed, you may have just got yourself a team which could be called DevOps.

What would a DevOps team do?

If we’re ready to make someone accountable for addressing the problems, the two main things that their team could do are:

  1. Provide an appropriate set of services to Development and Operations (on-going).  See here for a full discussion.
  2. Run a change programme to expedite adoption of Continuous Delivery (temporary).  See here for a full discussion.

A quick critique of “how to do DevOps without a team” advice that I’ve read

Give developers more visibility of production.
Great idea, but this isn’t going to build an automated tool chain.

Encourage developers and operations staff to have lunch together 
Not bad, will probably locally improve the productivity of the developers who now know buddies in Operations ready to cut them special treatment.  This type of thing is going to take a long time to make a difference in a big enterprise.

Standardise the metrics that influence performance based-pay across development and Operations. 
Like this one. But extremely difficult in a major organisation and certainly not without adverse risks of unexpected side-affects.

More than happy to discuss all other ideas anyone cares to suggest!

Conclusion

If you accept that the above activities are not an anti-pattern, perhaps the only remaining argument is what to call the responsible team.  If you don’t accept them, as one last go, please drill down to the detailed discussions on service and programme teams.

Personally I think that so long as the people involved understand what DevOps actually is, the name DevOps gives them a daily reminder not to lose sight of their real objective.  Being in a release management team that slows down delivery may carry no stigma.  There is nothing more embarrassing that being in a DevOps team gets associated with impeding DevOps!

Advertisements

Having a DevOps Change Programme Team is a Pattern

As I described here, calling DevOps teams an anti-pattern IS an anti-pattern because needing no DevOps a team is the destination but not necessarily the journey.

The problem was I got very over-enthusiastic and the blog became very long.  Hence I’ve broken out describing why a DevOps Service team is a pattern here.

I described two valid types of team

  1. One that provides an appropriate set of services to Development and Operations (on-going).  Described here.
  2. One that runs a change programme to expedite adoption of Continuous Delivery (temporary).  Described right here in this blog!

DevOps Change Programme Team

For some organisations, depending on how far they have to go, some steps towards Continuous Delivery will just be too hard to deliver from one of the core teams or a service.

Creating a DevOps programme/project team may not be for everyone, but is a valid endeavour.  Such a programme would take end-to-end ownership of increasing Continuous Delivery maturity and achieving a globally optimised IT function.  Programmes may bring a number of advantages.  By requiring investment they automatically incur a level of expectations for senior folk which can help establish credibility i.e. that this is really happening.

The success of the programme should be measured by three outcomes:

  1. the programme-assessed improvements against the original baseline,
  2. the recognition of those improvements by Development and Operations, and
  3. the general view of the outcome from Development and Operations.

Projects making up the programmes could include analysis of current processes, taking a baseline of current performance and performing R&D on tools and processes.

R&D is interesting because it can achieve a lot without even looking outside an organisation.  Most organisations will have systems and processes of varying maturity which provide ready-made case studies of what works and what does not.  They will also have people from a variety of backgrounds and experiences combined with context specific insight into how and what to improve.

Projects can be used to set up the acceptable services listed above.  I’d even go a little further to say my test 1 about not abstracting people from their core responsibilities can be temporarily relaxed (pending later correction).  For example if introducing a major new tool like Puppet or Chef the level of effort may warrant temporary extra assistance from a dedicated experienced team to write the first cut of configuration files.  Obviously this must be done with caution and mindfulness of the need to transition back responsibilities to the rightful owners.

Programmes also have a natural advantage of being painfully aware that they have to convince both Development and Operations that have their best interests at heart.  Staying on the subject of introducing automated Configuration Management tools, when these are driven from Development, they face an almighty challenge getting taken seriously and anywhere near the production environment.  When they are driven from Operations, the sell to developers that they now need to taken previously Operations concerns is not an easy sell.  When they are driven by an independent team, these concerns need to be addressed openly up front.

Conclusion

A well executed DevOps Change Programme can be very effective and having a DevOps Change Programme Team is a Pattern!

Having a DevOps Service Team is a Pattern

As I described here, calling DevOps teams an anti-pattern IS an anti-pattern because needing no DevOps a team is the destination but not necessarily the journey.

The problem was I got very over-enthusiastic and the blog became very long.  Hence I’ve broken out describing why a DevOps Service team is a pattern here.

I described two valid types of team

  1. One that provides an appropriate set of services to Development and Operations (on-going).  Described right here in this blog!
  2. One that runs a change programme to expedite adoption of Continuous Delivery (temporary).  Described here.

DevOps Service Team

Determining what services are appropriate at the heart of the controversy around DevOps teams.  Another way of putting it is “when is an acceptable service not also an evil functional silo?”  For me, any service that a DevOps team might provide must pass the following tests:

  1. Test: Does it abstract Development and/or Operations from the consequences of their actions and their core responsibilities?
  2. Test: Does it complicate communications and the interface between Development and Operations?

If a DevOps Service team run the Continuous Integration / Continuous Delivery orchestration tooling.  This does not move the developers away from writing or testing code and it doesn’t move the operators away from their core concerns, so passes test 1.  These tools are also great for improving communication because they make mundane but critical information such as ‘what code has been deployed where’ transparent, thus freeing up communication channels for more complex matters, thus passing test 2. CloudBees have demonstrated people want this by offering Jenkins as a SaaS.  In the organisation that I work, we also provide a similar (but more extensive) service.

What about hosting Version Control tools?  If your organisation has problems with version control, it will not fail either of my above tests by making the tools the problem of someone from a DevOps team.  GitHub make the success of this model pretty obvious and may also be an appropriate solution for you.  However, alternatively (for various reasons e.g. possibly cheaper in the long-term, consistent with company Infosec requirements etc) an internal DevOps team could be your solution.

What about writing Automated Tests?  Ok, ok, just checking you are concentrating.  This immediately fails my first test.  If Development doesn’t write the tests, they are abstracted from whether the code they are writing even works!  DevOps teams shouldn’t write automated tests!

What about setting up Automated Build Scripts?  This fails my first test.  If a developer doesn’t even know how their code is going to be built for release outside their local workstation, they have been abstracted too far away from writing working code.  However, we live in an imperfect world and there are times when in reality you have to balance test 1 and 2 against test 3:

  1. Test: Do the Development or Operations teams have the capacity and skills to deliver these?

Thankfully generally developers do understand how to create build scripts.  But there are certainly cases, for example with many packaged products, that the developer process for building and deploying code is not easy to script are decouple from being a manual trigger inside the IDE.  Almost invariably this leads to the unceremonious end to a Continuous Delivery journey before even achieving Continuous Integration.  Use of a DevOps service with specialist skills could be appropriate for owning and solving this problem of extracting the requisite commadline build process.   I also want to draw a distinction between setting up scripts and maintaining them.  Once created, Developers need to own build scripts and be empowered to own them without having an inefficient and/or asynchronous external dependency.

As a side point specialist skills, these are much can be much easier to develop-within or attract-to a dedicated DevOps service team.  Trying to persuade some developers that they are interested in scripts or even any code that won’t feature prominently in production isn’t easy.   Trying to persuade operators that they need to write more scripts is usually an easier sell, but giving them a dedicated focus can help a lot.  If you do hire specialists, you also need to be able to protect them.  I’ve seen a number of examples of specialists hired to work on DevOps processes getting quickly irreversibly dragged into normal Development or Operations work (i.e. thus helping with what those teams typically achieve – see sections above!).

DevOps Service Team: Development and Test environments

What about creating and supporting Development and Test environments?  By this I mean ownership of non-production environments where code is almost continually refreshed and tested and may be under constant use by 10’s if not *gulp* 100’s of people.  If environments are experiencing unplanned outages or even just experiencing slow code deployments, this costs the organisation a lot of wasted productivity (not to mention morale).  Since this type of activity is usually on the critical path for releasing software, lost non-production environment time will usually also mean delayed releases.

To an extent creating and supporting non-production environments with a DevOps team fails test 1.  The further an Operations team is from the environments, the later they will find out about new changes that will impact production.  It also fails test 2 as the classic “it works on my development machine” response to things not working in production now has an evil need breed of mutant cousins “it works in the System Test environment”, “it works in the Integration Test environment” etc.

But despite this apparent “new” problem created above, in my experience it usually already exists because I rarely see Operations successfully owning all of the lower environments.  Instead they will be owned by Development (or specifically “Testing”).  This not only distances Operations from the environments but usually breeds frustration within Development. Wherever they have “control”, things appear to “work” (or at least get fixed according to their priorities), but wherever in the life cycle Operations take over is carnage. The later it is, the worse the effect. In fact, you get a problem with a lot of parallels to Martin Fowler’s infamous software integration phase.

Even if creating and supporting Development and Test environments is owned by Operations often this doesn’t always go well because test 3 fails.  Operations then reduce the burden of supporting non-production environments (aka “optimising” efficiency), by handing over full access to the environments to Development.  This quickly breeds snowflakes and the Operations are once again abstracted from the details of non-production environments and trouble is stored up until the first environment that they keep locked down.

Another challenge with Operations managing non-Production environments is that they have achieve the perception of overall service “stability” by playing a numbers game i.e. prioritising based effort based on perceived level of impact.  This equation rarely favours anything non-production and Operations teams always have a solid gold excuse for neglecting development and test environments and tools: “the Prod Card”.  Playing the Prod Card grants impunity from blame despite almost any amount of to non-Production.  The only way to protect non-Production is by identifying people who are dedicated and this could be a shared DevOps service.

Yet another challenge with Operations managing non-Production environments could be that they don’t yet exist.  It’s not unusual on projects that the operator has either not yet been appointed or if internal not yet been commissioned to work on a project.  I’m not arguing that this is desirable, but in those cases having a separate service to manage things can be more effective than having nothing at all.

All in all, non-production environment management is often a big problem which can be solved by a dedicated team provided they acknowledge their existence in relation to the 3 tests and manage their shortcomings accordingly. Also, providing they are aware of the impending ‘integration phase’ when transitioning to Operations.

Conclusion

A well designed DevOps Service Team is a Pattern!

Continuous Delivery as a Practice for Everyone

In my blog about DevOps, I couldn’t help but mention the associated “buzzwords” Continuous Delivery.

When a powerful idea like Continuous Delivery graduates into a buzzword, there are downsides.  A buzzword’s popularity will often make it prone to misuse for example people indiscriminately tagging arguments/ideas with the word to create credibility irrespective of whether it is actually appropriate.  Often the original buzzword meaning is poorly understood or gets overloaded enough to cut casual abuser enough ambiguity to proliferate the problem (and maybe lead people astray / advance their evil conquest).

Obviously damage is done when bad arguments/ideas gain traction on account of the strength of the misapplied buzzword.  There can also be irony when a bad argument/idea is actually in contradiction to the original meaning.  But the worst part is the dilution / loss of the original great idea.

A couple of recent bad cases of buzzword abuse I’ve heard against Continuous Delivery have motivated me to write this in defence.

Continuous Delivery is a real practice applicable to anyone who wants their software releases to be one of more of:

  • Regular
  • Predictable / low risk
  • Efficient

I personally think it’s relevant to EVERYONE.

Continuous Delivery as a capability in an organisation means that any new change to code or configuration is capable of being fully tested to the extent that it can promoted into live service with just one single manual step: the act of clicking “do the production deployment”.

This means after code or configuration is committed to version control the whole remainder of the software delivery life-cycle… compilation, packaging, unit testing, functional testing, performance testing, security testing, user acceptance testing, operational acceptance testing etc, and all dependent code, configuration and data deployments required to non-production environments …must be automated!

Obviously tools and automation are only one part of the complex full equation and the devastating effect of people on actual outcomes can be dramatic.  Martin Fowler provides an elegant definition of whether you are actually achieving the desired intentions here.

Naturally there is one step beyond this where the production deployment is also automated.  Generally this is called Continuous Deployment although I prefer the more explicit “Continuous Deployment to Production”.  The act of promoting code to non-Production environments is usually also called a deployment and continuously deploying to lower test environments is essential to achieving Continuous Delivery.

Continuous Delivery as a capability is an extremely powerful goal, but sadly so far off the way many organisations currently operate that they are put off altogether.  This is even more likely if the distinction of Continuous Delivery versus Continuous Deployment to Production is not understood.

Don’t be scared by the Continuous Delivery capability, instead do the practice!

Continuous Delivery as a practice is the act of pro-actively evolving your software development and release processes towards the ultimate target of achieving the Continuous Delivery capability.  This doesn’t just mean creating isolated bits of automation (some of which you hopefully already have), it means orchestrating all your automated tasks into a workflow capable of getting as close to implementing “the capability” as the your automated tasks will permit.  This delivers value.

Think about it like a new guitar player trying to learn Jimi Hendrix.  They know realistically it is going to take a long time to nail Voodoo Chile (full version not the Slight Return 😉 ) with authentic dexterity and feeling, but it doesn’t stop them tangibly improving their playing whilst they try (and having a lot fun).  Likewise, every step towards Continuous Delivery is valuable to the organisation as a whole (and to the people involved).

A very early step towards achieving a Continuous Delivery capability, is to properly sort out version control of your code and configuration.  If these are not adequately controlled, there is no real hope of a change even triggering a code build, let alone the following chain of automated quality assurance processes.  Even if you just do this, you will no doubt have improved productivity and predictability in your organisation.

Another early staple is to have commits to the version control repository automatically trigger code builds, static code analysis and unit testing,  This practice pre-dates Continuous Delivery and is usually called Continuous Integration.  Again, this is extremely valuable in its own right.

This sequence of automated processes and quality gates was to my knowledge first described as a Deployment Production Line in this paper which argues ‘why stop at Continuous Integration’ and ‘why stop at application and configuration when you can consider infrastructure as well’.   Both great arguments.  It then went on to be described as Deployment Pipeline in this excellent book (by Humble and Farley).

As you stick to the Continuous Delivery practice and the Deployment Pipeline matures you will expect to see greater throughput of change, granted not yet all the way to production, but still extremely valuable.  Building your Deployment Pipeline is a key element of the practice and something I hope to write about soon.

In summary, don’t be blinded by the ambitious end capability.  If you aren’t already doing it, start thinking about adopting the Continuous Delivery practice TODAY!