Using ADOP and Docker to Learn Ansible

As I have written here, the DevOps Platform (aka ADOP) is an integration of open source tools that is designed to provide the tooling capability required for Continuous Delivery.  Through the concept of cartridges (plugins) ADOP also makes it very easy to re-use automation.

In this blog I will describe an ADOP Cartridge that I created as an easy way to experiment with Ansible.  Of course there are many other ways of experimenting with Ansible such as using Vagrant.  I chose to create an ADOP cartridge because ADOP is so easy to provision and predictable.  If you have an ADOP instance running you will be able to experience Ansible doing various interesting things in under 15 minutes.

To try this for yourself:

  1. Spin up and ADOP instance
  2. Load the Ansible 101 Cartridge (instructions)
  3. Run the jobs one-by-one and in each case read the console output.
  4. Re-run the jobs with different input parameters.

To anyone only loosely familiar with ADOP, Docker and Ansible, I recognise that this blog could be hard to follow so here is a quick diagram of what is going on.

docker-ansible

The Jenkins Jobs in the Cartridge

The jobs do the following things:

As the name suggests, this job just demonstrates how to install Ansible on Centos.  It installs Ansible in a Docker container in order to keep things simple and easy to clean up.  Having build a Docker image with Ansible installed, it tests the image just by running inside the container.

$ ansible --version

2_Run_Example_Adhoc_Commands

This job is a lot more interesting than the previous.  As the name suggests, the job is designed to run some adhoc Ansible commands (which is one of the first things you’ll do when learning Ansible).

Since the purpose of Ansible is infrastructure automation we first need to set up and environment to run commands against.  My idea was to set up an environment of Docker containers pretending to be servers.  In real life I don’t think we would ever want Ansible configuring running Docker containers (we normally want Docker containers to be immutable and certainly don’t want them to have ssh access enabled).  However I felt it a quick way to get started and create something repeatable and disposable.

The environment created resembles the diagram above.  As you can see we create two Docker containers (acting as servers) calling themselves web-node and one calling it’s self db-node.  The images already contain a public key (the same one vagrant uses actually) so that they can be ssh’d to (once again not good practice with Docker containers, but needed so that we can treat them like servers and use Ansible).  We then use an image which we refer to as the Ansible Control Container.  We create this image by installing Ansible installation and adding a Ansible hosts file that tells Ansible how to connect to the db and web “nodes” using the same key mentioned above.

With the environment in place the job runs the following ad hoc Ansible commands:

  1. ping all web nodes using the Ansible ping module: ansible web -m ping
  2. gather facts about the db node using the Ansible setup module: ansible db -m setup
  3. add a user to all web servers using the Ansible user module:  ansible web -b -m user -a “name=johnd comment=”John Doe” uid=1040″

By running the job and reading the console output you can see Ansible in action and then update the job to learn more.

3_Run_Your_Adhoc_Command

This job is identical to the job above in terms of setting up an environment to run Ansible.  However instead of having the hard-coded ad hoc Ansible commands listed above, it allows you to enter your own commands when running the job.  By default it pings all nodes:

ansible all -m ping

4_Run_A_Playbook

This job is identical to the job above in terms of setting up an environment to run Ansible.  However instead of passing in an ad hoc Ansible command, it lets you pass in an Ansible playbook to also run against the nodes.  By default the playbook that gets run installs Apache on the web nodes and PostgreSQL on the db node.  Of course you can change this to run any playbook you like so long as it is set to run on a host expression that matches: web-node-1, web-node-2, and/or db-node (or “all”).

How the jobs 2-4 work

To understand exactly how jobs 2-4 work, the code is reasonably well commented and should be fairly readable.  However, at a high-level the following steps are run:

  1. Create the Ansible inventory (hosts) file that our Ansible Control Container will need so that it can connect (ssh) to our db and web “nodes” to control them.
  2. Build the Docker image for our Ansible Control Container (install Ansible like the first Jenkins job, and then add the inventory file)
  3. Create a Docker network for our pretend server containers and our Ansible Control container to all run on.
  4. Create a docker-compose file for our pretend servers environment
  5. Use docker-compose to create our pretend servers environment
  6. Run the Ansible Control Container mounting in the Jenkins workspace if we want to run a local playbook file or if not just running the ad hoc Ansible command.

Conclusion

I hope this has been a useful read and has clarified a few things about Ansible, ADOP and Docker.  If you find this useful please star the GitHub repo and or share a pull request!

Bonus: here is an ADOP Platform Extension for Ansible Tower.

Advertisements

Reducing Continuous Delivery Impedance – Part 5: Learned Helplessness

Nearly two years ago, I started this blog series to describe the main challenges I’d experienced trying to implement Continuous Delivery.  At the time, the last post in the series was about four challenges related to people.  Since then I’ve observed a fifth challenge and discovered it has been studied in psychology and has a name.

In this post I’ll attempt to describe how to recognise and tackle Learned Helplessness.  Please share your comments (especially if my Psychology-by-Wikipedia needs guidance).

Through various interactions with clients, at meetups, conferences and even with my own team, I’ve witnessed the following phenomena:

  • Something is done (or not done) on an engagement that makes Continuous Delivery difficult (for example the development team accepting SonarQube saying some seriously defamatory things about their unit test coverage but neglecting even to gradually address this).
  • When questioned:
    • many people already appreciate that this is very wrong.
    • hardly anyone can really explain or justify why this is happening.
    • hardly anyone seems worked up about a solution.

It gave me an impression that people had experienced good practice in the past, but having joined this particular engagement had somehow lost the inclination to do it.  It’s possible that for some people, in the past when things just worked, they didn’t question it, so never really appreciated the value of particular practices.  But I think most people are more analytical than that.  I started to realise that people probably had gone through an experience like this:

  • Joined the engagement, didn’t understand why certain things were / weren’t done, but opted to observe before speaking up.
  • Realised things actually weren’t magically working in some new logic- / experience- defying way.
  • Spoke up but didn’t really get listened to.
  • Spoke up again several times , but didn’t really ever get listened to.
  • Gave up and accepted things for the sorry way that they are.

I figured there must be a name for this, started googling and realised it is called Learned Helplessness, something that was first experimented in the 1960’s by some scientists we can probably assume weren’t dog lovers…

The experiments are best described here on Wikipedia but in extremely simplified form:

  1. some dogs were given no random electric shocks,
  2. some dogs were given shocks and also given a button to press to disable the shocks,
  3. some dogs received shocks at the same time as group 2 dogs but had no button.  Group 3 dogs were paired with Group 2 dogs and were shocked until their Group 2 pair happened to press the button (which was at a random time from the Group 3 dog’s perspective).

The learned helplessness of Group 3 was demonstrated in the second part of the experiments when dogs had the opportunity to cross over a small wall to avoid getting shocks.  Whereas groups 1 and 2 quickly learned how to avoid shocks, group 3 all failed to learn and sat their accepting their fate in pain.

http://wariscrime.com/new/wp-content/uploads/2015/02/seligman-2.jpg

The similarity of the above diagram to diagrams about DevOps like this made me smile!

Subsequent experiments demonstrated the ineffectiveness of threats or even rewards on motivating group 3 to change their location.  Only by physically teaching the group 3 dogs to move more than twice did they learn to overcome the helplessness.  Later experiments also proved the same phenomena in humans (without electricity).

So how do we overcome this?

Here are some things I’m experimenting with:

  • Try some introspection – ask yourself what you’ve learnt to accept, really look around for things that are stopping your project going faster – no matter how obvious, and start to ask why, perhaps at least 5 times.
  • Ask others around you ideally at all levels of experience less, the same and more than you what they think is preventing learning and improvement and consider asking “5 Whys” with them.
  • Pay close attention to new joiners to your team – they are the only ones not yet infected by Learned Helplessness.
  • Be sensitive with people.  No-one wants to be told they are “helpless” or hear your amateur psychobabble.  Tread carefully.
  • If you are looking to impart a change, don’t over estimate the impact of threatening or incentivising the people who need to change – they may already be too apathetic.  Instead expect to need to show them multiple times:
    • That the proposed change is possible.  You need to demonstrate it to them (for example if it relates to Continuous Delivery something like the DevOps Platform may help make things real).
    • That their opinions count and they have an important voice.

How is Learned Helplessness harming your organisation and to what extent are you suffering?

 

Neither Carrot Nor Stick

Often when we talk about motivating people, the idiom of having the choice of using a Carrot or a Stick is used. I believe this originates from the conventional wisdom about the best ways to get a mule (as in the four legged horse-like animal) to move. You could try using a carrot, which might be enough of a treat for the mule to move in order to reach it. Or you could try a stick, which might be enough of a threat to get the mule to move in order to avoid being hit.

The idiom works because the carrot is analogous to offering someone an incentive (such as pay rises or bonuses) to get them to do something. The stick is analogous to offering them the threat of punishment (such as being fired or demoted). It’s curious how threat and treat differ by just one letter…

This all makes sense for a horse but not really for people.

The idiom has a major flaw because humans are significantly more complex than animals (all of us!).

Instead if we want to influence someone effectively and sustain-ably, we need to think about how to help them to have an emotional attachment to the thing your are looking to achieve.

I think this comes down to the following:

  • Being open exploring both their and your personal motivations with a view to maximising the achievement of both – in particular the overlap.
  • Starting from an open mind and only looking to agree the desired outcome. This is not the same as agreeing the approach. The approach is key to the satisfaction and motivation of the implementer and key to their attachment to achieving a great solution.
  • Supporting them in their chosen approach taking care not to challenge unnecessarily or do anything that risks eroding their sense of your trust.
  • Being transparent about the consequences of not delivering the desired outcome and clarifying your own role in shielding them from blame and creating a safe environment to operate.

Of course these ideas are not my own.  I would encourage you to explore some of these great materials that I have taken inspiration from:

And I’d love to hear your own ideas and recommended reading.

 

Agile, Waterfall, Common Sense, Experience

Even if you work for an old large traditional enterprise, it’s almost impossible now not to feel the influence of Agile and my advice is embrace it!

Agile:

  • creates / lays claim to a lot of great ideas. Some of these pre-dated it, but there is no harm at all getting them distilled all in one place
  • is extremely popular and even mandated by many organisations.
  • Mis-understood, mis-quoted, mis-interpreted, mis-used, spells TROUBLE.

Therefore, I highly recommend getting up to speed, for example by reading the manifesto. But don’t stop at that, there is huge amounts of great information out there in books and blogs to discover.

I would encourage some effort to learning the lingo. I don’t want to start ranting about how many times I’ve heard Agile terminology misquoted for defending some truly bad (non-Agile) practices (it’s many many times).

However, if you are starting out dabbling in Agile within an established organisation with entrenched processes my advice is to keep the following close to hand:

  • What Agile has to say about how to do things (and what it calls things!)
  • You common sense
  • Your experiences

There really is no excuse for ignoring any of these (and believe me I’ve seen people do it time and time again).

In my head, there is a parallel to be drawn between a methodology like Agile and an Enterprise software package like SAP (stick with me). Both are powerful, both are capable of doing massive things, “out-of-the-box” neither will ever be as effective as they were when they were first created as bespoke solutions to the unique problems that prompted them. – The problem you are solving (whether with a software package or software delivery methodology) will always be unique, and hence so must be the solution

Based on what I’ve seen over the last 4-5 years here are key success factors when following Agile in a project-based scenario:

Key Success Factors

1. Have a project initiation phase that does not complete until the sacrosanct high-level priorities of the sponsor and key high-level architectural decisions are agreed.
2. Adoption requires full co-operation of the Business as well as IT. It also requires all teams to be well trained and to distribute people with Agile experience across scrum teams.
3. Ensure that your performance management process aligns closely with the expected behaviours of people performing Agile roles.
4. Invest heavily in ensuring your application is fast and predictable to release. Without this, the benefits of faster development are lost when changes hit a bottleneck of live implementation.
5. Document and publish development patterns and coding standards and ensure you continually monitor conformance.
6. Constantly strive to reduce quality feedback time, i.e. the time from a developer making a change until the identification anything that related to the change that would indicate it as unworthy of promotion to production. This involves automating as much testing as possible and optimising the execution time of your tests.
7. Allow scrum teams to estimate and plan their own sprints and track the accuracy of this process over time.
8. Ensure you programme is heavily metric-driven and expect to need to continually refine the of your methodology over time to improve effectiveness .

Here are common pitfalls:

1. Forgetting that adopting Agile affects the whole organisation and it will not succeed in silos.
2. Making the assumption that documentation is no longer necessary. If something cannot be validated through an automated test or system monitoring, it still needs to be written down.
3. Using Agile’s innate scope flexibility to the extent that the project direction is in a constant state of flux leading to delays to benefit realisation and a lost recognition of the overall business case.
4. Becoming functionality obsessed and overlooking non-functional requirements such as performance, scalability and operability.
5. Putting an overemphasis on getting “working code” at the expense of poor application architecture and convoluted code in urgent need of re-factoring to adopt patterns and standards to reduce the cost of maintenance.
6. Devoting too much focus to executing pure Agile, as opposed to tailoring an approach that retains the key benefits whilst minimising risk on a large-scale distributed programme.
7. Making the assumption that less planning is required. Agile requires highly detailed planning involving the whole team and an iterative approach that complements and responds to the Agile delivery
8. Focusing on increasing development/test velocity when the bottleneck lies in your test environment stability and ability to predictably deploy code.
9. Ignoring the impact on service and operations teams who are expected to support new systems at a faster rate.
10. Overlooking the how to accommodate potentially slower moving dependencies such as infrastructure provisioning.
11. Doing things based on a perception of what “Agile” tells you to do and neglecting experience and knowledge of your organisation.

However, I must concede most of these pointers are based on common sense and my experience, and most could be applied to Agile, Waterfall, or any methodology.

Please let me know your thoughts.