Using ADOP and Docker to Learn Ansible

As I have written here, the DevOps Platform (aka ADOP) is an integration of open source tools that is designed to provide the tooling capability required for Continuous Delivery.  Through the concept of cartridges (plugins) ADOP also makes it very easy to re-use automation.

In this blog I will describe an ADOP Cartridge that I created as an easy way to experiment with Ansible.  Of course there are many other ways of experimenting with Ansible such as using Vagrant.  I chose to create an ADOP cartridge because ADOP is so easy to provision and predictable.  If you have an ADOP instance running you will be able to experience Ansible doing various interesting things in under 15 minutes.

To try this for yourself:

  1. Spin up and ADOP instance
  2. Load the Ansible 101 Cartridge (instructions)
  3. Run the jobs one-by-one and in each case read the console output.
  4. Re-run the jobs with different input parameters.

To anyone only loosely familiar with ADOP, Docker and Ansible, I recognise that this blog could be hard to follow so here is a quick diagram of what is going on.

docker-ansible

The Jenkins Jobs in the Cartridge

The jobs do the following things:

As the name suggests, this job just demonstrates how to install Ansible on Centos.  It installs Ansible in a Docker container in order to keep things simple and easy to clean up.  Having build a Docker image with Ansible installed, it tests the image just by running inside the container.

$ ansible --version

2_Run_Example_Adhoc_Commands

This job is a lot more interesting than the previous.  As the name suggests, the job is designed to run some adhoc Ansible commands (which is one of the first things you’ll do when learning Ansible).

Since the purpose of Ansible is infrastructure automation we first need to set up and environment to run commands against.  My idea was to set up an environment of Docker containers pretending to be servers.  In real life I don’t think we would ever want Ansible configuring running Docker containers (we normally want Docker containers to be immutable and certainly don’t want them to have ssh access enabled).  However I felt it a quick way to get started and create something repeatable and disposable.

The environment created resembles the diagram above.  As you can see we create two Docker containers (acting as servers) calling themselves web-node and one calling it’s self db-node.  The images already contain a public key (the same one vagrant uses actually) so that they can be ssh’d to (once again not good practice with Docker containers, but needed so that we can treat them like servers and use Ansible).  We then use an image which we refer to as the Ansible Control Container.  We create this image by installing Ansible installation and adding a Ansible hosts file that tells Ansible how to connect to the db and web “nodes” using the same key mentioned above.

With the environment in place the job runs the following ad hoc Ansible commands:

  1. ping all web nodes using the Ansible ping module: ansible web -m ping
  2. gather facts about the db node using the Ansible setup module: ansible db -m setup
  3. add a user to all web servers using the Ansible user module:  ansible web -b -m user -a “name=johnd comment=”John Doe” uid=1040″

By running the job and reading the console output you can see Ansible in action and then update the job to learn more.

3_Run_Your_Adhoc_Command

This job is identical to the job above in terms of setting up an environment to run Ansible.  However instead of having the hard-coded ad hoc Ansible commands listed above, it allows you to enter your own commands when running the job.  By default it pings all nodes:

ansible all -m ping

4_Run_A_Playbook

This job is identical to the job above in terms of setting up an environment to run Ansible.  However instead of passing in an ad hoc Ansible command, it lets you pass in an Ansible playbook to also run against the nodes.  By default the playbook that gets run installs Apache on the web nodes and PostgreSQL on the db node.  Of course you can change this to run any playbook you like so long as it is set to run on a host expression that matches: web-node-1, web-node-2, and/or db-node (or “all”).

How the jobs 2-4 work

To understand exactly how jobs 2-4 work, the code is reasonably well commented and should be fairly readable.  However, at a high-level the following steps are run:

  1. Create the Ansible inventory (hosts) file that our Ansible Control Container will need so that it can connect (ssh) to our db and web “nodes” to control them.
  2. Build the Docker image for our Ansible Control Container (install Ansible like the first Jenkins job, and then add the inventory file)
  3. Create a Docker network for our pretend server containers and our Ansible Control container to all run on.
  4. Create a docker-compose file for our pretend servers environment
  5. Use docker-compose to create our pretend servers environment
  6. Run the Ansible Control Container mounting in the Jenkins workspace if we want to run a local playbook file or if not just running the ad hoc Ansible command.

Conclusion

I hope this has been a useful read and has clarified a few things about Ansible, ADOP and Docker.  If you find this useful please star the GitHub repo and or share a pull request!

Bonus: here is an ADOP Platform Extension for Ansible Tower.

Abstraction is not Obsoletion – Abstraction is Survival

Successfully delivering Enterprise IT is a complicated, probably even complex problem.  What’s surprising, is that as an industry, many of us are still comfortable accepting so much of the problem as our own to manage.

Let’s consider an albeit very simplified and arguably imprecise view of The “full stack”:

  • Physical electrical characteristics of materials (e.g. copper / p-type silicon, …)
  • Electronic components (resistor, capacitor, transistor)
  • Integrated circuits
  • CPUs and storage
  • Hardware devices
  • Operating Systems
  • Assembly Language
  • Modern Software Languages
  • Middleware Software
  • Business Software Systems
  • Business Logic

When you examine this view, hopefully (irrespective of what you think about what’s included or missing and the order) it is clear that when we do “IT” we are already extremely comfortable being abstracted from detail. We are already fully ready to use things which we do not and may never understand. When we build an eCommerce Platform, an ERP, or CRM system, little thought it given to Electronic components for example.

My challenge to the industry as a whole is to recognise more openly the immense benefit of abstraction for which we are already entirely dependent and to embrace it even more urgently!

Here is my thinking:

  • Electrons are hard – we take them for granted
  • Integrated circuits are hard – so we take them for granted
  • Hardware devices (servers for example) are hard – so why are so many enterprises still buying and managing them?
  • The software that it takes to make servers useful for hosting an application is hard – so why are we still doing this by default?

For solutions that still involve writing code, the most extreme example of abstraction I’ve experienced so far is the Lambda service from AWS.  Some seem to have started calling such things ServerLess computing.

With Lambda you write your software functions and upload them ready for AWS to run for you. Then you configure the triggering event that would cause your function to run. Then you sit back and pay for the privilege whilst enjoying the benefits. Obviously if the benefits outweigh the cost for the service you are making money. (Or perhaps in the world of venture capital, if the benefits are generating lots of revenue or even just active users growth, for now you don’t care…)

Let’s take a mobile example. Anyone with enough time and dedication can sit at home on a laptop and start writing mobile applications. If they write it as a purely standalone, offline application, and charge a small fee for it, theoretically they can make enough money to retire-on without even knowing how to spell server.  But in practice most applications (even if they just rely on in app-adverts) require network enabled services. But for this our app developer still doesn’t need to spell server, they just need to use the API of the online add company e.g. Adwords and their app will start generating advertising revenue. Next perhaps the application relies on persisting data off the device or notifications to be pushed to it. The developer still only needs to use another API to do this, for example Parse can provide that to you all as a programming service.  You just use the software development kit and are completely abstracted from servers.

So why are so many enterprises still exposing themselves to so much of the “full stack” above?  I wonder how much inertia there was to integrated circuits in the 1950s and how many people argued against abstraction from transistors…

To survive is to embrace Abstraction!

 


[1] Abstraction in a general computer science sense not a mathematical one (as used by Joel Spolsky in his excellent Law of Leaky Abstractions blog.)

Running the DevOps Platform on Microsoft Azure

As per my last post about GCE sometimes knowing something is possible just isn’t good enough.  So here is how I spun up the DevOps Platform on the Microsoft Azure cloud.  Warning thanks to Docker Machine, this post is very similar to this earlier one.

1. I needed an Azure account.

2. I logged into my Azure account and didn’t click “view the new Portal”.

3. On the left hand menu, I scrolled down to the bottom (it didn’t look immediately to me like it will scroll so hover) and clicked settings.  Here I was able to see my subscription ID and copy it.

4. (Having previously installed Docker Toolbox, see here) I opened Git Bash (as an Administrator) and ran this command:

$ docker-machine create --driver azure --azure-size Standard_A3 --azure-subscription-id <the ID I just copied> markos01

I was prompted to open a url in my brower, enter a confirmation code, and then login with my Azure credentials.  Credit to Microsoft, this was easier than GCE for which I needed to install the gcloud commandline utility!

You will notice that this is fairly standard.  I picked an Standard_A3 machine type which is roughly equivalent to what we use for AWS and GCP.

5. I waited while a machine was created in Azure containing Docker

6. I cloned the ADOP Docker Compose repository from GitHub:

$ git clone https://github.com/Accenture/adop-docker-compose
$ cd adop-docker-compose

7. I ran the normal startup.sh command as follows:

$ ./startup.sh -m markos01 -c NA

And entered a user name (thanks to this recent enhancement), hey presto

...
SUCCESS, your new ADOP instance is ready!
Run these commands in your shell:
eval \"$(docker-machine env $MACHINE_NAME)\"
source env.config.sh
Navigate to http://52.160.97.159 in your browser to use your new DevOps Platform!

And just to prove it:

$ whois 52.160.97.159 | grep Org
Organization: Microsoft Corporation (MSFT)
OrgName: Microsoft Corporation
OrgId: MSFT

8. I had to go to All resources > markos01-firewall > Inbound security rules and added a rule to allow HTTP to my server on port 80.

9. I viewed my new ADOP on Azure hosted instance in (of course…) Chrome! 😉

More lovely stuff!

 

Running the DevOps Platform on Google Compute Engine

Sometimes knowing something is possible just isn’t good enough.  So here is how I spun up the DevOps Platform on Google Compute Engine (GCE).

1. I needed a Google Compute Engine account.

2. I enabled the Google Compute APIs for my GCE account

3. I installed the Google Cloud commandline API

4. I opened the Google Cloud SDK Shell link that had appeared in my Windows Start menu and ran:

C:\> gcloud auth login

This popped open a Chrome window and asked me to authenticate against my GCE account.

5. (Having previously installed Docker Toolbox, see here) I opened Git Bash (as an Administrator) and ran this command:

$ docker-machine create --driver google \
                 --google-project <a project in my GCE account> \
                 --google-machine-type n1-standard-2 \
                 markosadop01

You will notice that this is fairly standard.  I picked an n1-standard-2 machine type which is roughly equivalent to what we use for AWS.

6. I waited while a machine was created in Google containing Docker

7. I cloned the ADOP Docker Compose repository from GitHub:

$ git clone https://github.com/Accenture/adop-docker-compose
$ cd adop-docker-compose

8. I ran the normal startup.sh command as follows:

$ git clone https://github.com/Accenture/adop-docker-compose
$ ./startup.sh -m markosadop01 -c NA

And hey presto:

...
SUCCESS, your new ADOP instance is ready!
Run these commands in your shell:
eval "$(docker-machine env $MACHINE_NAME)"
source env.config.sh
Navigate to http://104.197.235.64 in your browser to use your new DevOps Platform!

And just to prove it:

$ whois 104.197.235.64 | grep Org
Registrant Organization: Google Inc.
Admin Organization: Google Inc.
Tech Organization: Google Inc.

9. I had to go to Networks > Firewall rules and added a rule to allow HTTP to my server.

10. I viewed my new ADOP on Google instance in (of course…) Chrome!

Lovely stuff!

Reusable Docker Testing Approach

In this blog I will describe a reusable approach to testing Docker that I have been working on.

By ‘testing Docker’ I mean the performing the following actions:

  • Static code analysis of the Dockerfile i.e. is the file syntactically valid and written to our expected standards?
  • Unit testing the Docker Image created by performing a build with our Dockerfile i.e. does our Dockerfile look like it created the Image we were expecting?
  • Functional testing the Container created by running an instance of our container i.e. when running does it look and do as we expected?

I wanted a solution that was very easy to adopt and extend so I chose to:

  • implement it in Docker so that it will work for anyone using Docker (see this diagram)
  • use Docker Compose to make it as easy to trigger
  • reuse Dockerlint
  • use Ruby because it is fairly widespread as a required skill in infrastructure-as-code people (for now until Go takes over…), and because the docker-api Gem is very powerful, albeit expects you to learn more about the Docker API in order to use it.
  • use RSpec an ServerSpec as testing framework because they have good documentation and the support BDD

So what is the solution?

Essentially it is a Docker image called test-docker.  To use it, you must mount-in your ‘Dockerfile’ and your ‘tests’ directory, it then:

  1. Runs Dockerlint to perform the static code analysis on the Docker file
  2. Runs your tests which I encourage you write for both inspecting the image and testing a running container.

How to see it in action?

To run this you need Docker installed and functioning happily.  Personally I’m using:

  • a Windows laptop
  • Docker Toolbox which gave me: docker-machine which manages a Linux virtual machine for me running on a local installation of Virtual Box
  • docker-compose installed (I did it manually)
  • git bash aka Git For Windows as my terminal

With the above or equivalent, you simply need to do:

$ git clone https://github.com/kramos/test-docker.git
$ cd test-docker
$ docker-compose -f docker-compose-test-docker.yml up

You should see an output like this:

Creating testdocker
Creating testdocker_lintdocker_1
Attaching to testdocker, testdocker_lintdocker_1
testdocker   | /usr/local/bin/ruby -I/usr/local/bundle/gems/rspec-support-3.4.0/lib:/usr/local/bundle/gems/rspec-core-3.4.0/lib /usr/local/bundle/gems/rspec-core-3.4.0/exe/rspec --pattern spec/\*_spec.rb
lintdocker_1 | Check passed!
testdocker_lintdocker_1 exited with code 0
testdocker   |
testdocker   | Container
testdocker   |   get running
testdocker   |     check ruby
testdocker   |       Command "ruby --version"
testdocker   |         stdout
testdocker   |           should match /ruby/
testdocker   |         stderr
testdocker   |           should be empty
testdocker   |
testdocker   | Image
testdocker   |   inpsect metadata
testdocker   |     should not expose any ports
testdocker   |
testdocker   | Finished in 1.48 seconds (files took 1.45 seconds to load)
testdocker   | 3 examples, 0 failures
testdocker   |
testdocker exited with code 0
Gracefully stopping... (press Ctrl+C again to force)


 

All good.  But what happened?  Well everything I’ve said we wanted to happen, against the test-docker tool.  #Dogfood and all that.

You can also try out another example e.g.:

$ docker-compose -f examples/redis/docker-compose.yml up

So how to use this for your own work?

Hopefully you’ll agree this is very easy (at least to get started):

  1. Replace the Dockerfile in the root of the test-docker folder with your own Dockerfile (plus any other local resources your Dockerfile needs)
  2. Run the following (this time we allow docker-compose to use the default configuration file which you also pulled from Git:
$ docker-compose up
  1. You will find out what Dockerlint thinks of your code, followed by finding out whether by extreme luck any of the tests that were written for the test-docker image (as opposed to your image) pass
  2. Open the rb file (in tests/spec) and update it to test your application using anything that you can do to a stopped container using the docker-api
  3. Open the rb file (in tests/spec) and update it to test your application using anything that you can do to a running container using the docker-api and Serverspec.
  4. I suggest remove the .git folder and initialise a git repository to manage your Dockerfile and your tests.

 

Functional tests that run the container require two subtly different approaches according to whether your Docker image is expected to run as a daemon or just run, do something and stop.  In the former case, you can use a lot of Serverspec functionality.  In the latter, your choices are more limited to running the container multiple times and in each case grabbing the output and parsing it.

 

Conclusion

 

There were a surprising number of things I had to learn on the fly here to get this working, but I don’t want this blog to drag on.  Let me know how you get on and I will happily share more, especially when any of magic things don’t work as expected – for example when writing tests.

 

I’ll leave you with my current list of things I want to improve:

  • Make work with Ruby slim (the image is huge)
  • Get working with Inspec instead of ServerSpc
  • Provide better examples of tests
  • I should really draw a diagram to help anyone new to this understand all this inception computing…

 

Credits:

I took a huge amount of help from:
http://www.unixdaemon.net/tools/testing-dockerfiles-with-serverspec/
https://github.com/sherzberg/docker-hhvm/
https://github.com/18F/docker-elasticsearch

 

Docker Inception


Sometimes when you are working with Docker it feels a bit like the movie inception  It doesn’t help when you are doing things like this.  So here is a diagram that might make things clearer.

docker-inception

Good Platform Opinions lead to Great Systems

Having just got back from the excellent operability.io conference and taken a lot of inspiration, especially from Andrew Shafer, Gareth Rushgrove, Matt Skelton, and Bridget Kromhout (links are to their slides), I felt compelled to write this.

I’ve previously documented my journey to Platform Application enlightenment and even prosed a reference architecture for Platform Applications.  I’ve also documented what I believe all self-respecting Platform Applications should do.  However, I believe there is one critical aspect of any PaaS/PaaA system that needs a lot more attention and credit.

The rules of engagement between the Platform and the Business applications, also known as the opinions of the Platform.

The old adage: “good boundaries create good neighbours” sums it up perfectly for me.

If the Platform imposes a clear set of opinions on the application, I believe this leads to:

  • Short time to mobilise applications to standardised contracts for things like deployments
  • High efficiency due to ready-for-use Platform services such as monitoring and logging
  • Operable applications due to some enforced behaviour for example standard way to stop/start
  • An operable overall system due to consistency for example standardised logging approach
  • Enough abstraction for Platforms to be enhanced with reduced disruption to applications
  • Application portability i.e. increasing the ability for applications to be migrated between different Platforms.*

* Obviously a dependency packaging container solution like Docker can help with this, but there is a lot more to a Platform Application that just  being able to run something.

A great example of Opinions is the 12 Factor App by Heroku.  I’ve heard it come under some criticism as being a ploy to encourage people to write applications that will run on Heroku.  I think that argument is laughable.  I know it’s also come under some criticism that it’s opinions are wrong.  To quote the Big Lebowski, “that’s just your opinion, man”.  Taking it as a definitive set of commandments written in stone for all to follow is the wrong interpretation.  These are the rules for Heroku.  All of them are sensible.  All of them are important to Heroku.  Some are transferable.  Some are not.

I think there is relevance from this taxonomy referenced by Andrew Shafer:

Principles > Process > Tools

Why he amusingly paralleled to Warren Buffet’s:

Innovators > Imitators > Idiots

I think we can parallel:

Principles > Platform Opinions > Platform Implementations.

12 Factor App (or something equivalent e.g. the equivalent opinions from Cloud Foundry – are these so nicely documented?) may have principles you like.  But equally some concrete opinions e.g. log to standard out you may not.

Gareth Rusgrove also called for an end to proliferations of lightweight Docker PaaS’ and:

“Publish more schemas and fewer incompatible duplicate implementations”

I think Platform Opinions can also be thought of schemas.

What about downsides of Platform Opinions?  

The only things I can think of would only be caused if opinions were created badly:

  • They could be inappropriate for the Business Applications, for example mandating that all state must be persisted in a database could be a good idea. But if an application that absolutely must store things to disk (e.g. Open LDAP) is needed, the opinion must be updated.  But this really is just bad design.
  • They could drive inhibit Dev-to-Ops-harmony by the Platform Application team (Operations) enforcing not-invented-here rules on Developer (Business Application) teams. I think close initial collaboration around the creation of rules is key here
  • They could be too prescriptive where one size may not fit all applications and lead to strange application designs
  • They could be too loose meaning they fail to make things predictable for the Platform and achieve the outcomes above

So I believe clearly defined, and optimised-for Platform Opinions are an unsung hero of the PaaS/PaaA prospect.

In fact, I feel so strongly about this, I think it almost calls for Platform Opinions schema and even a Manifesto.  I’m very surprised to find very little when searching Google for this. I’d like to see a books like “Patterns for Platform Opinions”, “How to write good Platform Opinions”.  Is there anything like this out there, is anyone with me?

PaaA_Opinions

(Great conference HighOps and thanks to everyone for the inspiration.)

Reducing Continuous Delivery Impedance – Part 2: Solution Complexity

This is my second post in a series I’m writing about impedance to doing continuous delivery and how to overcome it.  Part 1 about Infrastructure challenges can be found here.  I also promised to write about complexity in continuous delivery in this earlier post about delivery pipelines.

I’m defining “a solution” as the software application or multiple applications under current development than need to work together (and hence be tested in an integrated manner) before release to production.

In my experience, continuous delivery pipelines work extremely well when you have a simple solution with the following convenient characteristics:

  1. All code and configuration is stored in one version control repository (e.g. Git)
  2. The full solution can be deployed all the way to production without needing to test it in conjunction with other applications / components under development
  3. You are using a 3rd party PaaS (treated as a black box, like HerokuGoogle App Engine, or AWS Elastic BeanStalk)
  4. The build is quick to run i.e. less than 5 minutes
  5. The automated tests are quick to run, i.e. minutes
  6. The automated test coverage is sufficient that the risks associated of releasing software can be understood to be lower in value than the benefits of releasing.

The first 3 characteristics are what I am calling “Solution Complexity” and what I want to discuss this post.

Here is a nice simple depiction of an application ticking all the above boxes.

perfect

Developers can make changes in one place, know that their change will be fully tested and know that when deployed into the production platform, their application should behave exactly as expected.  (I’ve squashed the continuous delivery (CD) pipeline into just one box, but inside it I’d expect to see a succession of code deployments, and automated quality gates like this.)

 

But what about when our solution is more complex?

What about if we fail to meet the first characteristic and our code is in multiple places and possibly not all in version control?  This definitely a common problem I’ve seen, in particular for configuration and data loading scripts.  However, this isn’t particularly difficult to solve from a technical perspective (more on the people-side in a future post!).  Get everything managed by a version control tool like Git.

Depending on the SCM tool you use, it may not be appropriate to feel obliged to use one repository.  If you do use multiple, most continuous integration tools (e.g. Jenkins) can be set up in such a way as to support handling builds that consume from multiple repositories.  If you are using Git, you can even handle this complexity within your version control repository e.g. by using sub-modules.

 

What about if your solution includes multiple applications like the following?

complex

Suddenly our beautiful pipeline metaphor is broken and we have a network of pipelines that need to converge (analogous to fan in in electronics).  This is far from a rarity and I would say it is overwhelmingly the norm.  This certainly makes things more difficult and we now have to carefully consider how our plumbing is going to work.  We need to build what I call an “integrated pipeline”.

Designing an integrated pipeline is all about determining the “points of integration” aka POI i.e. the first time that testing involves the combination two or more components.  At this point, you need to record the versions of each component so that they are kept consistent for the rest of the pipeline.  If you fail to do this, earlier quality gates in the pipeline are invalidated.

In the below example, Applications A and B have their own CD pipelines where they will be deployed to independent test environments and face a succession of independent quality gates.  Whenever a version of Application A or B gets to the end of its respective pipeline, instead of going into production, it moves into the Integrated Pipeline and creates a new integrated or composite build number.  After this “POI” the applications progress towards production in the same pipeline and can only move in sync.  In the diagram, version A4 of Application A and version B7 of B have made it into integration build I8.  If integration build I8 makes it through the pipeline it will be worthy to progress to production.

intDepending on the tool you use for orchestration, there are different solutions for achieving the above.  Fundamentally it doesn’t have to be particularly complicated.  You are simply aggregating version numbers in which can easily be stored together in a text document in any format you like (YAMLPOMJSON etc).

Some people reading this may by now be boiling up inside ready to scream “MICRO SERVICES” at their screens.  Micro services are by design independently deploy-able services.  The independence is achieved by ensuring that they fulfill and expect to consume strict contract APIs so that integration with other services can be managed and components can be upgraded independently.  A convention like SemVer can be adopted to manage change to contract compatibility.  I’ve for a while had this tagged in my head as the eBay way or Amazon way of doing this but micro services are now gaining a lot of attention.  If you are implementing micro services and achieving this independence between pipelines, that’s great.  Personally on the one micro services solution I’ve worked on so far, we still opted for an integrated pipeline that operated on an integrated build and produce predictable upgrades to production (we are looking to relax that at some point in the future).

Depending on how you are implementing your automated deployment, you may have deployment automation scripts that live separately to your application code.  Obviously we want to use consistent version of these through out deployments to different environments in the pipeline.  Therefore I strongly advise managing these scripts as a component in the same manner.

What about if you are not using a PaaS?  In my experience, this represents the vast majority of solutions I’ve worked on.  If you are not deploying into a fully managed container, you have to care about the version of the environment that you are deploying into.  The great thing about treating infrastructure as code (assuming you overcome that associated impedance) is that you can treat it like an application, give it a pipeline and feed it into the integrated pipeline (probably at a POI very early).  Effectively you are creating your own platform and performing continuous delivery on that.  Obviously the further your production environment is from being a version-able component like this, the great the manual effort to keep environments in sync.

paas

 

Coming soon: more sources of impedance to doing continuous delivery: Software packages, Organisation size, Organisation structure, etc.

 

(Thanks to Tom Kuhlmann for the graphic symbols.)