ADOP with Pivotal Cloud Foundry

As I have written here, the DevOps Platform (aka ADOP) is an integration of open source tools that is designed to provide the tooling capability required for Continuous Delivery.

In this blog I will describe integrating ADOP and the Cloud Foundry public PaaS from Pivotal.  Whilst it is of course technically possible to run all of the tools found in ADOP on Cloud Foundry, that wasn’t our intention.  Instead we wanted to combine the Continuous Delivery pipeline capabilities of ADOP with the industrial grade cloud first environments that Cloud Foundry offers.

Many ADOP cartridges for example the Java Petclinic one contain two Continuous Delivery pipelines:

  • The first to build and test the infrastructure code and build the Platform Application
  • The second to build and test the application code and deploy it to an environment built on the Platform Application.

The beauty of using a Public PaaS like Pivotal Cloud Foundry is that your platforms and environments are taken care of leaving you much more time to focus on the application code.  However you do of course still need to create an account and provision your environments.

  1. Register here
  2. Click Pivotal Web Services
  3. Create a free tier account
  4. Create and organisation
  5. Create one or more spaces

With this in place you are ready to:

  1. Spin up and ADOP instance
  2. Store your Cloud Foundry credentials in Jenkins’ Secure Store
  3. Load the Cloud Foundry Cartridge (instructions)
  4. Trigger the Continuous Delivery pipeline.

Having done all of this, the pipeline now does the following:

  1. Builds the code (which happens to be the JPetStore
  2. Runs the Unit Test and performs Static Code Analysis using SonarQube
  3. Deploys the code to an environment also known in Cloud Foundry as a Space
  4. Performs functional testing using Selenium and some security testing using OWASP ZAPP.
  5. Performs some performance testing using Gatling.
  6. Kills the running application in environment and waits to verify that Cloud Foundry automatically restores it.
  7. Deploys the application to a multi node Cloud Foundry environment.
  8. Kills one of the nodes in Cloud Foundry and validates that Cloud Foundry automatically avoids sending traffic to the killed node.

The beauty of ADOP is that all of this great Continuous Delivery automation is fully portable and can be loaded time and time again into any ADOP instance running on any cloud.

There is plenty more we could have done with the cartridge to really put the PaaS through its paces such as generating load and watching auto-scaling in action.  Everything is on Github, so pull requests will be warmly welcomed!  If you’ve tried to follow along but got stuck at all, please comment on this blog.

Good Platform Opinions lead to Great Systems

Having just got back from the excellent operability.io conference and taken a lot of inspiration, especially from Andrew Shafer, Gareth Rushgrove, Matt Skelton, and Bridget Kromhout (links are to their slides), I felt compelled to write this.

I’ve previously documented my journey to Platform Application enlightenment and even prosed a reference architecture for Platform Applications.  I’ve also documented what I believe all self-respecting Platform Applications should do.  However, I believe there is one critical aspect of any PaaS/PaaA system that needs a lot more attention and credit.

The rules of engagement between the Platform and the Business applications, also known as the opinions of the Platform.

The old adage: “good boundaries create good neighbours” sums it up perfectly for me.

If the Platform imposes a clear set of opinions on the application, I believe this leads to:

  • Short time to mobilise applications to standardised contracts for things like deployments
  • High efficiency due to ready-for-use Platform services such as monitoring and logging
  • Operable applications due to some enforced behaviour for example standard way to stop/start
  • An operable overall system due to consistency for example standardised logging approach
  • Enough abstraction for Platforms to be enhanced with reduced disruption to applications
  • Application portability i.e. increasing the ability for applications to be migrated between different Platforms.*

* Obviously a dependency packaging container solution like Docker can help with this, but there is a lot more to a Platform Application that just  being able to run something.

A great example of Opinions is the 12 Factor App by Heroku.  I’ve heard it come under some criticism as being a ploy to encourage people to write applications that will run on Heroku.  I think that argument is laughable.  I know it’s also come under some criticism that it’s opinions are wrong.  To quote the Big Lebowski, “that’s just your opinion, man”.  Taking it as a definitive set of commandments written in stone for all to follow is the wrong interpretation.  These are the rules for Heroku.  All of them are sensible.  All of them are important to Heroku.  Some are transferable.  Some are not.

I think there is relevance from this taxonomy referenced by Andrew Shafer:

Principles > Process > Tools

Why he amusingly paralleled to Warren Buffet’s:

Innovators > Imitators > Idiots

I think we can parallel:

Principles > Platform Opinions > Platform Implementations.

12 Factor App (or something equivalent e.g. the equivalent opinions from Cloud Foundry – are these so nicely documented?) may have principles you like.  But equally some concrete opinions e.g. log to standard out you may not.

Gareth Rusgrove also called for an end to proliferations of lightweight Docker PaaS’ and:

“Publish more schemas and fewer incompatible duplicate implementations”

I think Platform Opinions can also be thought of schemas.

What about downsides of Platform Opinions?  

The only things I can think of would only be caused if opinions were created badly:

  • They could be inappropriate for the Business Applications, for example mandating that all state must be persisted in a database could be a good idea. But if an application that absolutely must store things to disk (e.g. Open LDAP) is needed, the opinion must be updated.  But this really is just bad design.
  • They could drive inhibit Dev-to-Ops-harmony by the Platform Application team (Operations) enforcing not-invented-here rules on Developer (Business Application) teams. I think close initial collaboration around the creation of rules is key here
  • They could be too prescriptive where one size may not fit all applications and lead to strange application designs
  • They could be too loose meaning they fail to make things predictable for the Platform and achieve the outcomes above

So I believe clearly defined, and optimised-for Platform Opinions are an unsung hero of the PaaS/PaaA prospect.

In fact, I feel so strongly about this, I think it almost calls for Platform Opinions schema and even a Manifesto.  I’m very surprised to find very little when searching Google for this. I’d like to see a books like “Patterns for Platform Opinions”, “How to write good Platform Opinions”.  Is there anything like this out there, is anyone with me?

PaaA_Opinions

(Great conference HighOps and thanks to everyone for the inspiration.)