Not such a dramatic title this time… DevOpsDays London – Day 2

Here are some highlights of Devops Days London November 2013 – Day 2 that I wanted to share.  If you missed my blog about Day 1, it’s here.

First up was John Willis aka botchagalupe who is well-known in the DevOps community, partly due his podcasts DevOpsCafe.  John was a great presenter and only a few minutes in, I was resolute to listen to more DevOpsCafe.

John spoke about software driven networks (SDN) which he described as the next frontier of DevOps.   He started off explaining how SDN works.  I won’t attempt to regurgitate it, but I can highly recommend this video about SDN that I watched after someone recommended it at the last DevOpsDays London.

John provided an excellent list of resources on SDN here.

John drew parallels between the adoption of automated Configuration Management tools (e.g. Puppet and Chef) and the adoption of SDN.  He predicted that SDN will experience similar reactions that automated Configuration Management received such as fear and resistance.  I’m sure he’s right.  

It was interesting to me that he regarded adoption of automated Configuration Management tools as something that happened between 2000 and 2010.  This was a reminder of quite how far ahead of the game some organisations are with infrastructure-as-code.  Personally I wasn’t even aware of Puppet until at least 2009!  

John mentioned that Frenetic is an important language in SDN and that SDN has its own equivalent to Vagrant in the form of Mininet.  If that sentence doesn’t make sense, basically Vagrant is a tool for managing virtual images on your local development machine so that you can develop against more live-like servers and so that you can develop and test infrastructure-as-code code. So Mininet is a tool to developing and testing SDN code.  If you aren’t yet into Vagrant, read this.

Next up was Jeffrey Fredrick to talk about culture and mutual learning.  It was a great contrast to John’s talk and a reminder that in addition to tools and processes, the DevOps movement is heavily concerned with improving the culture at organisations.  (NB. John did mention culture in particular CAMS

Jeffrey talked though some of the concepts on here and I particularly liked the graph that true collaboration being dependent on mutual trust and respect, and a willingness and freedom to disagree.  It make me reflect a lot on particular situations in the past when conflict had occurred and I wondered whether it was indeed due to lacking mutual trust and respect.

Jeffrey described a psychology experiment where smoke had been pumped into a doctors’ surgery waiting room.  When one person was in the room, the person was most likely to draw the obvious logical conclusion that the smoke wasn’t a good sign and leave the room.  However, when multiple people were in the waiting room more often than not people didn’t move for fear of embarrassment.  I think this is called the Bystander Effect.  Anyway, the main point that I took from it was around avoiding embarrassing your colleagues.  I certainly agree that working hard to think about how your actions may make others feel is extremely important, especially during high stress times like service outages.

He mentioned Chris Argyris who I’d come across before due to his work about organisations and learning.  Definitely going to read some of his books.

The last full talk of the day was from Pieter_Hintjens.  Peter spoke about successful rules for open source software projects  He spoke without slides and stood in front of the stage and I must admit I found this format a little harder to follow.  He described how well the ZeroMQ community works and it certainly impressed me enough to want to find out more by literally reading the ZeroMQ manual and specifically the chapter about the community.  I also read his wikipedia page which highlights his strong and very interesting views about patents.

What a great DevOpsDays event, can’t wait for the next one!

At what point will the human race depend on DevOps?

Here are some highlights of Devops Days London November 2013 – Day 1 that I wanted to share.

Mark Burgess gave an interesting talk based on his new book In Search Of Certainty.  To those that don’t already know Mark, he was the creator of CFEngine which is father of Puppet and the grandfather of Chef.  CFEngine began 20 years ago, so it is very interesting for those of us who have only a couple of years working with active Configuration Management tools to learn intricacies from someone who has been evolving them and observing them for 20 years.

Mark started off challenging the notion that “consistency equals quality” recognizing the almost organic nature of highly complex systems.  All in all, the talk was fairly challenging stuff to start the day with…

Mark ended his talk with a sobering thought that the human race is becoming so dependent on complex IT systems that at some point our whole prospects of survival will depend on them.  This left me thinking, it’s hard enough using DevOps techniques to delivery more continuously, did he have to raise the stakes so much higher!?

Doug Barth from Pager Duty did a great talk called Failure Fridays, which also could be called Chaos Monkey for Mortals (i.e. artificially creating failure for testing when you are not NetFlix).  It was a great session because he really explained the in’s and out’s of what they do, which is basically spending one hour per week (in total) systematically breaking, observing and fixing their system (only creating failures at a scale that they expect to be able to tolerate without impacting their customers).

Where I work, system delivery projects include formally testing all of these things up front and we call it Operator Acceptance Testing.  However, I definitely like the idea of it being an on-going activity (certainly resonates with the “if it’s difficult, do it often mantra”).

Practical considerations of doing it weekly with the live system included:

  • constrain to one hour (end to end)
  • involve of all impacted parties all in one room (chat/video conference room if necessary)
  • lots of planing and careful assessment of risk before hand
  • careful what you disable:
    • disabling auto-healing mechanisms = probably good
    • disabling monitoring and alerts (things you want to test) = bad
  • effective communications before during and afterwards (this is not like a surprise fire drill)
  • starting small e.g. one server and scaling up to a bank

Doug also described other virtues of this activity such as it creating a great opportunity to train new team members and the effect that it had on increasing focus on failure with developers.  I’d bet that at least half the people in the audience went away wanting to implement Failure Fridays and many of them will actually do that.  Just not on a Friday though – why jeopardize Saturdays?

Ben Hughes from Etsy did the next talk about Security and it was a very lively, entertaining and informative session.  Rather than try to do it justice, instead I just highly recommend watching the video of his 5 minute ignite from DevOpsDays Portland.  However just one quote I can’t resist sharing here (credited to Ben’s boss apparently)

“If you go down from 35% on fire to 24% on fire, you are still on fire”

– referring to issues that you have to accept exist and manage rather than trying to avoid / deny the possibility of happening.

After my ignite where I stood up for DevOps teams, I’ll admin, my head was a bit of a blur and I didn’t take any notes.  However, a special highlight to Daniel Breston for teaching us about oobeya.

The afternoon Open Sessions of course delivered some lively debate.  It was particularly interesting to see some early examples of discussions about Docker versus Configuration Management.  A debate that is likely to rage!

All in all, an excellent first day (even without needing to mention the free bar in the evening)!  Thanks DevOpsDays!