DevOps: Culture,

Agile Development, &

Cloud Native Technologies
CSCI-GA 2820, Graduate Division, Computer Science

Instructor:
John J Rofrano
Senior Technical Staff Member | DevOps Champion
IBM T.J. Watson Research Center
[email protected] (@JohnRofrano)
1
 What are businesses up against?

52%
of the Fortune 500 have
disappeared since the year
2000

2
@JohnRofrano https://www.capgemini.com/consulting/wp-content/uploads/sites/30/2017/07/digital_disruption_1.pdf
 DevOps

Disruptive Business Models

of companies believe their

business model will remain
Example: economically viable through
Call Taxi Service Uber digitization

Source: McKinsey Digital Global Survey, 2016 and 2017; McKinsey analysis

@JohnRofrano 3
 DevOps

Don’t Underestimate Impact of Digitization + Business

Model

A ridesharing service is 40% cheaper than a regular When was the last time you used a travel agent,
cab for a 5 mile trip into Los Angeles bought a GPS device, Or carried a point-and-shoot
camera separate from your phone?
$$$ Ridesharing

$$$$$ Taxi

@JohnRofrano 4
 DevOps

Technology is the enabler of innovation

...it is not the driver of innovation

@JohnRofrano 5
 DevOps

Unlearn
what you
have learned
Often easier said
than done

@JohnRofrano
6 © 2018 IBM Corporation 27 February 2019 IBM Services
 DevOps

Consider this...

• What if you could fail fast and roll back quickly?

• Then the impact of failure would be a minimal "blast radius”
• What if you could test in-market instead of analyzing?
• Then you could experiment with customers instead of second-guessing them
• What if your application design allowed individual components to be replaced?
• Then you wouldn't have "big bang" release weekends

@JohnRofrano 7
 DevOps

Allspaw @ Velocity 2009

https://www.youtube.com/watch?v=LdOe18KhtT4

• Most people will cite John Allspaw

(@allspaw) and Paul Hammond’s
Velocity 2009 presentation titled “10+
Deploys Per Day: Dev and Ops
Cooperation at Flickr” as a pivotal
moment in the Devops movement.

• “In the last week there were 67 deploys

of 496 changes by 18 people” – Flickr
Dev Blog, December 17th 2008.

https://conferences.oreilly.com/velocity/velocity2009/public/schedule/detail/7641
@JohnRofrano 8
 DevOps

2011 Etsy Deploys to Production every ~25 min*

• In January 2011 (a month in

which we did over a billion
page views)
• Code committed by 76 unique
individuals
• Was deployed to production by
63 different folks
• A total of 517 times
* Based on 22 days 10 hrs/day (517 / 22 / 10 = 2.35/hr)
@JohnRofrano 9
 DevOps

“Our deployment environment requires a lot of trust,

transparency, communication, coordination, and
discipline across the team.”
–Chad Dickerson, CTO Etsy

@JohnRofrano 10 http://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/
 DevOps

...but these are

"mythical"
companies,
right?

...big
enterprisers
can't possibly
work like this,
right?
@JohnRofrano 11
 DevOps

2016 DevOps Enterprise Summit

• Ticketmaster - 98% reduction in MTTR

• Nordstrom - 20% shorter Lead Time

2016 • Target - Full Stack Deploy 3 months to

minutes

• USAA - Release from 28 days to 7 days

• ING - 500 application teams doing DevOps

• CSG - From 200 incidents per release to 18

@JohnRofrano 12
 DevOps

How are they doing this?

@JohnRofrano 13
 DevOps

They have embraced the DevOps

culture

@JohnRofrano 14
 DevOps

What is DevOps?

@JohnRofrano 15
 DevOps

“The term (development and operations) is an extension of

agile development environments that aims to enhance the
process of software delivery as a whole.”

–Patrick Debois, 2009

@JohnRofrano 16
 DevOps

What is DevOps?

DevOps is a recognition that Development and Operations needs to

stop working alone in their "siloed" towers and start working together

• To do this we need:
• A culture of collaboration valuing openness, trust, and transparency
• An application design that does not require entire systems to be redeployed just
to add a single function
• Automation that accelerates and improves the consistency of application delivery
so that we can develop and deliver software with speed and stability
• A dynamic software-defined, programmable platform to continuously deploy
onto
@JohnRofrano 17
 DevOps

“DevOps is the practice of development and operations

engineers working together during the entire software
lifecycle, following lean and agile principles that allow
them to deliver software in a rapid and continuous
manner.”

@JohnRofrano 18
 DevOps

Principles of DevOps (CAMS)

• Culture - A DevOps culture is collaborative and

employees across departments are in constant
communication and embrace rapid change
• Automation - The replacement of manual processes
to accelerate digital product delivery, improve
ef ciency, and reduces the potential of human error
• Measuring - By measuring the outcomes of your
projects you can prove the value of the process.
• Sharing - Information sharing improves ef ciencies by
reducing duplicative efforts

@JohnRofrano 19
fi
fi
 DevOps

Things DevOps is Not

• DevOps is not simply combining

Development & Operations teams
• DevOps is not a separate team
• DevOps is not a tool
• DevOps is not a one-size-fits-all strategy
• DevOps is not just automation

@JohnRofrano 20
 DevOps

What's so hard about getting Dev and

Ops to work together?

@JohnRofrano 21
 DevOps

Dev verses Ops

It's not my
It's not my
code, it's your
machines, it's
machines
your code

vs

Little bit weird Pulls levers & turns knobs

Sits closer to the boss Easily excited
Thinks too hard Yells a lot in emergencies
@JohnRofrano 22
 DevOps

To fully appreciate where we are...

We must first understand from where
we came

@JohnRofrano 23
 BRIEF
HISTORY OF
DEVOPS

@JohnRofrano 24
 DevOps

Pre-DevOps History Review

• Architects worked for months designing the system on

paper and then released the documents to development
• Development worked for months on features in an
isolated development environment and then released the
code to testing
• Testing opened defects and sent the code back to
development until no more sev 1 or 2’s
• At some point, development then released the code to
operations for deployment
• The operations team took forever to deploy the
application and then kept it running from that point on
@JohnRofrano 25
 DevOps

Traditional Waterfall Development

• Each step ended when the next begins
Requirements
• Mistakes found in the later stages are
Design more expensive to x
Code
Integration
• No provisions for changing
requirements Test

Deploy
• No idea if it works until the end
@JohnRofrano 26
fi
 DevOps

Problems with this approach?

• There was usually a long time between

software releases
• Because all of the teams worked
separately, the development team was
not always aware of operational
roadblocks that might prevent the
program from working as anticipated
• The people the furthest from the code
who knew the least about it were
deploying it into production
@JohnRofrano 27
 DevOps

Extreme Programming (XP)

• In 1996 Kent Beck introduced Extreme

Programming
• Based on an interactive approach to
software development
• Intended to improve software quality
and responsiveness to changing
customer requirements
• Was one of the rst agile methods

@JohnRofrano 28
fi
 DevOps

Agile Manifesto

• In 2001, seventeen software developers

met at a resort in Snowbird, Utah to
discuss these lightweight development
methods
• Including among others Kent Beck,
Ward Cunningham, Dave Thomas, Jeff
Sutherland, Ken Schwaber, Jim
Highsmith, Alistair Cockburn, and Bob
Martin
• Together they published the Manifesto
for Agile Software Development
@JohnRofrano 29
 DevOps

Agile Development

• Requirements and solutions evolve

through the collaborative effort of self-
organizing and cross-functional teams
and their customers
• It advocates adaptive planning,
evolutionary development, early delivery,
and continual improvement
• It encourages rapid and exible
response to change

@JohnRofrano 30
fl
 DevOps

...but we're already Agile, right?

@JohnRofrano 31
 DevOps

Dysfunctional
DevOps

@JohnRofrano 32
 DevOps

Dysfunctional
DevOps

@JohnRofrano 33
 DevOps

Dysfunctional
DevOps

@JohnRofrano 34
 DevOps

Dysfunctional Water
DevOps

Organizations that
Scrum
wants to change
without actually
changing anything!

Fall

2011 David West of Forester

@JohnRofrano 35 https://www.verheulconsultants.nl/water-scrum-fall_Forrester.pdf
 DevOps

The Agile Dilemma

• While Agile improved the speed and

accuracy of software for developers I want I want
stability!
change!
• It did nothing for operations
• Many development teams just got
frustrated by ops not being able to
deliver at the speed of development

@JohnRofrano 36 Source: Andrew Clay Shafer, Senior Director of Technology at Pivotal

 DevOps

Why isn’t Agile alone good enough?

• While Agile improved the speed and accuracy of software for developers, it did nothing for
operations

@JohnRofrano 37 https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile
 DevOps

The results of 2 speed IT

• Many development teams just got frustrated by ops not being able to deliver at the
speed of development

@JohnRofrano Cloud 38
 DevOps

This is how "Shadow IT" is born

@JohnRofrano 39
 DevOps

...enter DevOps

@JohnRofrano 40
 DevOps

2007 Patrick Debois

• Recognized Dev and Ops worked ineffectively
DevOps and not together
Started in
2007
2008 Andrew Clay Shafer
• Agile 2008 Conference BoF "Agile
Infrastructure"

2009 John Allspaw

• Velocity 2009 “10+ Deploys Per Day: Dev
and Ops Cooperation at Flickr”

@JohnRofrano 41
 DevOps

DevOpsDays 2009

• Patrick Debois (often called the Father of

DevOps) created the rst DevOpsDays
conference in Ghent, Belgium in October
2009
• It was “The conference that brings
development and operations together”
- This is where the term DevOps was rst
used
• DevOpsDays is now a local conference
held internationally several times a year in
different cities
https://www.devopsdays.org
@JohnRofrano 42
fi
fi
 DevOpsDays
2019
40 Events in 21
countries are
scheduled for
2019
(10 years later)

Bridget Kromhout
(Lead 2015-2020)
@JohnRofrano 43
 DevOps

2011 Continuous Delivery

• Jez Humble and David Farley write this

groundbreaking book that sets out the
principles and technical practices that enable
rapid, incremental delivery of high quality,
valuable new functionality to users
• Through automation of the build, deployment,
and testing process, and improved
collaboration between developers, testers, and
operations, delivery teams can get changes
released in a matter of hours —sometimes
even minutes– no matter what the size of a
project or the complexity of its code base

@JohnRofrano 44
 DevOps

2015 The Phoenix Project

• Gene Kim, Kevin Behr, and George

Spafford write this seminal book on
running Lean Operations
• Based on The Goal: A Process of
Ongoing Improvement by Eliyahu M.
Goldratt
• Updated from a Lean Manufacturing
story to Software Development and
Delivery

@JohnRofrano 45
 DevOps

The Three Ways

The Phoenix Project - Gene Kim

The First Way helps us understand how to create fast

flow of work as it moves from Development into IT
Operations, because that’s what’s between the
business and the customer.

The Second Way shows us how to shorten and

amplify feedback loops, so we can fix quality at the
source and avoid rework.

And the Third Way shows us how to create a culture

that simultaneously fosters experimentation, learning
from failure, and understanding that repetition and
practice are the prerequisites to mastery.
@JohnRofrano 46 https://itrevolution.com/the-three-ways-principles-underpinning-devops/
 DevOps

State of DevOps Report

• Puppet Labs and DORA

began measuring
DevOps success in 2015
lead by Dr. Nicole
Forsgren
• We have seen steady
increase in companies
adopting DevOps since
then

@JohnRofrano 47
 DevOps

Current DevOps Impact

State of DevOps Report by Puppet Labs 2017

• Taking an experimental approach to

product development can improve your
IT and organizational performance
• High-performing organizations are
decisively outperforming their lower-
performing peers in terms of throughput
• Undertaking a technology
transformation initiative can produce
sizeable cost savings for any
organization
@JohnRofrano https://puppet.com/resources/whitepaper/state-of-devops-report 48
 DevOps

2016 The DevOps Handbook

• Gene Kim, Jez Humble, Patrick Debois,

and John Willis write this follow-on to
The Phoenix Project
• Serves as a practical guide on how to
implement the concepts introduced in
The Phoenix Project
• John Willis worked at Docker and Chef
and is a DevOpsDays coordinator after
being at the original in Ghent 2009

@JohnRofrano 49
 DevOps

Major
Influencers
of the early
DevOps Patrick Debois Andrew Clay Shafer John Allspaw Jez Humble

Movement
(10 years later)

Gene Kim John Willis Bridget Kromhout Nicole Forsgren

@JohnRofrano 50
 DevOps

Why is the history important?

It reminds us that DevOps is...

• from the practitioners, by
practitioners
• not a product, specification, job title
• an experience-based movement
• decentralized and open to all

Damon Edwards (https://www.youtube.com/watch?v=o7-IuYS0iSE)

@JohnRofrano 51
 DevOps

What is the Goal?

@JohnRofrano 52
 DevOps

Agility is the goal

• Smart experimentation
• Moving in‐market with
maximum velocity and
minimum risk
• Gaining quick valuable insight
to continuously change the
value proposition and quality

@JohnRofrano 53
 DevOps

Agility: The Three Pillars

Cultural Change
Loose Coupling/Binding
Automated Pipeline
RESTful APIs
Everything as Code
Designed to resist failures
Immutable Infrastructure
Test by break / fail fast
DevOps Microservices

AGILITY

Containers Portability
Developer Centric
Ecosystem enabler
Fast startup
@JohnRofrano 54
 The Perfect Storm
DevOps for speed and agility
Microservices for small deployments
Containers for ephemeral runtimes

@JohnRofrano 55
 DevOps

“DevOps starts with learning how to work differently. It

embraces cross-functional teams with openness,
transparency, and respect as pillars.”

–Tony Stafford, Shadow Soft

@JohnRofrano 56
 DevOps

Application Evolution

Delivery Waterfall Agile DevOps

Service Oriented
Architecture Monoliths Architecture
Microservices

Virtual
Infrastructure Physical Servers
Machines
Containers

@JohnRofrano 57
 DevOps

“Culture is the #1 success factor in DevOps. Building a

culture of shared responsibility, transparency and faster
feedback is the foundation of every high performing
DevOps team.”
–Atlassian

@JohnRofrano 58
 DevOps

DevOps has Three Dimensions

1.Culture
2.Methods
3.Tools

@JohnRofrano 59
 DevOps

“While tools and methods are important

… it’s the culture that has the biggest impact”

@JohnRofrano 60
 DevOps

How do you change a culture?

@JohnRofrano 61
 DevOps

@JohnRofrano 62 https://www.quora.com/What-is-the-importance-of-culture
 DevOps

How do you change a culture?

• You must change the way people think

• You must change the way people work
• You must change the way people are organized
• You must change the way people are measured

@JohnRofrano 63
 DevOps

You must change the

way people think

@JohnRofrano 64
 DevOps

DevOps Thinking

• Social Coding
• Behavior and Test Driven Development
• Working in small batches*
• Build Minimum Viable Products for gaining insights*
• Failure leads to understanding

* from Lean Manufacturing

@JohnRofrano 65
 DevOps

Traditional Thinking

• In the past developers worked on private repositories and you

had to be a member of the team to contribute

• You see a project that is 80% of what you need but there are
some missing features but…

• You feel that if you make a feature request of the project owner,
your request will be rejected or go to the bottom of their priorities

• So you rebuild 100% of what you need so as not to have a

dependency on another project

@JohnRofrano 66
 DevOps

Think Social Coding

• Repositories are public and everyone is encouraged to Fork the

code and contribute to the project

• Discuss the new feature with the repo owner and agree to
develop it

• Open an Issue and assign it to yourself so that everyone knows

what you are working on

• Fork the code, create a branch, and make your changes

• Issue a Pull Request when you are ready to review and merge
your work back into the main project
@JohnRofrano 67
 DevOps

Think Minimum Viable Product (MVP)

• MVP is NOT the result of "Phase 1" of a

project
• It IS the cheapest/easiest thing you can
build to start testing your value
hypothesis and learning
• The former focuses on delivery, while the
latter focuses on learning
• At the end of each MVP you decide
whether to pivot or persevere

@JohnRofrano 68
 DevOps

Minimum Viable Product Example

https://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp
@JohnRofrano 69
69
 DevOps

“To fully leverage DevOps,

you need to think differently about application
design.”

@JohnRofrano 70
 DevOps

Think Cloud Native

• The Twelve-Factor App describes patterns for cloud-

native architectures which leverage microservices
• Applications are design as a collection of stateless
microservices
• State is maintained in separate databases and
persistent object stores
• Resilience and horizontal scaling is achieved through
deploying multiple instances
• Failing instances are killed and re-spawned, not
debugged and patched (cattle not pets)
• DevOps pipelines help manage continuous delivery of
services
@JohnRofrano 71 https://www.nginx.com/blog/introduction-to-microservices/
 DevOps

Think Microservices
"…the microservice architectural style is an
approach to developing a single application as
a suite of small services, each running in
its own process and communicating with
lightweight mechanisms, often an HTTP
resource API. These services are built around
business capabilities and independently
deployable by fully automated deployment
machinery."

- Martin Fowler and James Lewis

@JohnRofrano
72 © 2018 IBM Corporation 27 February 2019 IBM Services
 DevOps

Monolith vs Microservices Architecture

Monolith Microservices
• Agility
• Scalability
• Resilience

A microservice is a granular, decoupled component

within a broader application
@JohnRofrano 73
 DevOps

“Since failure is inevitable, architect your

software to be resilient and to scale
horizontally.”

@JohnRofrano 74
 DevOps

Design for Failure

• Embrace failures - they will happen!

• Move from: How to avoid —> How to identify & what to do about it
• Move from: Pure operational concern —> developer concern
• External calls to other services that you don’t control are especially prone to problems
• Use separate thread pools
• Time out quickly
• Circuit breaker pattern – identify problem and do something about it to avoid
cascading failures
• Bulkhead pattern - Isolation from start to limit scope of failure (separate thread pools)
• Monkey testing – test by breaking (yes, on purpose! see: Netflix Chaos Monkey and
Simian Army)

@JohnRofrano 75
 DevOps

Think Continuous Automation

• Continuous Integration (CI)

• Continuous Delivery (CD)
• Build Automation
• Canary Rollouts / Blue-Green
• Failing Forward
• Application Release Automation

@JohnRofrano 76
 DevOps

Automation is not just about

speed... it's about repeatability.

@JohnRofrano 77
 DevOps

Case Study: Knight Capital

• The Knight Capital Group was an American

global financial services firm engaging in market
making, electronic execution, and institutional
sales and trading

• With its high-frequency trading algorithms

Knight was:

• The largest trader in U.S. equities

• with a market share of 17.3% on NYSE

• and 16.9% on NASDAQ

@JohnRofrano 78
 DevOps

Case Study: Knight Capital

(continued)

• Upon deployment, the new Retail Liquidity Program (RLP) code in SMARS was
intended to replace unused code in the relevant portion of the order router.
• This unused code previously had been used for functionality called “Power Peg,”
which Knight had discontinued using many years earlier.
• Despite the lack of use, the Power Peg functionality remained present and callable
at the time of the RLP deployment.
• The new RLP code also repurposed a flag that was formerly used to activate the
Power Peg code.
• Knight intended to delete the Power Peg code so that when this flag was set to
“yes,” the new RLP functionality—rather than Power Peg—would be engaged.
@JohnRofrano 79
 DevOps

Knight
Capital’s
$460
Million
Release
Failure

@JohnRofrano 80
 DevOps

Think Continuous Improvement

Actually it’s a continuous pipeline that exists in a giant feedback loop

@JohnRofrano 81
 DevOps

DevOps Pipeline Automation

Plan Develop Build Test Deploy

• Use tools like Jira or ZenHub that promote Agile planning

• Use GitHub for Code, Issues, and Pull Requests
• Use tools like GitHub Actions, Travis CI, Jenkins, or Continuous Delivery for Continuous Build
• Automate all testing, promote only if successful
• Deploy to Cloud dev/test/production with a repeatable automated process
@JohnRofrano 82
 DevOps

The value of the DevOps pipeline

• Everything from source, including intermediate artifacts

• If it's not in GitHub it's not part of the pipeline
• Automated testing
• Without automated tests you cannot have a pipeline
• Infrastructure as code (not just the application)
• Better consistency between environments
• Automated release
• No more manual changes

@JohnRofrano 83
 DevOps

“Being able to recover quickly from failure is more

important than having failures less often.”
–John Allspaw, CTO at Etsy

@JohnRofrano 84 http://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/
 DevOps

How do you change a culture?

@JohnRofrano 85
 DevOps

You must change the

way people work

@JohnRofrano 86
 DevOps

Working DevOps

• Facilitate a culture of teaming and collaboration

• Establish agile development as a shared
discipline
• Automate relentlessly to enable rapid DevOps
response
• Push smaller releases faster, measure and
remediate impact

@JohnRofrano 87
 DevOps

We have worked
the same way
since the industrial
revolution

@JohnRofrano
88 © 2018 IBM Corporation 27 February 2019 IBM Services
 DevOps

Taylorism

Named after the US industrial engineer Frederick Winslow Taylor (1856-1915) who in
his 1911 book 'Principles Of Scientific Management' laid down the fundamental
principles of large-scale manufacturing through assembly-line factories.

• Adoption of Command and Control Management

• The dominant method of management in the Western world
• Organizations divided into (ostensibly) independent functional silos
• Workers are separated into task specific roles for greater efficiency
• Decision-making is separated from work
• Managers do the planning and decide what workers should do
• Workers mindlessly do the tasks they are ask to accomplish
@JohnRofrano 89
 DevOps

“It doesn’t make sense to hire smart people and

tell them what to do; we hire smart people so they
can tell us what to do.”
–Steve Jobs

@JohnRofrano 90
 DevOps

Impact of Taylorism on Information Technology

Optimized roles may work great making cars (assembly line), not so good for
software development (yields waterfall / silos)
hand-off hand-off hand-off hand-off
hand-off

Project
Management Architects Developers Testers Operations Security
(Planners)

@JohnRofrano 91
 Organizational Impact of DevOps

Software
Development is
NOT like
Assembling
Automobiles

In software
development, most of
the parts don't even
exist yet

1928 Ford Rouge Complex Model A assembly line negative from the Ford
Motor Company archives.

@JohnRofrano 92
 DevOps

DevOps represents the

reincarnation of "craft work"
...because software development is always bespoke

@JohnRofrano 93
 DevOps

Taylorism is not appropriate for Craft Work

• Taylorism may have been good during the industrial
revolution, but not so much in the technology revolution

• Automation has resolved many problems which Taylor

sought to address

• The people power requirements have been lowered

• Taylorism is not appropriate for "knowledge" work like

software development

@JohnRofrano 94
 Organizational Impact of DevOps

Software
Engineering
vs
Civil
Engineering

@JohnRofrano 95
 Organizational Impact of DevOps

The project model is fundamentally flawed

• Software development efforts are usually run as if they were civil

engineering projects

• When a project is complete, the system gets tossed over the wall to
operations to run and maintain as part of a "business as usual" effort

• All the people in the project team get reallocated to new work

• The project model is fundamentally flawed as a way of doing

software development

• Software development should be treated as product development

instead
https://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/

@JohnRofrano 96
 DevOps

The Kobayashi Maru

...the no-win scenario

• Development wants Innovation

• Keep up users’ changing needs by I want I want

change stability
developing and deploying new and
enhanced capabilities.

WALL OF CONFUSION
• Operations want Stability

• Ensure that users can use those

services and applications and that their
data and information is kept safe. Development Operations

Source: Andrew Clay Shafer, Senior Director of Technology at Pivotal

@JohnRofrano 97
 DevOps

Diametrically Opposed Views

• Enterprises are built around end-to-end processes that

see “new” as complex, high risk, expensive and time
consuming

• To be delivered as a one-time project

• DevOps seeks to turn this on its head

• DevOps delivers a continual series of small changes

• These cannot survive traditional overheads

@JohnRofrano 98
 DevOps

Operations View of Development

It's not my
machines, it's
your code
• Development teams throw dead cats
over the wall
• Manually implemented changes
• Lack of back-out plans
• Lack of testing
• Environments that don’t look like
Production
@JohnRofrano 99
 DevOps

Development View of Operations

It's not my
code, it's your
machines
• All-or-nothing changes
• Change windows in the dead of night
• Implemented by people furthest away
from the application
• Using cut-and-paste from Word
documents called “run-books”

@JohnRofrano 100
 DevOps

IT Silos
Breed
Apathy

@JohnRofrano 101
 DevOps

“If the web site works, the developers get the praise

If the web site is down, operations gets the blame!”

…did I mention the Kobayashi Maru?

@JohnRofrano 102
 DevOps

Required DevOps behaviors

Shared ownership and high

Organizational silos and hand-o s
collaboration
Risk management by embracing
Fear of change
change

Build once, hand-crafted

Ephemeral infrastructure as code
"snow akes"

Manual ful llment Automated self-service

Feedback loops and data-driven

Alarms, callbacks, and escalations
responses
@JohnRofrano 103
fl
fi
ff
 DevOps

How DevOps Manages Risk

DevOps manages risk through increasing the rate of change rather than aversion to it

• Deployment is king
• Deployment must be painless
• You have deployed the same thing several times before it gets
to production
• Deployment is decoupled from activation
• Risk is managed via activation controls (blue-green deploys
with canary testing)
• 10% of the user base, waves of activation, etc.
• Deployment is not “one size fits all”
• It is a rail yard of interconnecting steps and microprocesses
@JohnRofrano 104
 DevOps

Servers are Cattle not Pets

• Pets are given names like pussinboots.cern.sh

• They are unique, lovingly hand raised and cared for
• When they get ill, you nurse them back to health

• Cattle are given numbers like vm0042.cern.ch

• They are almost identical to other cattle
• When they get ill, you get another one
@JohnRofrano 105
 DevOps

Ephemeral Infrastructure
The Cloud has enabled new ways of managing server builds, software releases, and server entropy

• Cattle, not Pets

• Servers are built on demand via automation
• Logging into the server is seen as failure

• Release through parallel infrastructure

• Build the new version on new infrastructure; stage transition between
environments (zero downtime)
• Transient Infrastructure
• Throw away when it is no longer needed
• This eliminates entropy - a major source of failure

@JohnRofrano 106
 DevOps

Immutable Delivery

• Applications are packaged in containers

• Same container that developer runs on
their laptop runs in production
• Rolling updates with immediate roll-back
• No variance limits side-effects
• Dependencies are contained

@JohnRofrano 107
 DevOps

Zero-Downtime Deployments

• Blue-green deployment is a zero-downtime deployment technique that consists of two

nearly identical production environments, called Blue and Green.

• They differ by the artifacts that the developer has intentionally changed, typically by the
version of the application. At any given time, at least one of the environments is active.

• Using the blue-green deployment technique, you can realize the following benefits:

• Take software quickly from the final stage of testing to live production.

• Deploy a new version of an application without disrupting traffic to the application.

• Rollback rapidly. If there is something wrong with one of your environments, you can
quickly switch to the other environment.
@JohnRofrano 108
 DevOps

“DevOps is about breaking down the silos and

working as a Single Agile Team.”

@JohnRofrano 109
 DevOps

Everyone Working Agile

Agile Manifesto
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right,
@JohnRofrano
we value the items 110
on the left more.
 Organizational Impact of DevOps

How Agile are your teams?

@JohnRofrano 111
 DevOps

Working as an Agile Team

• Iterative Sprints
• Groomed Backlogs
• Customer Stories
• 2 Week Deliverables

@JohnRofrano 112
 DevOps

How do you change a culture?

@JohnRofrano 113
 DevOps

You must change the way

people are organized

@JohnRofrano 114
 DevOps

Conway’s Law

Any organization that designs a

system (defined broadly) will produce
a design whose structure is a copy of
the organization's communication
structure
- Melvin Conway, Datamation, 1968

e.g., if you ask an organization with 4 teams to write a

compiler… you will get a 4-pass compiler!
@JohnRofrano 115 http://www.melconway.com/Home/Conways_Law.html
 DevOps

Traditional Organization around Technology (Bad)

Organization Structure Application Structure

Web Tier

User Interface Team

App Tier

Application Team

Database

Database (DBA) Team

@JohnRofrano
116
 DevOps

Organization around Business Domains (Good)

Login Inventory
Registration Personalization Shipping
Users Receiving

Account Team Personalization Team Warehouse Team

@JohnRofrano 117
 DevOps

“If you are not going to organize around

business domains, you are not going to get
the full benefit of DevOps”

@JohnRofrano 118
 Spotify Engineering Culture

https://www.youtube.com/watch?v=R2o-Xm3UVjs
@JohnRofrano 119
 DevOps

Engineering Culture
Rules are a good start, then break them when needed

Agile > Scrum

Principles > Practices
Servant Leaders > Process Master
Community > Structure
Enable > Serve
Trust > Control
Failure Recovery > Failure Avoidance
Impact > Velocity
Innovation > Predictability
Value Delivery > Plan Fulfillment
Chaos > Bureaucracy
@JohnRofrano 120
 DevOps

Organizational Structure
• Squads are grouped into Tribes
(light-weight matrix)
• Chapters of competency areas are
formed across Squads
• Guilds are informal light-weight
community of interests across the
company

@JohnRofrano 121
 DevOps

Autonomous Squads
Loosely coupled, Tightly aligned squads

• Each Squad has it’s own mission aligned with the business
• Feels like a "mini-startup"
• Self Organizing / Cross-functional
• 5-7 engineers, less than 10
• Squads have end-to-end responsibility for what they build
• Build, commit, deploy, maintenance, operations, EVERYTHING!!!
• With a long term mission usually around a single business domain

@JohnRofrano 122
 DevOps

There's No Such Thing as a

"DevOps Team"
That’s an anti-pattern

@JohnRofrano 123
 DevOps
Select your job function:
From the web site of a major player who should know better
DevOps
Identity
Problem

https://johnrofrano.medium.com/devops-engineer-is-the-new-sysadmin-5bc46b86d413
@JohnRofrano 124
 DevOps

Different Perspectives of DevOps

Large Enterprises
Dev DevOps Ops
Cloud Native Startups

DevOps is something Ops does

Dev Ops Dev Ops

Dev DevOps Ops

Traditional DevOps is the company

Development & DevOps is something Devs do culture
Operations

Dev DevOps Ops

DevOps is a separate team

@JohnRofrano 125
 DevOps

“The DevOps movement addresses the dysfunction that results

from organizations composed of functional silos. Thus, creating
another functional silo that sits between dev and ops is clearly
a poor (and ironic) way to try and solve these problems.”
– Jez Humble, The DevOps Handbook

https://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/
@JohnRofrano 126
 DevOps

Creating a DevOps team makes it worse

I want a new
job!
I want I want
change! stability!

WALL OF CONFUSION
DevOps Operations
Development
Team Team Team
@JohnRofrano 127
 DevOps

DevOps
Organizational
Objective
Shared Consciousness

…with

Distributed (local) Control

@JohnRofrano
128 © 2018 IBM Corporation 27 February 2019 IBM Services
 DevOps

“Bad behavior arises when you abstract people away from

the consequences of their actions.”

– Jez Humble

https://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/

@JohnRofrano 129
 DevOps

Functional Silos Breed Bad Behavior

• Bad behavior arises when you

abstract people away from the
consequences of their actions.

• Functional silos abstract people

away from the consequences of their
actions.

• For example: By adding a QA Team,

developers are abstracted away
from the consequences of writing
buggy code.

@JohnRofrano 130
 DevOps

Actions have Consequences

• Make people aware of the consequences of their actions

• Create cross-functional teams - or -
• Have developers rotate through operations teams
• Have operations people attend developer standups and
showcases
• Make people responsible for the consequences of their
actions
• Having developers on Pager Duty, or own the SLA for the
products and services they build
https://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/

@JohnRofrano 131
 DevOps

“You build it, you run it!”

–Werner Vogels, CTO of Amazon

https://aws.amazon.com/blogs/enterprise-strategy/enterprise-devops-why-you-should-run-what-you-build/

@JohnRofrano 132
 DevOps

How do you change a culture?

@JohnRofrano 133
 DevOps

You must change the way

people are measured

@JohnRofrano 134
 DevOps

On the folly of rewarding for A, while hoping for B

“Whether dealings with monkeys, rats, or human beings, it is

hardly controversial to state that most organisms seek
information concerning what activities are rewarded, and
then seek to do (or at least pretend to do) those things, often
to the exclusion of activities not rewarded. The extent to
which this occurs of course will depend on the perceived
attractiveness of the rewards offered, but neither operant nor
expectancy theorists would quarrel with the essence of this
notion.”
–Steven Kerr, Ohio State University
Academy of Management Journal, Dec 1975
@JohnRofrano 135
 DevOps

You get what you measure

@JohnRofrano 136
 DevOps

Measure what Matters

…because you get what you Measure

• If you measure widget production you will get lots of widgets

• If you measure lines of code you will get many lines of code
• If you measure people against each other by ranking them,
you will get anti-social behavior
• But if you measure developers by their social interaction and
sharing… you will get Social Coders!

@JohnRofrano 137
 DevOps

DevOps Changes the Objective

• Old School is focused on Mean Time To

Failure (MTTF)

- Make sure you never go down

• DevOps is focused on Mean Time To

Recovery (MTTR)

- You will go down, make sure you can

recover quickly!

@JohnRofrano 138
 DevOps

DevOps Metrics

• A BASELINE provides a concrete number for comparison as you implement your DevOps
changes:
• It currently requires six team members 10 hours to deploy a new release of our
product.
• This costs us $X for every release
• Metric GOALS allow you to reason about these numbers and judge the success of your
transition process:
• Reduce deployment time from 10 hours to 2 hours.
• Increase percentage of defects detected in testing from 25% to 50%

@JohnRofrano 139
 DevOps

Vanity Metrics
…good for feeling awesome, bad for action

• Consider the total number of daily “hits” to your website is 10,000

• Now what? (what does a "hit" represent?)
• Do you really know what actions you took in the past that drove those
visitors to you?
• Do you really know which actions to take next?
• In most cases, I don’t think it’s very helpful

@JohnRofrano 140
 DevOps

Actionable Metrics

• Imagine you add a new feature to your website, and you do it using an A/B split-test in
which 50% of customers see the new feature and the other 50% don’t.
• A few days later, you take a look at the revenue you’ve earned from each set of
customers, noticing that group B has 20% higher revenue per-customer.
• Think of all the decisions you can make:
• Obviously, roll out the feature to 100% of your customers
• Continue to experiment with more features like this one and …
• Realize that you’ve probably learned something that’s particular valuable to your
customers.
http://tim.blog/2009/05/19/vanity-metrics-vs-actionable-metrics/
@JohnRofrano 141
 DevOps

Actionable Metric Examples

• Reduce time-to-market for new features.

• Increase overall availability of the product.
• Reduce the time it takes to deploy a software release.
• Increase the percentage of defects detected in testing before production
release.
• Make more ef cient use of hardware infrastructure.
• Provide performance and user feedback to the product manager in a more
timely manner.
http://tim.blog/2009/05/19/vanity-metrics-vs-actionable-metrics/
@JohnRofrano 142
fi
 DevOps

Top 4 Actionable Metric

1. Mean Lead Time

• How long does it take from idea to production?
2. Release Frequency
• How often can you deliver changes?
3. Change Failure Rate
• How often to changes fail?
4. Mean Time to Recovery (MTTR)
• How quickly can you recover from failure?
Nicole Forsgren - AWS re:Invent 2017: Tools Won’t Fix Your Broken DevOps
https://www.youtube.com/watch?v=gsjCWrCUjNg

@JohnRofrano 143
 DevOps

Culture Measurements
Measured on a Likert scale of Strongly Agree to Strongly Disagree

• On my team information is actively sought

• On my team failures are learning opportunities and messengers of them are not
punished

• On my team responsibilities are shared

• On my team cross functional collaboration is encouraged and rewarded

• On my team failure causes inquiry

• On my team new ideas are welcomed

Nicole Forsgren, CEO & Chief Scientist at DevOps Research and Assessment (DORA)
@JohnRofrano 144
 DevOps

Keys to High Performance

You MUST do all three!

1.Technical Practices
2.Lean Processes (Agile)
3.Culture

@JohnRofrano 145
 DevOps

“Measurements should encourage innovation

and collaboration, and not punish failure”

@JohnRofrano 146
 DevOps

Busted DevOps Myths

• You cannot buy DevOps In-A-Box

• You cannot order 20 units of DevOps for this quarter
• You cannot sprinkle DevOps on something to make it better
• You cannot become DevOps without changing your culture
- You can’t change your companies culture just by adopting
new tools …but they can help reinforce it
• Using Containers won't fix your broken culture
• You cannot maintain your current organizational structure and
become DevOps

@JohnRofrano 147
 DevOps

The Entire Organization Must Change

Studies show, that in hierarchical and bureaucratic organizations

trying to adopt DevOps on a team level without influencing the
environment which supports the change (executive level), people are
frequently taught by agile coaches about courage, focus,
commitment, respect and openness.
However, the system they work in very often promotes
politics, blame games and escalation paths, which are in
direct contradiction to the aforementioned values.

https://www.infoq.com/articles/devops-organizational-culture
@JohnRofrano 148
 DevOps

“Don't fear failure. Fear being in the exact

same place next year as you are today.”

@JohnRofrano 149
 DevOps

Technology is
the easy part

@JohnRofrano Source: CEB analysis, 2018 Gartner,

150 Inc and/or affiliates. All rights reserved CIO181509
 DevOps

DevOps Summary

• A Cultural Movement

• Emphasizing Collaboration, Sharing, and

Transparency

• Promoting Automation and Infrastructure as

Code

• Achieving Continuous Integration and

Delivery of Changes

• Immutable Delivery

• With One set of Metrics to rule them all

@JohnRofrano 151
 DevOps

DevOps Maturity Matrix

Cloud Computing: The Cloud and DevOps by Dave Linthicum

Maturity
People Process Technology
Level
• Silo based
• Manual processes • Manual builds and deployments
Level 1 • Blame and finger-pointing
• Tribal knowledge the norm • Manual testing
Ad Hoc • Dependent on experts
• Unpredictable and reactive • Environmental inconsistencies
• Lack of accountability
• Processes established within silos • Automated builds
Level 2 • Managed communications • No standards • Automated tests written as part of story
Repeatable • Limited knowledge sharing • Can repeat what is known, but can’t development
react to unknowns • Painful but repeatable releases
• Automated build and test cycle for every
• Collaboration exists • Process automated across the
Level 3 commit
• Shared decision making software life cycle
Defined • Push button deployments
• Shared accountability • Standards across organization
• Automated user and acceptance testing
• Build metrics visible and acted on
• Collaboration based on • Proactive monitoring
• Orchestrated deployments with automatic
Level 4 shared metrics with a • Metrics collected and analyzed against
rollbacks
Measured focus on removing business goals
• Nonfunctional requirements defined and
bottlenecks • Visibility and predictability
measured
• A culture of continuous • Self-service automation • Zero downtime deployments
Level 5
improvement permeates • Risk and cost optimization • Immutable infrastructure
Optimized
through the organization • High degree of experimentation • Actively enforce resiliency by forcing failures
@JohnRofrano 152
 DevOps

Key Takeaways

• DevOps is about breaking down the silos and working

as a Single Agile Team
• Culture is the #1 success factor in DevOps. Building a
culture of shared responsibility, transparency and faster
feedback is the foundation of every high performing
DevOps team
• DevOps starts with learning how to work differently. It
embraces cross-functional teams with openness,
transparency, and respect as pillars
• Being able to recover quickly from failure is more
important than having failures less often
• Measurements should encourage innovation and
collaboration, and not punish failure (blameless culture)
@JohnRofrano 153
 DevOps

“If you want to build a ship, don’t drum up the men to

gather wood, divide the work, and give orders.
Instead, teach them to yearn for the vast and endless sea.”

– Antoine de Saint-Exupery*

* Author of The Little Prince

@JohnRofrano 154
 DevOps

We don’t “Do" DevOps

@JohnRofrano 155
 DevOps

We become DevOps

@JohnRofrano 156