2013 State of

DevOps Report
PRESENTED BY

&
Key Findings
In December of 2012, Puppet Labs and IT Revolution
Press surveyed over 4,000 IT Operations professionals ORGANIZATIONS THAT HAVE IMPLEMENTED
DEVOPS SAW THESE BENEFITS:
and developers in the largest industry survey of this mag-
nitude. The survey results revealed accelerating adoption IMPROVED QUALITY OF
SOFTWARE DEPLOYMENTS 63%
of DevOps practices in IT organizations across companies
of all sizes. Sixty-three percent of respondents have MORE FREQUENT
SOFTWARE RELEASES 63%
implemented DevOps practices, a 26 percent increase in
adoption rate since 2011.1 IMPROVED VISIBILITY INTO IT
PROCESS AND REQUIREMENTS 61%
Respondents from organizations that implemented
CULTURAL CHANGE
DevOps reported improved software deployment quality COLLABORATION/ COOPERATION 55%
and more frequent software releases. Our findings sub-
MORE RESPONSIVENESS
stantiate the business value of DevOps, too: It enables TO BUSINESS NEEDS 55%
high performance by increasing agility and reliability. We
MORE AGILE
found that high performing organizations: DEVELOPMENT 51%
Ship code 30x faster MORE AGILE CHANGE
and complete those deployments 8,000 times faster than their MANAGEMENT PROCESS 45%
peers.
IMPROVED
Have 50% fewer failures QUALITY OF CODE 38%
and restore service 12 times faster than their peers.

HIGH PERFORMANCE BY DEVOPS MATURITY

Not surprisingly, organizations that have implemented
DevOps practices are up to five times more likely to be
52%
high-performing than those that have not. In fact, the
42%
longer organizations have been using DevOps practices,
21%
the better their performance: The best are getting better. 9%

Not Currently Implemented Implemented

DevOps adoption is at a tipping point. Adopting DevOps Implementing Implementing <12 Months >12 Months

practices improves performance, and the longer those

practices are in place, the greater the performance advan-
tage. Waiting to implement is no longer an option.
Following our recommendations will put you ahead of
the curve as DevOps approaches critical mass.

1
Respondents of both the 2011 and 2012 survey are likely to be early adopters, making the increase in DevOps adoption
rate much more telling of the overall growth of DevOps during this time period.
Who Took The Survey?
We saw respondents from over 90 countries Respondents worked for organizations running
across organizations of all sizes. The majority of the gamut from startups to small/medium enter-
respondents identified themselves as administra- prises to WebOps giants, revealing that DevOps is
tors, engineers, or developers, with only 16 per- happening everywhere.
cent manager-level or above.

SURVEY AREA

90+
COUNTRIES

COMPANY SIZE (# OF EMPLOYEES) DEPARTMENTS

IT OPERATIONS DEVELOPMENT/ ENGINEERING QA / OTHER

20%
16% 70% 23% 7%
1-19
ROLES
20-99

 
100-499

16% 500-9999 26%

75% 17%
10000+

22%
ADMINS/
ENGINEERS
MANAGERS/
C- LEVEL  8%
CONSULTANTS
Key Findings
DevOps Increases Agility and Reliability
IT organizations are expected to respond more quickly to
urgent business needs while simultaneously providing
stable, secure, and predictable IT service. However, the
systems on which the business operates are typically
fragile and hostile to change. Adopting Agile develop-
ment processes without improving operational reliability
or communication between developers and operations
only makes this problem worse. The increased frequency Rob Cornish, CTO at the International
of releases from development creates even more of a Securities Exchange (ISE), a leading
burden on an already strained IT organization. Similarly, US options exchange that exceeds
adopting rigid ITIL/ITSM standards without addressing daily trading volumes of 2.5 million
development issues and improving communication chan- contracts.
nels results in an inflexible IT organization that simply
"DevOps has been instrumental in
cannot respond to business needs quickly enough.
reducing our software deployment
DevOps picks up where Agile methodology and IT stan-
cycle from days to minutes. We are able
dards left off.
to deploy new features to our customers
For the first time ever, by adopting DevOps practices, IT quickly and safely, and rapidly respond
organizations can be simultaneously agile and reliable, to market changes."
thus fulfilling the promise of over 20 years of Agile and
ITIL/ITSM.
High performing organizations:
Deploy code 30 times more frequently.
High performing organizations deploy code 30 times more often,
and 8000 times faster than their peers, deploying multiple times
a day, versus an average of once a month. Frequent deployments
coupled with faster change lead times enable operational agility.

Have 50 percent fewer failures.

High performing organizations have double the change success
rate and restore service 12 times faster than their peers. Fewer
failures and faster recovery mean less risk to the business when
changes are deployed.
DevOps practices enable IT organizations to quickly and
safely deploy changes, freeing them to work on higher
level business objectives. In the next section, we break
out four key metrics that show the impact DevOps has on
performance.
 KeyDevOps
How Findings
Enables High Performance
A high-performing organization is characterized We found that organizations that had implement-
by its ability to ship business-critical applications ed DevOps practices were up to five times more
quickly without disrupting service. The classic likely to be high-performing than those that had
examples of this are big WebOps shops—Google, not, and the more mature the implementation,
Amazon, Twitter, and Etsy—known for deploying the higher the performance.
multiple times a day.

DEPLOY FREQUENCY CHANGE LEAD TIME

8% 14% 20% 27% 7% 10% 15% 23%

ON DEMAND LESS THAN
ONE HOUR

BETWEEN ONCE LESS THAN

A WEEK & ONCE ONE DAY
A MONTH
DEPLOY FREQUENCY

BETWEEN ONE DAY

CHANGE LEAD TIME

& ONE WEEK

BETWEEN ONCE
A MONTH & ONCE BETWEEN ONE
EVERY 6 MONTHS WEEK & ONE
MONTH
BETWEEN ONCE
A DAY & ONCE BETWEEN ONE &
A WEEK SIX MONTHS
< ONCE EVERY 6 MONTHS > 6 MONTHS
NOT CURRENTLY IMPLEMENTED IMPLEMENTED NOT CURRENTLY IMPLEMENTED IMPLEMENTED
IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS

CHANGE FAILURE RATE MEAN TIME TO RECOVER

16% 16% 22% 23% 17% 26% 35% 47%

<5% MINUTES

5–10%
MEAN TIME TO RECOVER
CHANGE FAILURE RATE

LESS THAN
ONE HOUR

10–30%

HOURS
30–50%
>50% DAYS
NOT CURRENTLY IMPLEMENTED IMPLEMENTED NOT CURRENTLY IMPLEMENTED IMPLEMENTED
IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS

We analyzed four key DevOps performance metrics—deploy frequency, change lead time, change failure
rate, and mean time to recover—by DevOps maturity, ranging from not implemented to implemented over
12 months ago. Organizations with more mature DevOps implementations saw significantly higher
performance across all metrics compared to those that had not yet implemented DevOps.
Key Agility Performance Indicators
DEPLOYMENT FREQUENCY Obamas DevOps Success
High-performing organizations deploy at least once a week,
and often multiple times a day. On average, this is 95 percent
less time between deployments than lower performing orga-
nizations, enabling these organizations to quickly respond to
market changes or customer feedback, and iterate on new
ideas.

CHANGE LEAD TIME

Being able to quickly make changes is a basic measure of agil-

ity. If your infrastructure is bogged down by technical debt, the Using DevOps practices, Obama’s
time to get code successfully running in production is signifi- tech team was able to manage 2,000
cantly longer and your changes are more likely to fail. nodes across three data centers, pro-
High-performing organizations make changes with a few min- cessing over 180TB and 8.5 billion
utes’ notice, while their peers have change lead times of up to requests, with just 30 minutes of
a month. Agile organizations can make 8,000 changes before downtime during the entire 18
their slower competitors can vet and deploy a single change. months of the election campaign.
2

Key Reliability Performance Indicators

MEAN TIME TO RECOVER

DEVOPS VALUE IN ACTION:
High-performing organizations recover from outages within
VELOCITY AT AMAZON AWS 3

minutes, compared to lower performing organizations that

take an hour or longer to recover: over 30 times faster. Most Max deployments/
hour
organizations can't afford even a few minutes of down-
time—that can mean thousands of lost transactions, resulting 10,000
in lost revenue and lower customer satisfaction.

CHANGE FAIL RATE

11.6 .001%
When changes fail, your organization has to devote time and
resources to recovering, usually by taking those resources from
Mean time Software
developing, releasing, and supporting new products or between deployments
updates. High-performing organizations have over 50 percent deployments causing an
(seconds) outage
fewer failures from code changes. Changes often fail because
the development and test environments don't match the
production environment.

2
Condliffe, Jamie, “Inside Obama’s Tech Team.” http://gizmodo.com/5960790/inside-obamas-team-tech. Nov 15, 2012.
3
Jenkins, Jon. "Velocity Culture." Keynote, Velocity Conference, Santa Clara, Jun 16, 2011.
How to Achieve High Performance
Now that we know what high performance looks like, how ORGANIZATIONS USING VERSION
do you actually achieve it? High-performing organiza- CONTROL AND AUTOMATED CODE
DEPLOYMENT ARE HIGHER PERFORMING
tions share two common practices:
VERSION CONTROL
• 89 percent use version control systems for infrastructure
management 100% .5%
6%
• 82 percent have automated their code deployments

PERFORMANCE DISTRIBUTION
These practices have a direct impact on the key perfor-
52%
mance metrics that contribute to increased organization-
al agility and reliability.

Using version control as your single source of truth gives

you the ability to pinpoint the cause of failures, easily roll
back to a known state, and quickly deploy new service
instances. Everything required to run an application, 0% NO SOME VERSION
VERSION VERSION CONTROL
including the infrastructure and configuration code, CONTROL CONTROL

should be in version control. AUTOMATED CODE DEPLOYMENT

100% .4%
Automating code deployments provides several benefits
9%
that directly contribute to high performance. First, auto-
mating the configuration of your development, test, and
PERFORMANCE DISTRIBUTION

production machines eliminates configuration drift 57%

between environments—a common point of failure in the

deployment process. Second, automation significantly
reduces your change lead time by replacing manual work-
flows with a consistent, repeatable process. Third, auto-
mation tools give visibility into the impact of changes
before they’re promoted to production, reducing risk. 0% HAVE HAVE HAVE
NOT AUTOMATED AUTOMATED
AUTOMATED SOME

Version control and automation together enable the

PERFORMANCE SEGMENT
highest levels of efficiency and productivity. Developers
and QA engineers are able to request and provision prop- HIGH MEDIUM LOW

erly configured environments in one command. This

removes Operations as a bottleneck, empowers develop-
ment/QA teams, and enables everyone to focus on things
that matter to the business.
Key Findings
Identifying Barriers to DevOps Adoption
DevOps processes and tooling contribute to high perfor- BIGGEST DIFFICULTIES IN
IMPLEMENTING DEVOPS
mance, but these practices alone aren't enough to achieve
organizational success. The most common barriers to VALUE OF DEVOPS NOT
DevOps adoption are cultural: lack of manager or team UNDERSTOOD OUTSIDE MY GROUP 48%
b u y - i n , o r t h e v a l u e o f D ev O p s i s n ' t u n d e r s t o o d NO COMMON MANAGEMENT
outside of a specific group. These barriers come from STRUCTURE BETWEEN DEV AND OPS 43%
communication breakdowns, and working to solve them
DON' T HAVE TOOLS IN PLACE 33%
may help address other commonly identified blockers.
DON' T HAVE TIME TO
How common are these cultural barriers? IMPLEMENT DEVOPS 31%
• 49 percent of respondents who had no plans to implement DON' T HAVE SUPPORT
TO BE SUCCESSFUL 19%
DevOps identified a lack of manager buy-in as a blocker—this
was the most frequently identified barrier for this group, IT' S TOO EXPENSIVE 5%
followed by "lack of team buy-in" (38%).

• 48 percent of all respondents indicated that one of the biggest THOSE WHO HAD NO PLANS TO IMPLEMENT
difficulties in implementing DevOps was that the value wasn't DEVOPS CITED THE FOLLOWING REASONS:
understood outside of their group.
LACK OF MANAGER BUY- IN 49%
The best way to overcome these barriers is to start a con-
versation. Invite someone from another team to lunch, or LACK OF TEAM BUY- IN 38%
ask them about their laptop sticker. Find out about the
problems their team is facing, and tell them about your BUDGET CONSTRAINTS 19%
challenges. It's likely that there are skills and processes
that can be shared between teams to solve specific prob- MORE HYPE THAN SUBSTANCE 14%
lems, and they just haven't been identified yet. Creating
these open channels of communication builds empathy
and helps break down silos between Operations and
Development teams.
Demand for DevOps Skills is Growing
As DevOps adoption continues to accelerate, demand for RELATIVE GROWTH: MENTIONS OF
DevOps skills has followed. Job listings for "DevOps" DEVOPS AS A SKILL

increased by 75 percent from January 2012 to January DEVOPS

2013 (Indeed.com), and mentions of "DevOps" as a skill CONTINUOUS
DEPLOYMENT
increased by 50 percent (LinkedIn.com).
ELASTIC LOAD
INDEED BALANCER
COBBLER

GANGLIA
0 10 20 30 40 50

We wanted to know what constitutes the “DevOps skills”

IT professionals need to stay ahead of the curve. We
found coding and scripting high on the list of coveted
skills, reflecting another emerging trend in IT: the need to
automate manual tasks with modular, sharable bits of
code. People skills were next, because communication
and collaboration are the key to DevOps success. Process
re-engineering was also popular, indicating a need for a
holistic view of the system, rather than one-off solutions.

Desired skills for DevOps roles

[
84% 60% 56% 19%

Coding/Scripting People Skills Process Experience w/
Re-Engineering Specific Tools
Skills
Interestingly, experience with specific tools was not a TOP 5 TOOLS USED TO SUPPORT
priority. You can teach the tools more readily than you DEVOPS INITIATIVES
can teach the other skills. Regardless of the tools you
choose, sharing the same toolchain eases communication VERSION CONTROL SYSTEMS
84%
across teams, allowing everyone to speak a common
language. Version control systems and configuration CONFIGURATION MANAGEMENT 78%
management tools were seen as the biggest enablers of
DevOps. TICKETING SYSTEM 68%
One way to start building a common toolchain is to talk
to your Development or Operations team and see if there
RESOURCE MONITORING 60%
are ways you can apply their tools to what you do. Consider
PROVISIONING 56%
identifying a common pain point between both
teams—such as deploying from dev to production—and
see if you can jointly improve that process.
Recommendations for Implementing DevOps
As DevOps adoption continues to accelerate and more
organizations demonstrate success, those that aren’t
practicing DevOps are at risk of being left behind. To stay
ahead of the curve, and achieve higher levels of perfor-
mance, organizations need to foster those skills within
their teams.

RECOMMENDATIONS FOR SYSADMINS,

DEVELOPERS, OR SQA ENGINEERS

Automate. Automate. Automate. Pick one source of truth and make it so.
Automation is the single biggest driver of high Consolidate multiple sources of information into
performance, increasing the overall quality and one source of truth by creating synchronization
speed of code deployments. Greenfield environ- scripts for your HR system, CMDB, Asset DB, Policy
ments have the advantage of not being bogged DB, etc. Whether you use a service, a database
down by legacy processes and technical debt, but (SQL or Hiera on disk), or pure data in version control
even established IT organizations can make incre- (a YAML or JSON file), the important thing is that
mental improvements using automation. Auto- all data inputs to your configuration state are
mate a single pain point such as DNS, NTP, or root stored centrally and accessible via your configu-
passwords. Start small, prove the value, and use ration management system.
the visibility that success brings to tackle bigger
Learn the tools.
projects.
Sharing a common toolchain can help foster com-
Break down cultural barriers. munication across teams and spread empathy
DevOps doesn’t require buy-in from the whole about the challenges they face.
company. If you’re in Operations, find a developer
who writes the code you deploy. If you’re a devel-
oper, find one of the ops people who deploys your
code. Have coffee. Hang out. Building relation-
ships "across the aisle" will increase everyone's
understanding of the problems facing different
parts of the organization, which goes a long way
towards getting everyone working towards the
same goals.
 RECOMMENDATIONS FOR MANAGERS

Foster DevOps skills within your team. Encourage lateral communication.

You almost certainly have people with DevOps Foster a culture of direct communication between
skills already working for you. Support them. peers, rather than using the top-down approach.
Listen to their ideas and help them succeed. You Often, the best ideas will bubble up from the
don’t need to hire a DevOps team and create yet bottom: The more people that are collaborating,
another functional silo. Instead, experiment with the more dynamic the exchange of ideas.
embedding ops and dev people on the opposite
team, or creating a cross-functional team respon-
sible for delivering a specific product or service.

Develop and use metrics.

Metrics are critical to tell the story of your suc-
cess. They help you understand how you and your
team are performing as well as help others under-
stand why the DevOps investment is worthwhile.
Use agility and reliability metrics such as deploy
rate, change lead time, change failure rate, and
mean time to recover to show business value. Use
functional metrics like test cycle time, deploy-
ment time, defect rate in production and helpdesk
ticket counts to demonstrate your success.

Organizations that follow these practices will not only

increase agility and reliability, they will also have happier,
more productive employees. Employees who know how to
foster these environments will have more opportunities
for growth as demand for these skills continues to grow.
In the end, everyone wins—employees, the business, and
your customers.

START AUTOMATING YOUR INFRASTRUCTURE TODAY HTTPS://PUPPETLABS.COM/PUPPET/PUPPET-ENTERPRISE/

Key Findings
In December of 2012, Puppet Labs and IT Revolution
Press surveyed over 4,000 IT Operations professionals ORGANIZATIONS THAT HAVE IMPLEMENTED
DEVOPS SAW THESE BENEFITS:
and developers in the largest industry survey of this mag-
nitude. The survey results revealed accelerating adoption IMPROVED QUALITY OF
SOFTWARE DEPLOYMENTS 63%
of DevOps practices in IT organizations across companies
of all sizes. Sixty-three percent of respondents have MORE FREQUENT
SOFTWARE RELEASES 63%
implemented DevOps practices, a 26 percent increase in
adoption rate since 2011.1 IMPROVED VISIBILITY INTO IT
PROCESS AND REQUIREMENTS 61%
Respondents from organizations that implemented
CULTURAL CHANGE
DevOps reported improved software deployment quality COLLABORATION/ COOPERATION 55%
and more frequent software releases. Our findings sub-
MORE RESPONSIVENESS
stantiate the business value of DevOps, too: It enables TO BUSINESS NEEDS 55%
high performance by increasing agility and reliability. We
MORE AGILE
found that high performing organizations: DEVELOPMENT 51%
Ship code 30x faster MORE AGILE CHANGE
and complete those deployments 8,000 times faster than their MANAGEMENT PROCESS 45%
peers.
IMPROVED
Have 50% fewer failures QUALITY OF CODE 38%
and restore service 12 times faster than their peers.

HIGH PERFORMANCE BY DEVOPS MATURITY

Not surprisingly, organizations that have implemented
DevOps practices are up to five times more likely to be
52%
high-performing than those that have not. In fact, the
42%
longer organizations have been using DevOps practices,
21%
the better their performance: The best are getting better. 9%

Not Currently Implemented Implemented

DevOps adoption is at a tipping point. Adopting DevOps Implementing Implementing <12 Months >12 Months

practices improves performance, and the longer those

practices are in place, the greater the performance advan-
tage. Waiting to implement is no longer an option.
Following our recommendations will put you ahead of
the curve as DevOps approaches critical mass.

1
Respondents of both the 2011 and 2012 survey are likely to be early adopters, making the increase in DevOps adoption
rate much more telling of the overall growth of DevOps during this time period.
Who Took The Survey?
We saw respondents from over 90 countries Respondents worked for organizations running
across organizations of all sizes. The majority of the gamut from startups to small/medium enter-
respondents identified themselves as administra- prises to WebOps giants, revealing that DevOps is
tors, engineers, or developers, with only 16 per- happening everywhere.
cent manager-level or above.

SURVEY AREA

90+
COUNTRIES

COMPANY SIZE (# OF EMPLOYEES) DEPARTMENTS

IT OPERATIONS DEVELOPMENT/ ENGINEERING QA / OTHER

20%
16% 70% 23% 7%
1-19
ROLES
20-99

 
100-499

16% 500-9999 26%

75% 17%
10000+

22%
ADMINS/
ENGINEERS
MANAGERS/
C- LEVEL  8%
CONSULTANTS
Key Findings
DevOps Increases Agility and Reliability
IT organizations are expected to respond more quickly to
urgent business needs while simultaneously providing
stable, secure, and predictable IT service. However, the
systems on which the business operates are typically
fragile and hostile to change. Adopting Agile develop-
ment processes without improving operational reliability
or communication between developers and operations
only makes this problem worse. The increased frequency Rob Cornish, CTO at the International
of releases from development creates even more of a Securities Exchange (ISE), a leading
burden on an already strained IT organization. Similarly, US options exchange that exceeds
adopting rigid ITIL/ITSM standards without addressing daily trading volumes of 2.5 million
development issues and improving communication chan- contracts.
nels results in an inflexible IT organization that simply
"DevOps has been instrumental in
cannot respond to business needs quickly enough.
reducing our software deployment
DevOps picks up where Agile methodology and IT stan-
cycle from days to minutes. We are able
dards left off.
to deploy new features to our customers
For the first time ever, by adopting DevOps practices, IT quickly and safely, and rapidly respond
organizations can be simultaneously agile and reliable, to market changes."
thus fulfilling the promise of over 20 years of Agile and
ITIL/ITSM.
High performing organizations:
Deploy code 30 times more frequently.
High performing organizations deploy code 30 times more often,
and 8000 times faster than their peers, deploying multiple times
a day, versus an average of once a month. Frequent deployments
coupled with faster change lead times enable operational agility.

Have 50 percent fewer failures.

High performing organizations have double the change success
rate and restore service 12 times faster than their peers. Fewer
failures and faster recovery mean less risk to the business when
changes are deployed.
DevOps practices enable IT organizations to quickly and
safely deploy changes, freeing them to work on higher
level business objectives. In the next section, we break
out four key metrics that show the impact DevOps has on
performance.
 KeyDevOps
How Findings
Enables High Performance
A high-performing organization is characterized We found that organizations that had implement-
by its ability to ship business-critical applications ed DevOps practices were up to five times more
quickly without disrupting service. The classic likely to be high-performing than those that had
examples of this are big WebOps shops—Google, not, and the more mature the implementation,
Amazon, Twitter, and Etsy—known for deploying the higher the performance.
multiple times a day.

DEPLOY FREQUENCY CHANGE LEAD TIME

8% 14% 20% 27% 7% 10% 15% 23%

ON DEMAND LESS THAN
ONE HOUR

BETWEEN ONCE LESS THAN

A WEEK & ONCE ONE DAY
A MONTH
DEPLOY FREQUENCY

BETWEEN ONE DAY

CHANGE LEAD TIME

& ONE WEEK

BETWEEN ONCE
A MONTH & ONCE BETWEEN ONE
EVERY 6 MONTHS WEEK & ONE
MONTH
BETWEEN ONCE
A DAY & ONCE BETWEEN ONE &
A WEEK SIX MONTHS
< ONCE EVERY 6 MONTHS > 6 MONTHS
NOT CURRENTLY IMPLEMENTED IMPLEMENTED NOT CURRENTLY IMPLEMENTED IMPLEMENTED
IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS

CHANGE FAILURE RATE MEAN TIME TO RECOVER

16% 16% 22% 23% 17% 26% 35% 47%

<5% MINUTES

5–10%
MEAN TIME TO RECOVER
CHANGE FAILURE RATE

LESS THAN
ONE HOUR

10–30%

HOURS
30–50%
>50% DAYS
NOT CURRENTLY IMPLEMENTED IMPLEMENTED NOT CURRENTLY IMPLEMENTED IMPLEMENTED
IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS IMPLEMENTED IMPLEMENTING <12 MONTHS >12 MONTHS

We analyzed four key DevOps performance metrics—deploy frequency, change lead time, change failure
rate, and mean time to recover—by DevOps maturity, ranging from not implemented to implemented over
12 months ago. Organizations with more mature DevOps implementations saw significantly higher
performance across all metrics compared to those that had not yet implemented DevOps.
Key Agility Performance Indicators
DEPLOYMENT FREQUENCY Obamas DevOps Success
High-performing organizations deploy at least once a week,
and often multiple times a day. On average, this is 95 percent
less time between deployments than lower performing orga-
nizations, enabling these organizations to quickly respond to
market changes or customer feedback, and iterate on new
ideas.

CHANGE LEAD TIME

Being able to quickly make changes is a basic measure of agil-

ity. If your infrastructure is bogged down by technical debt, the Using DevOps practices, Obama’s
time to get code successfully running in production is signifi- tech team was able to manage 2,000
cantly longer and your changes are more likely to fail. nodes across three data centers, pro-
High-performing organizations make changes with a few min- cessing over 180TB and 8.5 billion
utes’ notice, while their peers have change lead times of up to requests, with just 30 minutes of
a month. Agile organizations can make 8,000 changes before downtime during the entire 18
their slower competitors can vet and deploy a single change. months of the election campaign.
2

Key Reliability Performance Indicators

MEAN TIME TO RECOVER

DEVOPS VALUE IN ACTION:
High-performing organizations recover from outages within
VELOCITY AT AMAZON AWS 3

minutes, compared to lower performing organizations that

take an hour or longer to recover: over 30 times faster. Most Max deployments/
hour
organizations can't afford even a few minutes of down-
time—that can mean thousands of lost transactions, resulting 10,000
in lost revenue and lower customer satisfaction.

CHANGE FAIL RATE

11.6 .001%
When changes fail, your organization has to devote time and
resources to recovering, usually by taking those resources from
Mean time Software
developing, releasing, and supporting new products or between deployments
updates. High-performing organizations have over 50 percent deployments causing an
(seconds) outage
fewer failures from code changes. Changes often fail because
the development and test environments don't match the
production environment.

2
Condliffe, Jamie, “Inside Obama’s Tech Team.” http://gizmodo.com/5960790/inside-obamas-team-tech. Nov 15, 2012.
3
Jenkins, Jon. "Velocity Culture." Keynote, Velocity Conference, Santa Clara, Jun 16, 2011.
How to Achieve High Performance
Now that we know what high performance looks like, how ORGANIZATIONS USING VERSION
do you actually achieve it? High-performing organiza- CONTROL AND AUTOMATED CODE
DEPLOYMENT ARE HIGHER PERFORMING
tions share two common practices:
VERSION CONTROL
• 89 percent use version control systems for infrastructure
management 100% .5%
6%
• 82 percent have automated their code deployments

PERFORMANCE DISTRIBUTION
These practices have a direct impact on the key perfor-
52%
mance metrics that contribute to increased organization-
al agility and reliability.

Using version control as your single source of truth gives

you the ability to pinpoint the cause of failures, easily roll
back to a known state, and quickly deploy new service
instances. Everything required to run an application, 0% NO SOME VERSION
VERSION VERSION CONTROL
including the infrastructure and configuration code, CONTROL CONTROL

should be in version control. AUTOMATED CODE DEPLOYMENT

100% .4%
Automating code deployments provides several benefits
9%
that directly contribute to high performance. First, auto-
mating the configuration of your development, test, and
PERFORMANCE DISTRIBUTION

production machines eliminates configuration drift 57%

between environments—a common point of failure in the

deployment process. Second, automation significantly
reduces your change lead time by replacing manual work-
flows with a consistent, repeatable process. Third, auto-
mation tools give visibility into the impact of changes
before they’re promoted to production, reducing risk. 0% HAVE HAVE HAVE
NOT AUTOMATED AUTOMATED
AUTOMATED SOME

Version control and automation together enable the

PERFORMANCE SEGMENT
highest levels of efficiency and productivity. Developers
and QA engineers are able to request and provision prop- HIGH MEDIUM LOW

erly configured environments in one command. This

removes Operations as a bottleneck, empowers develop-
ment/QA teams, and enables everyone to focus on things
that matter to the business.
Key Findings
Identifying Barriers to DevOps Adoption
DevOps processes and tooling contribute to high perfor- BIGGEST DIFFICULTIES IN
IMPLEMENTING DEVOPS
mance, but these practices alone aren't enough to achieve
organizational success. The most common barriers to VALUE OF DEVOPS NOT
DevOps adoption are cultural: lack of manager or team UNDERSTOOD OUTSIDE MY GROUP 48%
b u y - i n , o r t h e v a l u e o f D ev O p s i s n ' t u n d e r s t o o d NO COMMON MANAGEMENT
outside of a specific group. These barriers come from STRUCTURE BETWEEN DEV AND OPS 43%
communication breakdowns, and working to solve them
DON' T HAVE TOOLS IN PLACE 33%
may help address other commonly identified blockers.
DON' T HAVE TIME TO
How common are these cultural barriers? IMPLEMENT DEVOPS 31%
• 49 percent of respondents who had no plans to implement DON' T HAVE SUPPORT
TO BE SUCCESSFUL 19%
DevOps identified a lack of manager buy-in as a blocker—this
was the most frequently identified barrier for this group, IT' S TOO EXPENSIVE 5%
followed by "lack of team buy-in" (38%).

• 48 percent of all respondents indicated that one of the biggest THOSE WHO HAD NO PLANS TO IMPLEMENT
difficulties in implementing DevOps was that the value wasn't DEVOPS CITED THE FOLLOWING REASONS:
understood outside of their group.
LACK OF MANAGER BUY- IN 49%
The best way to overcome these barriers is to start a con-
versation. Invite someone from another team to lunch, or LACK OF TEAM BUY- IN 38%
ask them about their laptop sticker. Find out about the
problems their team is facing, and tell them about your BUDGET CONSTRAINTS 19%
challenges. It's likely that there are skills and processes
that can be shared between teams to solve specific prob- MORE HYPE THAN SUBSTANCE 14%
lems, and they just haven't been identified yet. Creating
these open channels of communication builds empathy
and helps break down silos between Operations and
Development teams.
Demand for DevOps Skills is Growing
As DevOps adoption continues to accelerate, demand for RELATIVE GROWTH: MENTIONS OF
DevOps skills has followed. Job listings for "DevOps" DEVOPS AS A SKILL

increased by 75 percent from January 2012 to January DEVOPS

2013 (Indeed.com), and mentions of "DevOps" as a skill CONTINUOUS
DEPLOYMENT
increased by 50 percent (LinkedIn.com).
ELASTIC LOAD
INDEED BALANCER
COBBLER

GANGLIA
0 10 20 30 40 50

We wanted to know what constitutes the “DevOps skills”

IT professionals need to stay ahead of the curve. We
found coding and scripting high on the list of coveted
skills, reflecting another emerging trend in IT: the need to
automate manual tasks with modular, sharable bits of
code. People skills were next, because communication
and collaboration are the key to DevOps success. Process
re-engineering was also popular, indicating a need for a
holistic view of the system, rather than one-off solutions.

Desired skills for DevOps roles

[
84% 60% 56% 19%

Coding/Scripting People Skills Process Experience w/
Re-Engineering Specific Tools
Skills
Interestingly, experience with specific tools was not a TOP 5 TOOLS USED TO SUPPORT
priority. You can teach the tools more readily than you DEVOPS INITIATIVES
can teach the other skills. Regardless of the tools you
choose, sharing the same toolchain eases communication VERSION CONTROL SYSTEMS
84%
across teams, allowing everyone to speak a common
language. Version control systems and configuration CONFIGURATION MANAGEMENT 78%
management tools were seen as the biggest enablers of
DevOps. TICKETING SYSTEM 68%
One way to start building a common toolchain is to talk
to your Development or Operations team and see if there
RESOURCE MONITORING 60%
are ways you can apply their tools to what you do. Consider
PROVISIONING 56%
identifying a common pain point between both
teams—such as deploying from dev to production—and
see if you can jointly improve that process.
Recommendations for Implementing DevOps
As DevOps adoption continues to accelerate and more
organizations demonstrate success, those that aren’t
practicing DevOps are at risk of being left behind. To stay
ahead of the curve, and achieve higher levels of perfor-
mance, organizations need to foster those skills within
their teams.

RECOMMENDATIONS FOR SYSADMINS,

DEVELOPERS, OR SQA ENGINEERS

Automate. Automate. Automate. Pick one source of truth and make it so.
Automation is the single biggest driver of high Consolidate multiple sources of information into
performance, increasing the overall quality and one source of truth by creating synchronization
speed of code deployments. Greenfield environ- scripts for your HR system, CMDB, Asset DB, Policy
ments have the advantage of not being bogged DB, etc. Whether you use a service, a database
down by legacy processes and technical debt, but (SQL or Hiera on disk), or pure data in version control
even established IT organizations can make incre- (a YAML or JSON file), the important thing is that
mental improvements using automation. Auto- all data inputs to your configuration state are
mate a single pain point such as DNS, NTP, or root stored centrally and accessible via your configu-
passwords. Start small, prove the value, and use ration management system.
the visibility that success brings to tackle bigger
Learn the tools.
projects.
Sharing a common toolchain can help foster com-
Break down cultural barriers. munication across teams and spread empathy
DevOps doesn’t require buy-in from the whole about the challenges they face.
company. If you’re in Operations, find a developer
who writes the code you deploy. If you’re a devel-
oper, find one of the ops people who deploys your
code. Have coffee. Hang out. Building relation-
ships "across the aisle" will increase everyone's
understanding of the problems facing different
parts of the organization, which goes a long way
towards getting everyone working towards the
same goals.
 RECOMMENDATIONS FOR MANAGERS

Foster DevOps skills within your team. Encourage lateral communication.

You almost certainly have people with DevOps Foster a culture of direct communication between
skills already working for you. Support them. peers, rather than using the top-down approach.
Listen to their ideas and help them succeed. You Often, the best ideas will bubble up from the
don’t need to hire a DevOps team and create yet bottom: The more people that are collaborating,
another functional silo. Instead, experiment with the more dynamic the exchange of ideas.
embedding ops and dev people on the opposite
team, or creating a cross-functional team respon-
sible for delivering a specific product or service.

Develop and use metrics.

Metrics are critical to tell the story of your suc-
cess. They help you understand how you and your
team are performing as well as help others under-
stand why the DevOps investment is worthwhile.
Use agility and reliability metrics such as deploy
rate, change lead time, change failure rate, and
mean time to recover to show business value. Use
functional metrics like test cycle time, deploy-
ment time, defect rate in production and helpdesk
ticket counts to demonstrate your success.

Organizations that follow these practices will not only

increase agility and reliability, they will also have happier,
more productive employees. Employees who know how to
foster these environments will have more opportunities
for growth as demand for these skills continues to grow.
In the end, everyone wins—employees, the business, and
your customers.

START AUTOMATING YOUR INFRASTRUCTURE TODAY HTTPS://PUPPETLABS.COM/PUPPET/PUPPET-ENTERPRISE/