Opensource.

com

Starting a DevOps
Transformation

Breaking down walls between

people, process, and products
Open Source Cheat Sheets
Visit our cheat sheets collection for
free downloads, including:

Blender: Discover the most commonly and frequently used

hotkeys and mouse button presses.

Containers: Learn the lingo and get the basics in this quick and
easy containers primer.

Go: Find out about many uses of the go executable and the most
important packages in the Go standard library.

Inkscape: Inkscape is an incredibly powerful

vector graphics program that you can use to draw
scaleable illustrations or edit vector artwork that
other people have created.

Linux Networking: In this downloadable PDF cheat

sheet, get a list of Linux utilities and commands for
managing servers and networks.

Python 3.7: This cheat sheet rounds up a few

built-in pieces to get new Python programmers
started.

Raspberry Pi: See what you need to

boot your Pi, how to install the operating
system, how to enable SSH and connect
to WiFi, how to install software and update
your system, and links for where to get
further help.

SSH: Most people know SSH as a tool for

remote login, which it is, but it can be used
in many other ways.
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Opensource.com

About Opensource.com

What is Opensource.com?

Opensource.com publishes stories about creating,

adopting, and sharing open source
solutions. Visit Opensource.com to learn more about how the open source
way is improving technologies, education, business, government, health, law,
entertainment, humanitarian efforts, and more.

Submit a story idea: https://opensource.com/story

Email us: [email protected]

Chat with us in Freenode IRC: #opensource.com

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 3

About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Willy-Peter Schaub

Since mid-’80s, I have been striving for simplicity and

maintainability in software engineering. As a
software engineer, I analyse, design, develop, test, and support software
solutions. I am passionate about continuous
innovation and to share learnings from the digital
transformation, by Microsoft and the ALM | DevOps
Rangers, to a DevOps culture to adapt people,
process, and products to continuously deliver
value to our end users.

Contact Willy-Peter Schaub

Website: https://willys-cave.ghost.io
https://www.agents-of-chaos.org

Linked In: http://www.linkedin.com/in/wpschaub

Twitter: https://twitter.com/wpschaub

4 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Opensource.com

Introduction

Breaking down walls between people, process, 6

and products

Chapters

Blueprint for a team with a DevOps mindset 7

DevOps transformation: Key differences in small, 10
midsize, and large organizations
Analyzing the DNA of DevOps 13
Visualizing a DevOps mindset 17
Deploying new releases: Feature flags or rings? 20
What’s the cost of feature flags? 22

Get Involved | Additional Resources

Get involved | Additional Resources 25

Write for Us | Keep in Touch 26

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 5

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Breaking down walls

between people, process,
and products
DevOps transformation success hinges on removing the barriers inherent in an organization.

Over the past few months, I published a

few articles that explore the
three Ps that are core to your digital transformation, enabling
When we talk about process, we often gravitate to au-
tomation to enable efficient, stable, and consistent value
streams. It is important to also include goals to celebrate
you to move from a resource-optimized business model success as a team and organization, focus on quality from
based on capital expenses (CAPEX) to a market-optimized ideation to deprecation, create a lightweight and responsive
model based on operational expenses (OPEX). Among change-management process, embrace loosely coupled ar-
other benefits, the union of People, Process, and Products chitectures to enable scaling, and strive for multiple new-fea-
ture releases per day.
At the core of the trans-
formation are people and
their culture, not prod-
ucts, process, or even
the organization’s size.
People need to buy into
DevOps, understand how
their roles will be affected,
and take responsibility for
their part of the trans-
formation. People need
to realize that DevOps
is not limited to devel-
helps you drive core business values, such as increasing opment and operations, even if its very name excludes
the flow of business value, shortening delivery cycle times, other stakeholders. You need to break down all barriers
and enabling everyone to continuously learn, adapt, and and walls within your organization, bringing together all
improve. stakeholders, including development, data services, oper-
Engineers (like me) typically focus on products, tools, and ations, security, and business.
technologies. While these are pivotal to the success of To help you explore the three Ps, we’ve bundled a few
DevOps, it is important to realize that you cannot buy or articles for you to read.
install DevOps! Products
are enablers, not silver
bullets. They allow you to
focus on outcomes, such
as automation, consisten-
cy, reliability, maintainability,
progressively enabling or
disabling features, and visu-
alizing the flow of value.

6 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Blueprint for a team with a DevOps mindset

Blueprint for a team with a

DevOps mindset
Culture is the greatest challenge when embarking on a DevOps transformation.

I’ve had the privilege to work with some of the

brightest minds and leaders in
my 33 years of software engineering. I’ve also been fortu-
is core to agile practice. In addition, lean practices promote
reducing waste, creating short feedback loops, using light-
weight change approval, limiting work in progress (WIP),
nate to work for a manager who made me question my ca- reflecting and acting on feedback, and transparently visu-
reer daily and systematically broke down my passion—like alizing work management. All these strengths need to be
a destructive fire sucking the oxygen out of a sealed space. reflected in your blueprint.
It was an unnerving period, but once I broke free, I realized Let’s review an effective team blueprint’s genetic information.
I had the opportunity to reflect on one of the greatest an- As shown in the graphic below, self-organization is a nat-
ti-patterns for effective teams. ural process that creates order within the team. It outlines
It should come as no surprise that the culture of an or- HOW the autonomous team collaborates and coordinates.
ganization and its engineering teams is the greatest chal- Self-management defines how the diverse team members
lenge when embarking on a DevOps mindset [1] trans- work together in their own way, aligned with a shared vision
formation. The organization needs to influence through and governance, owned by the leadership.
leadership and autonomy, promoting a culture of learn-
ing and experimentation, where failure is an opportunity
to innovate, not persecute. Fear of retribution should be
frowned upon like the archaic Indian practice of Sati [2].
Teams need to feel they are operating in a safe environ-
ment, understand what the transformation entails, and
know how they will be affected.
So, what is the blueprint of an effective team? The con-
cept of autonomy, self-organization, and self-management

Line of autonomy and governance.

Team size is a topic that creates vibrant discussions.
Some say the ideal size is 7±2, others say 6±3, while oth-
ers maintain that there is no upper limit. Another argument
comes from anthropologist R.I.M. Dunbar [3], who says in
his paper [4] on the relationship between humans’ neocor-
tex size and group size, “there is a cognitive limit to the
number of individuals” in an effective team. He argues that

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 7

Blueprint for a team with a DevOps mindset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

if you want a highly cohesive team, keep your team size with others, while the vertical bar represents the depth of
smaller than 12. a single expertise. Assume we have a three-person team
Amazon CTO Werner Vogels’ famous quote “You build it, comprised of experts in development, testing, and user ex-
you run it” [5] is reminiscent of the great thematic quote from perience. Each has his or her own T-shaped skills. When
Spider-Man: “With great power comes great responsibility.” we combine the three experts’ genetic material, we get a
We need to foster ownership, accountability, and responsi- joint T-shaped team with specialization and broad exper-
bility. Everyone in the team must be empowered, trained to tise that can design, develop, test, and support features
run the business, responsible, and on call. When there is a as a cross-functional team. The culture of learning not only
live site incident, all designated response individuals, which ensures that the team’s combined broad and specialized
expertise is aligned with business needs, but it also em-
powers and motivates the team and its members. Infor-
EPIPHANY: The 2am wakeup call is the mation sharing and brown bag events are two excellent
team-building tools.
best motivation for a production-ready Shortage of testers? No problem; with the support of the
mindset. Pull any engineer test expert, the development and user experience experts
can temporarily blend into a test role. When testers auto-
into an early morning mate testing, learn and share good coding practices, help
live site incident a few improve feedback loops, and help the team broaden their
skills beyond unit and regression testing, the line between
times, and the quality bar developers and testers begins to blur. The developer and
magically shall improve. tester roles evolve and merge into the engineer role.
This raises the question: “What if team members refuse to
includes members from the associated feature team, must be cross-functional?” There is no place for heroes or knights
join the 2am call to gather evidence, investigate, and reme- in shining armor in an effective cross-functional team—find
diate at the root-cause level. them another home.
The 2am wakeup call is the best motivation for a produc- Inspire and celebrate success as a team and an orga-
tion-ready mindset. Pull any engineer into an early morning nization whenever achieving a goal. It motivates the team,
live site incident a few times, and the quality bar magically fuels team spirit, and acts as a major source of inspiration
shall improve. for others to excel. Your team can enjoy a round of nachos
Cross-functional is another key part of genetic informa- to confirm a job well done, play and learn with a game of
tion for an effective team blueprint. Contrary to common GetKanban, encourage team members to celebrate with
belief, it does not imply that everyone in the team can their family after a hectic sprint, or review and celebrate fast
do everything. Instead, as shown in the graph below, the feedback as part of your regular standups. The options are
cross-functional team is based on the concept of T-shaped endless—just ensure you spend time together as a team.
skills or T-shaped persons. The horizontal bar of the T For example, our scrum master always reflects on our
stands for broad expertise and the ability to collaborate achievements whether we pass or fail a sprint. It eventually

8 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Blueprint for a team with a DevOps mindset

dawned on me that it not only bonds us as a team, but others So, let’s get back to the greatest anti-pattern I mentioned.
aspire to be like our team. Effective teams, who are autonomous, empowered, self-or-
Finally, remember to keep it simple to reduce risk, cost of ganizing, and self-managed are based on trust, inspiration,
training, process, and products. Enable the just barely good and support of their leadership. If any of these important pil-
enough (JBGE) approach. Simplicity positively affects main- lars is missing, toxicity rises, passion declines, and you are
tainability and enables cross-functional teams to collaborate an eyewitness to the most destructive anti-pattern that will
and fill in for each other more effectively. doom any diverse, collaborative, and effective team.
You might be wondering why none of the above feels
like a new revelation. As discussed in Analyzing the DNA Links
of DevOps [6], we believe DevOps has inherited from de- [1] https://github.com/wpschaub/DevOps-mindset-essentials
cades of practices and learning, including waterfall, lean [2] https://en.wikipedia.org/wiki/Sati_(practice)
thinking, agile, and “real-world” 2am live site incident [3] https://en.wikipedia.org/wiki/Robin_Dunbar
calls. We are not inventing anything new; instead, we con- [4] http://www.uvm.edu/~pdodds/files/papers/others/1993/
tinue to reflect, learn, and streamline our blueprint. The dunbar1993a.pdf
DevOps mindset is based on a few matured foundations, [5] https://queue.acm.org/detail.cfm?id=1142065
and DevOps lights up when the team is razor-focused on [6] https://opensource.com/article/18/11/analyzing-devops-
delivery, telemetry, support in production, and (most im- dna-traces-waterfall-and-other-frameworks
portantly) bonding together as a team. A team that doesn’t
Adapted from “Blueprint for a team with a DevOps mindset” on Opensource.com,
enjoy being together is not a team; they are a group of published under a Creative Commons Attribution Share-Alike 4.0 International
individuals told to work together. License at https://opensource.com/article/18/12/blueprint-team-devops-mindset.

Blueprint for an effective team with a DevOps mindset.

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 9

DevOps transformation: Key differences in small, midsize, and large organizations . . . . .

DevOps transformation:
Key differences in small, midsize,
and large organizations
When embracing a DevOps mindset, does an organization’s size matter?

The continuous innovation journey to

DevOps is just that—
continuous—meaning it’s unlikely you’ll ever reach the
destination.
As depicted in the graphic below, the journey’s steps are:
assess and compare the organization with the rest of the
industry, ensure that the people buy into the transforma-
tion, introduce an engineering process and products that
enable teams to delight their stakeholders, continuously
deliver value, and scale solutions from hundreds to mil-
lions of users. Conceptually, the transformation should be
the same for any organization.
However, a DevOps
transformation in a com-
pany with a handful of en-
gineers is quite different
from one with hundreds or
thousands of engineers.
Instinctively, the DevOps
journey should be easiest
with small organizations,
as they are typically abun-
dant with passion and
an appetite for change.
However, small organi-
zations tend to be more
constrained on resources,
infrastructure, and bud-
get, while larger organi-
zations tend to have more
policies, governance, and
politics that affect the
transformation.
So, how does size mat-
DevOps transformation journey ter? Here’s what members

10 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . DevOps transformation: Key differences in small, midsize, and large organizations

Consider the DevOps-

as-a-Service and Fully
Embedded models Mat-
thew Skelton discusses
in “What Team Structure
is Right for DevOps to
Ability to embrace DevOps based on organization size Flourish?” [1] The former
model is ideal for small
organization and the latter
is feasible, as Microsoft’s
part-time ALM | DevOps
Ranger [2] community
demonstrates.

Midsize organizations
Midsize organizations’
rising resources and bud-
get often create an envi-
ronment where politics,
DevOps patterns in organizations by size policies, and technical
governance raise their
of the community said in a poll about how size affects the ugly heads. You are likely to find a focused IT operations
ease of transforming. team, one or a few engineering teams, and the dreaded
divide between development and operations. These are si-
Small and emerging organizations loed environments where development builds the solutions
In small organizations, management and engineering lines and IT ops supports the infrastructure and solutions.
of responsibility tend to blur, which naturally creates a lean Midsize organizations must focus on breaking down the si-
environment. Similarly, because resources are scarce, en- los, creating a common language, and establishing technical
gineering typically wears multiple technical and operation- governance that will unite the business, development, and
al hats, which organically creates cross-functional teams operations leadership and engineering. Talking about mani-
that are accountable for their solutions. Also, there is usu- festos (instead of governance) will reduce the angst and re-
ally an abundance of passion and appetite for new prod- sistance from engineering.
ucts and processes in small and emerging organizations, Consider the Temporary DevOps Team Skelton de-
which makes them ideal candidates for a transformation to scribes in the “team structure” article linked above. It is
a DevOps mindset. In addition, the DevOps transformation a good, albeit temporary strategy to bring dev and ops
is becoming a necessity to compete with larger competitors. closer together.
As management and engineering in small organizations
are lean, it is important to ensure there is a clear vision, engi- Large and established organizations
neering is empowered and accountable, the line of autonomy Few of us have the luxurious resources and budgets of
is respected, and the feedback and fail-fast processes are large organizations such as Amazon, Facebook, Google,
active. There also needs to be a clear “2 AM call” process or Microsoft. Large organizations have the diversity, ex-
for when a live site incident impacts the user experience— perience, and ability to divide their operations and devel-
in many cases, even the
CEO of a small organi-
zation must be on the
standby roster. Engineers
who are unwilling to ful-
fill a cross-functional role
are a risk for these orga-
nizations. Engineers must
have the right attitude, and
some may need to find an-
other home (within the or-
ganization or without). DevOps anti-patterns in organizations by size

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 11

DevOps transformation: Key differences in small, midsize, and large organizations . . . . .

opment teams into

many small, focused
teams. For example,
Microsoft has several
product-focused units,
such as Windows,
Office, and Azure
DevOps, each broken
down into many small
teams using a com-
mon engineering sys-
tem. The teams nat- DNA: Culture, leadership, process, and team
urally embrace agile
and DevOps practices, supported by a unified enterprise-level els Skelton discusses in his blog have proven successful
vision and transformation strategy. with large organizations that naturally embrace DevOps.
Consider establishing a community of excellence or cen- For others, the DevOps-as-a-Service and Temporary
ter of excellence to create special interest groups. Tapping DevOps Team models are ideal to bring dissimilar teams
thought leadership by skilled knowledge workers in this way closer together.
can enable and provide the organization with best practic-
es. These groups also support the concept of a DevOps- The right strategy
as-a-Service pattern to help move the organization to the Every organization is different, and assessing the right strat-
next level of awareness and maturity, share knowledge, cut egy is a combination of multiple characteristics, irrespective
waste, and innovate. Ensure you have representation from of size.
business, development, and operations! While all types of It is important to perform an assessment of the orga-
organizations would benefit from DevOps-as-a-Service, re- nization’s people, process, and products to reveal its
source requirements—budget and people—make it viable culture, leadership, teams, and appetite for change. More
for medium and large organizations only. importantly, the assessment will highlight the areas that
Some large organizations feel like midsized organizations, will transform naturally and the areas you need to nurture
with siloed leadership, business, operations, and develop- thoughtfully.
ment teams, just bigger and more segregated. Their challeng- At the core, it is people and their culture, not an organiza-
es include encouraging the entire organization to consider tion’s size, that molds and differentiates organizations from
and embrace DevOps practices, translating organizational one other.
business-speak into a common language, and introducing
unfamiliar concepts such as scrums, kanban, sprint cadence, Links
short delivery cycles, quick and lightweight feedback loops, [1] https://blog.matthewskelton.net/2013/10/22/what-team-
and continuous experimentation in production. While many structure-is-right-for-devops-to-flourish/
organizations see the value of bringing development and op- [2] https://opensource.com/article/17/11/devops-rangers-
erations closer together, you will experience resistance from transformation
waterfall teams that are used to stringent sequences, detailed
and predictable scope, and milestone-focused projects. Adapted from “DevOps transformation: Key differences in small, midsize,
and large organizations” on Opensource.com, published under a Creative
You may also experience leadership unintentionally inter- Commons Attribution Share-Alike 4.0 International License at https://
fering with engineering teams, blurring the line of auton- opensource.com/article/19/1/devops-small-medium-large-organizations.
omy that separates
the WHAT and WHY
(owned by leader-
ship) from the HOW
and WHEN (owned
by engineering).
The Fully Em-
bedded or Smooth
Collaboration mod- People, process, products challenges in DevOps

12 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Analyzing the DNA of DevOps

Analyzing the
DNA of DevOps
How have waterfall, agile, and other development frameworks
shaped the evolution of DevOps? Here’s what we discovered.

If you were
to analyze the DNA of
ways. The software devel-
opment lifecycle is becom-
ing an increasingly complex
DevOps, what would you system of services and mi-
find in its ancestry report? croservices, both intercon-
This chapter is not a meth- nected and instrumented.
odology bake-off, so if you As DevOps is pushed fur-
are looking for advice or a ther and faster than ever,
debate on the best approach the speed of change is wip-
to software engineering, ing out slower traditional
you can stop reading here. methodologies like waterfall.
Rather, we are going to explore the genetic sequences We are not slamming the waterfall approach—many or-
that have brought DevOps to the forefront of today’s digital ganizations have valid reasons to continue using it. How-
transformations. ever, mature organizations should aim to move away from
Much of DevOps has evolved through trial and error, as wasteful processes, and indeed, many startups have a
companies have struggled to be responsive to custom- competitive edge over companies that use more tradition-
ers’ demands while improving quality and standing out in al approaches in their day-to-day operations.
an increasingly competitive marketplace. Adding to the Ironically, lean, Kanban [1], continuous, and agile prin-
challenge is the transition from a product-driven to a ser- ciples and processes trace back to the early 1940’s, so
vice-driven global economy that connects people in new DevOps cannot claim to be a completely new idea.

Haplogroup—Paternal line for SDLC

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 13

Analyzing the DNA of DevOps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Let’s start by stepping back a few years and looking at

the waterfall, lean, and agile software development ap-
proaches. The figure below shows a “haplogroup” of the
software development lifecycle. (Remember, we are not
looking for the
best approach but
“A fool with a tool is still trying to under- Lean thinking is based on five principles [4]: value, value
stand which ap- stream, flow, pull, and perfection. The core of this approach
a fool.” –Mathew Mathai proach has pos- is to understand and support an effective value stream, elim-
itively influenced inate waste, and deliver continuous value to the user. It is
our combined 67 years of software engineering and the about delighting your users without interruption.
evolution to a DevOps mindset.)

The traditional waterfall method

From our perspective, the oldest genetic material comes
from the waterfall [2] model, first introduced by Dr. Winston
W. Royce in a paper published in the 1970’s.

Like a waterfall, this approach emphasizes a logical and

sequential progression through requirements, analysis,
coding, testing, and operations in a single pass. You Kaizen
must complete each sequence, meet criteria, and obtain Kaizen is based on incremental improvements; the
a signoff before you can begin the next one. The wa- Plan->Do->Check->Act lifecycle moved companies toward
terfall approach benefits projects that need stringent se- a continuous improvement mindset. Originally developed to
quences and that have a detailed and predictable scope improve the flow and processes of the assembly line, the Kai-
and milestone-based development. Contrary to popular zen concept also adds value across the supply chain. The
belief, it also allows teams to experiment and make early Toyota Production system was one of the early implementors
design changes during the requirements, analysis, and of Kaizen and continuous improvement. Kaizen and DevOps
design stages. work well together in environments where workflow goes from
design to production. Kaizen focuses on two areas:
• Flow
• Process

Continuous delivery
Kaizen inspired the development of processes and tools
to automate production. Companies were able to speed
up production and improve the quality, design, build, test,
and delivery phases by removing waste (including culture
and mindset) and automating as much as possible using
machines, software, and robotics. Much of the Kaizen
philosophy also applies to lean business and software
practices and continuous delivery deployment for DevOps
principles and goals.
Lean thinking
Although lean thinking dates to the Venetian Arsenal in the Agile
1450s, we start the clock when Toyota created the Toyota The Manifesto for Agile Software Development [5] appeared
Production System [3], developed by Japanese engineers in 2001, authored by Alistair Cockburn, Bob Martin, Jeff
between 1948 and 1972. Toyota published an official de- Sutherland, Jim Highsmith, Ken Schwaber, Kent Beck, Ward
scription of the system in 1992. Cunningham, and others.

14 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Analyzing the DNA of DevOps

principal DevOps manager Donovan Brown [14] defines

DevOps as “the union of people, process, and products to
enable continuous delivery of value to our end users.”

Agile [6] is not about throwing caution to the wind, ditching

design, or building software in the Wild West. It is about be-
ing able to create and respond to change. Agile development
is based on twelve principles [7] and a manifesto that values
individuals and collaboration, working software, customer Let’s go back to our original question: What would you find
collaboration, and responding to change. in the ancestry report of DevOps if you analyzed its DNA?

Disciplined agile We are looking at history dating back 80, 48, 26, and 17
Since the Agile Manifesto has remained static for 20 years, years—an eternity in today’s fast-paced and often turbulent
many agile practitioners have looked for ways to add choice environment. By nature, we humans continuously exper-
and subjectivity to the approach. Additionally, the Agile Man- iment, learn, and adapt, inheriting strengths and resolving
ifesto focuses heavily on development, so a tweak toward weaknesses from our genetic strands.
solutions rather than code or software is especially needed Under the microscope, we will find traces of waterfall,
in today’s fast-paced development environment. Scott Ambler lean thinking, agile, scrum, Kanban, and other genetic ma-
and Mark Lines co-authored Disciplined Agile Delivery [8] and terial. For example, there are traces of waterfall for detailed
The Disciplined Agile Framework [9], based on their expe- and predictable scope, traces of lean for cutting waste, and
riences at Rational, IBM, and organizations in which teams traces of agile for promoting increments of shippable code.
needed more choice or were not mature enough to implement The genetic strands that define when and how to ship the
lean practices, or where context didn’t fit the lifecycle. code are where DevOps lights up in our DNA exploration.
The significance of DAD and DA is that it is a process-de-
cision framework [10] that enables simplified process de-
cisions around incremental and iterative solution delivery.
DAD builds on the many practices of agile software de-
velopment, including scrum, agile modeling, lean software
development, and others. The extensive use of agile mod-
eling and refactoring, including encouraging automation You use the telemetry you collect from watching your solu-
through test-driven development (TDD), lean thinking such tion in production to drive experiments, confirm hypothe-
as Kanban, XP [11], scrum [12], and RUP [13] through a ses, and prioritize your product backlog. In other words,
choice of five agile lifecycles, and the introduction of the DevOps inherits from a variety of proven and evolving
architect owner, gives agile practitioners added mindsets, frameworks and enables you to transform your culture,
processes, and tools to successfully implement DevOps. use products as enablers, and most importantly, delight
your customers.
DevOps If you are comfortable with lean thinking and agile, you will
As far as we can gather, DevOps emerged during a series enjoy the full benefits of DevOps. If you come from a water-
of DevOpsDays in Belgium in 2009, going on to become the fall environment, you will receive help from a DevOps mind-
foundation for numerous digital transformations. Microsoft set, but your lean and agile counterparts will outperform you.

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 15

Analyzing the DNA of DevOps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

eDevOps • Loosely coupled technical services while bringing together

and delighting all stakeholders
• Building potentially consumable solutions every two weeks
or faster
• Collecting, measuring, analyzing, displaying, and automat-
ing actionable insight through the DevOps processes from
concept through live production use
• Continuous improvement following a Kaizen and disci-
plined agile approach

The next stage in the development of DevOps

Will DevOps ultimately be considered hype—a collection of

more tech thrown at corporations and added to the already
extensive list of buzzwords? Time, of course, will tell how
DevOps will progress. However, DevOps’ DNA must contin-
ue to mature and be refined, and developers must under-
In 2016, Brent Reed coined the term eDevOps (no Goo- stand that it is neither a silver bullet nor a remedy to cure all
gle or Wikipedia references exist to date), defining it as ailments and solve all problems.
“a way of working (WoW) that brings continuous improve- DevOps != Agile != Lean Thinking != Waterfall
ment across the enterprise seamlessly, through people, DevOps != Tools !=Technology
processes and tools.” DevOps c Agile c Lean Thinking c Waterfall
Brent found that agile was failing in IT: Businesses that
had adopted lean thinking were not achieving the value, fo- Links
cus, and velocity they expected from their trusted IT experts. [1] https://en.wikipedia.org/wiki/Kanban
Frustrated at seeing an “ivory tower” in which siloed IT ser- [2] https://airbrake.io/blog/sdlc/waterfall-model
vices were disconnected from architecture, development, [3] https://en.wikipedia.org/wiki/Toyota_Production_System
operations, and help desk support teams, he applied his [4] https://www.lean.org/WhatsLean/Principles.cfm
practical knowledge of disciplined agile delivery and added [5] http://agilemanifesto.org/
some goals and practical applications to the DAD toolset, [6] https://www.agilealliance.org/agile101
including: [7] http://agilemanifesto.org/principles.html
• Focus and drive of culture through a continuous improve- [8] https://books.google.com/books?id=CwvBEKsCY2gC
ment (Kaizen) mindset, bringing people together even [9] http://www.disciplinedagiledelivery.com/books/
when they are across the cubicle [10] https://en.wikipedia.org/wiki/Disciplined_agile_delivery
• Velocity through automation (TDD + refactoring everything [11] https://en.wikipedia.org/wiki/Extreme_programming
possible), removing waste and adopting a TOGAF [15], [12] https://www.scrum.org/resources/what-is-scrum
JBGE (just barely good enough) approach to documentation [13] https://en.wikipedia.org/wiki/Rational_Unified_Process
• Value through modeling (architecture modeling) and shift- [14] http://donovanbrown.com/
ing left to enable right through exposing anti-patterns while [15] http://www.opengroup.org/togaf
sharing through collaboration patterns in a more versatile
and strategic modern digital repository Co-authored by Brent Reed
Using his experience with AI at IBM, Brent designed a matu-
rity model for eDevOps that incrementally automates dash- Brent Aaron Reed is a pragmatic leader who has been in-
boards for measuring and decision-making purposes so that volved at the forefront of technology and its application in
continuous improvement through a continuous deployment bringing value to people. Brent strives for continuous im-
(automating from development to production) is a real possi- provement through education, awareness, collaboration and
bility for any organization. eDevOps in an effective transfor- passion. Brent is certified in Microsoft, Security+, Agile & Dis-
mation program based on disciplined DevOps that enables: ciplined Agile and many other frameworks and technologies.
• Business to DevOps (BizDevOps),
Adapted from “Analyzing the DNA of DevOps” on Opensource.com, published
• Security to DevOps (SecDevOps) under a Creative Commons Attribution Share-Alike 4.0 International License
• Information to DevOps (DataDevOps) at https://opensource.com/article/18/11/analyzing-devops.

16 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visualizing a DevOps mindset

Visualizing a
DevOps mindset
Use this graphical analysis to help develop a
DevOps strategy for your organization.

These days, organizations are moving from

a resource-optimized business
model based on capital expenses (CAPEX [1]) to a mar-
Being a visual-minded person, I created a presentation
with posters [4] for the recent Global DevOps Bootcamp
(GDBC) [5]. This annual community-driven event is hosted
ket-optimized model based on operational expenses (OPEX around the globe to create an environment in which par-
[2]). What’s driving this shift? Reducing time to market and ticipants collaboratively explore digital transformation and
continuously delighting customers with value. DevOps insights.
Welcome to digital transformation. Are you ready to em- Let’s explore four of the quick-reference posters [6]
brace a DevOps mindset in your organization? (also referred to as visuals and infographics). For a more
As defined by DevOps manager Donovan Brown [3], in-depth discussion of DevOps, refer to The DevOps
“DevOps is the union of people, process, and products to Handbook [7], by Gene Kim, Jez Humble, Patrick Debois,
enable continuous delivery of value to our end users.” and John Willis.

Practices
DevOps is not about magical unicorns and colorful Based on the DevOps Assessment [8], the first two posters
rainbows. It’s a journey of continuous learning and are intended to be used when reviewing the assessment
results with all stakeholders. The first one introduces five
improvement, with a destination you never quite
key practices:
get to. It’s the reason that
all of the images herein are
based on the infinity symbol:

Top performers encourage a culture that fosters a growth

mindset, reward innovation, collaboration, experimentation,
learning, and user empathy. Strive for a process with respon-
sive application delivery, flexible scheduling, and iterative
experiments. Monitor, identify and mitigate issues, and con-
tinuously eliminate wasteful bottlenecks. Only valuable key
performance indicators are measured and used to strive for
better outcomes, such as a low change failure rate (CFR),

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 17

Visualizing a DevOps mindset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

minimal time to recover (MTTR), and remediation of issues actively listen to your users, progressively enable and dis-
at the root level. Lastly, technology, which is an enabler, is able features, perform continuous experiments, and mea-
the focus of the next poster. sure key performance indicators. Use all available feedback
to maximize learnings and influence value.
Technology Shift left encourages reviews, validations, and approv-
Here’s a companion of the practices poster, focused on als for both testing and security as early as possible in
technology: the feature delivery cycle to drive quality and a fail-fast
mindset. When technical debt exceeds a predefined limit
(for example, five bugs per engineer), encourage feature
teams to suspend feature work until the technical debt is
paid down.
Team autonomy and enterprise alignment are concerned
with what, how, and why we build. You need a common ca-
dence, or heartbeat, across your organization to enable all
leadership and feature teams to collaborate transparent-
Version control manages versions of your application, con- ly and effectively. The most effective feature teams own a
figuration, infrastructure, and other code. It enables team feature from idea to production, with autonomy on how they
collaboration and monitoring activities such as deployments. develop and support their features.
Top performers use topic branches for short-term isolation, Production-first is a mindset that does not differentiate
continuously merge changes into master, review, and audit how features and bugs are handled during development,
using Git pull requests, and version everything. testing, and operational support. Everything should be au-
Testing must be viewed as a continuous activity, embed- tomated, versioned, and fine-tuned in production. Lean on
ded into both the developer workflow and the continuous in- ring-based deployment and rings [10] to limit the blast radius
tegration (CI) and continuous delivery (CD) pipeline. of feature changes in production, remediate all issues at the
The cloud enables you to effectively provision your infra- root cause level, and remember to be transparent about is-
structure and move as fast as necessary. sues, root cause, and resolution (as a user, I’m much more
Lastly, monitoring enables you to form a hypothesis, val- understanding if I have an insight into issues).
idate or disprove experiments, proactively detect issues as Infrastructure as a flexible resource describes how solu-
they occur, and understand application health. tion architectures are adapted to the cloud, containerization,
The black bar on the right of the poster lists products and microservices. Adopt a pragmatic transformation that
to consider when you’re investigating technology for your makes sense for your organization, goals, products, and cul-
development, production, common engineering, and oth- ture. As with the previous habits, it’s important to favor au-
er environments. Provide feedback on the listed products tonomy over a descriptive architecture and not to transform
and regularly update this volatile and opinionated part of everything all at once.
the visual.
Getting started
Habits The last visualization combines all of the above and sug-
Based on the Moving 65,000 engineers to DevOps with gests five steps to getting started with DevOps:
VSTS [9] story, this poster focuses on the five key habits we
learned about during our transformation. The customer-fo-
cused, team autonomy and enterprise alignment, and shift-
left habits are evolutions of Agile, and the production-first
mindset and infrastructure as a flexible resource are particu-
lar to a DevOps mindset.

Customer-focused is part of our quest to delight customers I prefer to start with the assessment to help identify key ar-
and our obsession with delivering customer value. You must eas that can be improved.

18 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visualizing a DevOps mindset

• Assessments provide a benchmark of your DevOps mind-

set and performance against the rest of the industry. It’s
important to understand where you’re doing well and where Improvement is possible for everyone if leadership
investment will help take you to the next level. Both the provides consistent support and team members
DORA [11] and Microsoft [12] DevOps Assessments are commit themselves to the work. -Accelerate: The
great starting points. In addition, gather metrics to use as a
Science of Lean Software and DevOps [13], by
base to measure progress—for example, deployment fre-
quency, lead time for changes, mean time to repair, and Nicole Forsgren, Jez Humble, and Gene Kim
change failure rate.
• People and culture are your biggest challenges. Every- Which visuals do you like? Which ones add no value?
one needs to buy into the transformation, understand how What is missing? Let’s collaborate [14] to demystify DevOps
they will be affected, and help you and your users shine. Users need to under-
encourage transpar- stand that they are not alone and know they can rely on prov-
Without committed people and ency, be engaged, en practices and real-world learning.
an experimental culture, the rest and take full re- Looking forward to your feedback and pull requests!
sponsibility for their
of the DevOps transformation value streams. This Links
journey is futile. includes leadership, [1] https://www.investopedia.com/terms/c/capitalexpenditure.asp
which needs to be [2]  https://en.wikipedia.org/wiki/Operating_expense
supportive, inspirational, empowering, and drive a clear [3] http://donovanbrown.com/post/what-is-devops
vision. You’ll make or break the transformation as a team. [4]  https://github.com/wpschaub/DevOps-mindset-essentials/
• Process is your engineering system, which enables the blob/master/src/presentations/devops-mindset-essentials-
teams to manage live site incidents, use lean management gdbc.pdf
and development, and continuously deliver value. A com- [5] https://globaldevopsbootcamp.com/
mon engineering system introduces consistency, empow- [6]  https://github.com/wpschaub/DevOps-mindset-essentials/
ers feature teams, and enables and encourages everyone tree/master/src/posters
to support and contribute to each other. Your top process [7] http://a.co/92KSNxJ
goals should include a focus on quality, a loosely coupled [8] https://aka.ms/devopsassessment
architecture to enable scaling, lightweight management, [9]  https://www.slideshare.net/WillyPeterSchaub/devconf-
automation, multiple releases per day, and celebration of moving-65000-microsofties-to-devops-with-visual-studio-
success as a team and as an organization. team-services
• Products are the easiest link in the chain. They enable ev- [10] https://opensource.com/article/18/2/feature-flags-ring-
eryone to focus on what’s important: Delivering value to deployment-model
end users. [11]  https://www.devops-survey.com/
• Value is all about delighting users. Key performance indica- [12] https://aka.ms/devopsassessment
tors include deployment frequency, lead time for changes, [13] https://t.co/smb82Y4i0M
change failure rate, and time to recover. [14] https://github.com/wpschaub/devOps-mindset-essentials
Whether you tackle all these steps all at once (“big bang”),
step-by-step (“peeling an onion”), or gradually innovate your
Adapted from “Visualizing a DevOps mindset” on Opensource.com, published
value chain across all steps (“broad-spectrum innovation”) is under a Creative Commons Attribution Share-Alike 4.0 International License
your choice. Just be pragmatic. at https://opensource.com/article/18/8/visualizing-devops-essentials-mindset.

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 19

Deploying new releases: Feature flags or rings? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Deploying new releases:

Feature flags or rings?
Use deployment rings to progressively expose a new release,
and feature flags to fine-tune each release in production.

DevOps enables us to deliver at speed,

learn from production
feedback, make better educated decisions, and increase
customer satisfaction, acquisition, and retention. We need
to fail fast on features that result in an indifferent or nega-
tive user experience and focus on features that make a pos-
itive difference. Progressive exposure is a DevOps practice,
based on feature flags and ring-based deployment, that al-
lows us to expose features to selected users in production,
to observe and validate before exposing them to all users.
You’re probably asking yourself whether to use ring-
based deployments, feature flags, or both to support progres-
sive exposure in your environment. Let’s start by exploring
these strategies. When your developers complete a pull request with pro-
posed changes to the master branch, (1) a continuous
Rings and feature flags integration build performs the build, unit testing, and trig-
Ring-based deployment was first discussed in Jez Hum- gers an automatic release to the Canaries environment
ble’s book, Continuous Delivery [1], as canary deployments. in production. When you’re confident that the release is
Rings limit impact on end users while gradually deploying ready for user acceptance and exploratory testing in pro-
and confirming change in production. With rings, we evalu- duction, (2) you approve the release to the Early Adopters
ate the impact, or “blast radius,” through observation, testing, ring. Similarly, when you’re confident that the release is
diagnosis of telemetry, and most importantly, user feedback. ready for prime time, (3) you approve the release to the
Rings make it possible to progressively deploy binary bits Users ring. The names and number of rings depends on
and have multiple production releases running in parallel. your preferences, but it’s important that all rings are using
You can gather feedback without the risk of affecting all us- the same production environment.
ers, decommission old releases, and distribute new releases Feature flags were first popularized by Martin Fowler [2].
when you are confident that everything is working properly. Flags decouple release deployment and feature exposure,
The following diagram show an implementation of the ring- give run-time control down to the individual user, and enable
based deployment process: hypothesis-driven development. Using and tying feature
flags back to telemetry allows you to
decide if a feature helps to increase
user satisfaction, acquisition, and
retention. You can also use feature
flags to do an emergency roll-back,
hide a feature in a region where it
shouldn’t be available, or enable te-
lemetry as needed.

A typical feature flag implementation

is based on (1) a management service

20 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Deploying new releases: Feature flags or rings?

that defines the flag, (2) a run-time query to figure out the a new release, then use feature flags to fine-tune each
value of the flag, and (3) an if-else programming construct, release in production.
as shown: I think of a when using the ring-based deployment
model to deploy a release and a when using feature
flags to fine-tune the release.
Happy ringing and flagging!
Comparing rings with flags within the context of our open
source extensions [8]:

DEPLOYMENT RING FEATURE FLAG

Both the feature flags and ring-based deployment model Progressive Yes Yes
exposure
strategies are invaluable, whether you’re working with an
A/B Testing All users within ring
All or selected users
open source extension [3] community or moving 65,000 en-
Cost Production Feature Flag
gineers to DevOps [4]. environment database and code
maintenance maintenance
Back to the question: Should you use feature Primary use Manage impact Show or hide
flags, rings, or both? “blast radius” features in a release
The quote “You do not respond to a mosquito bite with a Blast radius - 0-9 canary users
0, all, or specific
hammer,” by Patrick L.O. Lumumba, comes to mind. Canaries canary users
We use both rings and feature flags to progressively ex- Blast radius - 10-100 early adopter 0, all, or specific
Early Adopters users early adopter users
pose a new release in production, whether it’s a hot fix
Blast radius - 10000+ of users 0, all, or specific
or feature release for our commercial product, affecting
Users users
65,000 engineers and eventually hundreds of thousands
of users as the blast or impact radius of the release in- For more details, refer to Software Development with Feature
creases. Feature flags allow us to progressively reveal [5] Toggles [9], Phase the roll-out of your application through
new features of each release, perform A/B testing, and ex- rings [10], and Our Feature Flag Investigations [11].
periment in production. Because we’re working with cloud
services and extensions, we have a continuous feedback Links
loop with our users and the ability to fine-tune each re- [1] https://www.continuousdelivery.com/
lease by toggling feature flags. [2] https://martinfowler.com/bliki/FeatureToggle.html
For our open source community extensions, we primarily [3] https://aka.ms/vsarsolutions#Extensions
use ring-based deployment to progressively expose a new [4] https://aka.ms/devops
release to canary, early adopters, and users, as outlined in [5] https://youtu.be/ed3ziUDq_n0
the table below. We’re gradually implementing feature flags [6] https://blogs.msdn.microsoft.com/visualstudioalmrangers/
in selected extensions to experiment [6] and gather experi- tag/launchdarkly/
ence in fine-tuning features and managing the associated [7] https://code.facebook.com/posts/270314900139291/rapid-
technical debt. release-at-massive-scale/
The quasi-continuous delivery mode [7] is another ex- [8] https://aka.ms/vsarsolutions#Extensions
ample of using both strategies to deploy new features [9] https://msdn.microsoft.com/en-ca/magazine/dn683796.
to 1% of the users in the first ring, then 20%, 50%, and aspx
100%, continuing with the same pattern to the second [10] https://docs.microsoft.com/en-us/vsts/articles/phase-
ring, and so on. rollout-with-rings
You can use either ring-based deployment or feature [11] https://blogs.msdn.microsoft.com/visualstudioalmrangers/
flags to implement the progressive exposure DevOps tag/launchdarkly/
practice—they are symbiotic. It all boils down to how cau-
tious you want to be when rolling out releases and expos- Adapted from “Deploying new releases: Feature flags or rings?” on Opensource.
com, published under a Creative Commons Attribution Share-Alike 4.0
ing features. I recommend that you experiment with both. International License at https://opensource.com/article/18/2/feature-flags-
Start by using deployment rings to progressively expose ring-deployment-model.

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 21

What’s the cost of feature flags?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

What’s the cost of

feature flags?
Here’s what you need to know about managing feature flags in a
progressive exposure environment.

In a previous article, I introduced fea-

ture flags and ring-based
deployments, both enablers for the DevOps practice of
entering the realm of the new Global Data Protection Reg-
ulation (GDPR) [1].
Don’t build your own custom feature flag solution.
progressive exposure. There are enough options available, such as the Feature-
Progressive exposure enables us to mitigate the im- Toggle [2], NFeature [3], FeatureSwitcher [4], togglz [5],
pact of changes as they occur, perform iterative experi- and ff4j [6] frameworks, and software as a service (SaaS)
ments, assess features, and get rapid feedback on every such as LaunchDarkly [7]. With the latter, you delegate
change—all in production. Feature flags, for example, maintenance, updates, and infrastructure to your SaaS pro-
enable you to perform short-lived experiments, isolate vider so you can focus product features and deliver value to
unfinished work, fine-tune your customers.
continuous releases, and
dynamically manage long- Technical debt
lived operational configura- With feature flags, we break
tions and permissions. up a product into indepen-
It’s a practice that not only dent parts that can be re-
leads to happy customers but leased separately, giving
enables us to create effective our feature teams and busi-
and motivated feature teams. ness control over who gets
You’re probably asking, which feature and when.
“What’s the catch?” By breaking up your prod-
For ring-based deploy- uct, you’re adding a level
ments, your primary cost is of complexity, which needs
to manage the production environments covered by the maintenance to avoid stale flags and associated code.
rings with a “production-first” mindset. You need to mini- For example, when we add a simple ON|OFF feature
mize the “blast radius” for each release, monitor each re- flag to isolate a feature, we’re adding an if-else code con-
lease, and mitigate root issues quickly. For feature flags, struct and doubling our code and test paths, as shown
you need to manage your feature flag product, manage below. If it’s an experimental feature flag, its lifespan is
technical debt, and develop an insight into the implications typically weeks, after which it needs to be removed to
of simply “flipping a flag.” avoid technical debt. For other feature flags, the lifespan
Let’s explore some of the costs of feature flags. may be longer; however, the same principle applies: Re-
move the feature flags and associated code as soon as
Product investment and operational cost you do not need them.
You need to investigate and find the right feature flag solu-
tion for your environment. Some important considerations
include seamless integration with your DevOps process
and products, simple and cost-effective management of
flags, ability to perform an emergency roll-back, and sup-
port for auditing, fine-tuned permissions, and security. For
example, if you’re managing feature flags down to a spe- When we add a multi-value OFF|1|2 feature flag, we multi-
cific user, you’re likely capturing personal information and ply the feature code and test paths. After adding two feature

22 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What’s the cost of feature flags?

flags, we are maintaining two features, with five code and

test paths. Every path needs to be validated and tested with
every change as we have no guarantee when a feature flag
will be toggled.

But there’s more: Let’s add another simple ON|OFF fea- In the feature flag admin system, the feature has a friendly
ture flag for a new feature for which we would like to display name, Display Logs, and display-logs flag.
collect telemetry to examine a hypothesis. It again dou-
bles the feature code and test paths, incrementing the
paths that need to be validated to seven. More impor-
tantly, it introduces a dependency on a specific version
of the second feature. Do we wait for the dependency
to be enabled? Do we rely on an isolated and potentially
incomplete feature? Who ensures that a feature flag is
not inadvertently toggled to satisfy a dependency? Great
questions, which need to be answered as part of your
process transformation to encourage flexible schedul- At a glance, the relationship between activateFF and dis-
ing, iterative experiments, and close team collaboration play-logs is not obvious. As an engineer, I’m reluctant to
and to facilitate real-time ownership and management of make changes to the code without further investigation (cost).
these challenges.
Understanding of the implications of flipping a flag
There’s one more important cost you need to consider. Flip-
ping a feature flag is simple—the change ripples through
production quickly, and your users start using your new fea-
ture with excitement. You’re happy with the feature you just
enabled, but are you confident that you understand all the
side effects of flipping the flag?
We often share the following two experiences, which
demonstrate that even with the best process, we can have
bad days that result in bad customer experiences.
• A rough patch [9] —The team flipped a feature flag at a big
marketing event when corporate vice president Brian Harry
was on stage. As described in the blog post, it didn’t go
Imagine a product with hundreds of feature flags. How do well. The product experienced unexpected authentication
you identify stale feature flags and associated code and failures and eventually buckled under load.
test paths adding to our technical debt (cost)? How do you • How we learned about the 503 meltdown [10] —The team
convince your feature teams to change and remove code enabled feature flags in one of their most popular exten-
from a fully functional product? The feature teams need to sions. Users experienced 503 errors, followed by severe
own the feature from sunrise (idea) to sunset (deprecate), performance issues, and eventually the Azure Functions
use a common engineering process, and apply consistent handling feature flags failed under load.
code and naming conventions. Identifying and removing In both cases, there was a “failure under load.” It’s import-
stale feature flags and code must be simple. ant to flip a feature flag in a canary or early adopter envi-
Let’s have a quick look at an extract from the Roll-up ronment and simulate anticipated loads a few days before
Board [8] extension, which shows the ON and OFF code flipping the feature flag for all users. This is particularly true
paths for a feature flag that checks the value of activateFF. for big marketing events, where first impressions count.

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 23

What’s the cost of feature flags? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A solid engineering process and live telemetry enabled us References

to detect the issues as they occurred, identify the root cause, • A Rough Patch [11]
and mitigate the impact. • Continuous Delivery by Jez Humble [12]
• Feature Toggles by Martin Fowler [13]
• How we checked and fixed the 503 error and performance
issue in our Azure Function [14]
• Moving 65,000 engineers to DevOps [15]
• Phase the features of your application with feature flags [16]
• Phase the roll-out of your application through rings [17]

Links
[1]  https://ec.europa.eu/commission/priorities/justice-and-
fundamental-rights/data-protection/2018-reform-eu-data-
protection-rules_en
[2]  https://github.com/jason-roberts/FeatureToggle
[3]  https://github.com/benaston/NFeature
[4]  https://github.com/mexx/FeatureSwitcher
It’s important to learn from these mistakes, explore po- [5]  https://github.com/togglz/togglz
tential implications, have user empathy, and be transpar- [6]  https://github.com/clun/ff4j
ent about issues, root cause, and resolution of bad days [7]  https://www.launchdarkly.com/
[8]  https://github.com/ALM-Rangers/Roll-Up-Board-Widget-
Extension
[9] https://aka.ms/bh-ff-sos
DevOps is a journey of continuous [10] https://aka.ms/vsar-ff-sos
learning and improvement, with a [11] https://aka.ms/bh-ff-sos
[12] https://www.continuousdelivery.com/
destination you never quite get to! [13] https://martinfowler.com/bliki/FeatureToggle.html
[14] https://aka.ms/vsar-ff-sos
like ours. Users with an insight are typically more toler- [15] https://aka.ms/devops
ant and supportive of your continuous journey of learning [16] https://docs.microsoft.com/en-us/vsts/articles/phase-
and innovation. features-with-feature-flags
Once you are cognizant of and manage the risks and [17] https://www.visualstudio.com/en-us/articles/phase-rollout-
costs, your feature teams will be able to progressively with-rings
expose releases using ring-based deployments and fine-
tune them using feature flags. Adapted from “What's the cost of feature flags?” on Opensource.com,
published under a Creative Commons Attribution Share-Alike 4.0 International
Enjoy observing your motivated feature teams—and more License at https://opensource.com/article/18/7/does-progressive-exposure-really-
importantly, your happy customers! come-costx.

24 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com

 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Get Involved | Additional Resources

Get Involved

If you find these articles useful, get involved! Your feedback helps improve the status
quo for all things DevOps.
Contribute to the Opensource.com DevOps resource collection, and join the team of
DevOps practitioners and enthusiasts who want to share the open source stories
happening in the world of IT.
The Open Source DevOps team is looking for writers, curators, and others who can help
us explore the intersection of open source and DevOps. We’re especially interested in
stories on the following topics:

• D
 evOps practical how to’s
• D
 evOps and open source
• D
 evOps and talent
• D
 evOps and culture
• D
 evSecOps/rugged software

Learn more about the Opensource.com DevOps team: https://opensource.com/devops-team

Additional Resources

The open source guide to DevOps monitoring tools

This free download for sysadmin observability tools includes analysis of open source
monitoring, log aggregation, alerting/visualizations, and distributed tracing tools.
Download it now: The open source guide to DevOps monitoring tools

The ultimate DevOps hiring guide

This free download provides advice, tactics, and information about the state of DevOps
hiring for both job seekers and hiring managers.
Download it now: The ultimate DevOps hiring guide

The Open Organization Guide to IT Culture Change

In The Open Organization Guide to IT Culture Change, more than 25 contributors from
open communities, companies, and projects offer hard-won lessons and practical ad-
vice on how to create an open IT department that can deliver better, faster results and
unparalleled business value.
Download it now: The Open Organization Guide to IT Culture Change

Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com 25

Write For Us . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Write for Us

Would you like to write for Opensource.com? Our editorial calendar includes upcoming themes,
community columns, and topic suggestions: https://opensource.com/calendar
Learn more about writing for Opensource.com at: https://opensource.com/writers
We're always looking for open source-related articles on the following topics:
Big data: Open source big data tools, stories, communities, and news.
Command-line tips: Tricks and tips for the Linux command-line.
Containers and Kubernetes: Getting started with containers, best practices,
security, news, projects, and case studies.
Education: Open source projects, tools, solutions, and resources for educators,
students, and the classroom.
Geek culture: Open source-related geek culture stories.
Hardware: Open source hardware projects, maker culture, new products, howtos,
and tutorials.
Machine learning and AI: Open source tools, programs, projects and howtos for
machine learning and artificial intelligence.
Programming: Share your favorite scripts, tips for getting started, tricks for
developers, tutorials, and tell us about your favorite programming languages and
communities.
Security: Tips and tricks for securing your systems, best practices, checklists,
tutorials and tools, case studies, and security-related project updates.

Keep in touch!
Sign up to receive roundups of our best articles,
giveaway alerts, and community announcements.
Visit opensource.com/email-newsletter to subscribe.

26 Starting a DevOps transformation . CC BY-SA 4.0 . Opensource.com