Leading the Transformation: Applying Agile and DevOps Principles at Scale

UNDERSTANDING THE TRANSFORMATION

There are large organizations in the industry using

leading-edge

techniques such as Agile and DevOps to develop software faster and more efficiently than anyone ever thought possible. These are typically companies that learned how to architect and develop software well while they were still relatively small. They then grew large quickly because of these breakthrough capabilities. Think Google, Amazon, and Facebook.

Currently, however, the majority of software is not developed by

leading-edge

groups like these, but by more traditional organizations using less efficient approaches. This book is written to help leaders of these traditional organizations understand how to successfully transform their development and delivery processes.

Improving the effectiveness of software development in traditional organizations is essential because software is a key way businesses now compete across a broad range of industries. Mechanical engineers that designed and built cars led the automobile industry. Then, through no fault of their own, they found that computers had infiltrated their product and become a larger part of the value they provide their customers. Now, instead of the salesman showing off the car engine, they start with a screen for the entertainment and control system—all based on software. Financial institutions that used to depend on traders working the floor and brokers forging customer relationships are finding that software for managing trades and interacting with their customers is helping them stay competitive. Retail has gone from building, stocking, and managing stores to creating software that provides a common customer experience across stores, websites, and mobile devices and that manages inventory more efficiently across all these channels.

No industry is immune from the

far-reaching

changes based on the increasing influence of software. Jeff Immelt, CEO of General Electric, warns, In an industrial company, avoid [gaining mastery of] software at your peril. We are paranoid because a software company could someday disintermediate GE. I’m going back to school on big data and software.

While it is clear that software is becoming a more and more important aspect of how these companies need to compete, most large, traditional organizations are struggling to deliver. They can’t respond to changes in the marketplace fast enough, and the businesses are getting frustrated. These companies are typically struggling with lots of

hard-to

-change, tightly coupled legacy software that requires them to coordinate development, qualification, and deployment efforts across hundreds to thousands of engineers, making frequent deliveries impossible. The deliveries they do provide require lots of

brute-force

manual effort that is frustrating and burning out their teams.

The net result is that most large, traditional organizations are finding it more and more difficult to compete in the marketplace and deliver the software innovations that their businesses require. Their current software delivery approaches are constraining their businesses and limiting their ability to compete.

Because their current approaches don’t work, many larger organizations are looking to leverage the successes that smaller businesses have seen using Agile methodologies. They bring in Agile coaches and start forming Agile teams to apply Agile principles at the team level. The problem with this approach is that in small organizations, a couple of small Agile teams can organize to support the business. In large, traditional organizations, however, most of the time individual teams can’t independently deliver value to the customer because it requires integrating work across hundreds of developers and addressing all the inefficiencies of coordinating this work. These are issues that the individual teams can’t and won’t solve on their own. This is why the executives need to lead the transformation. They are uniquely positioned to lead the

all-important

cultural changes and muster the resources to make the necessary

organization-wide

technical changes.

In this book we, the authors, will provide a fundamentally different approach for transforming the software development processes in large, traditional organizations by addressing the

organization-wide

issues that you, the executives, are uniquely positioned to handle. While most Agile implementations start with a focus on applying Agile principles at the team level, the approach presented in this book focuses on applying the basic principles of Agile and DevOps across the organization. It is based on what we, as executives leading complex transitions in large, traditional organizations, have found to be most effective for delivering solid business results.

Many specifics referenced in this book are leveraged from a case study of transformation at HP, detailed in A Practical Approach to

Large-Scale

Agile Development, by Gary Gruver, Mike Young, and Pat Fulghum.

This case study includes the following dramatic results:

»  Development costs reduced from ~$100M to ~$55M

»  140% increase in the number of products being supported

»  Increased capacity for innovation from 5% to 40%

The organization at HP achieved these results through applying DevOps and Agile principles at scale. Our focus was on applying the principles at the executive staff level, and we left the teams with as much flexibility in operational choices as possible. There were some groups that applied all the

team-level

Agile principles and some that chose to operate with more traditional methods.

What we found in the end is that there were not dramatic differences in the teams’ productivity based on the methods they used. There were, though, dramatic improvements in the overall productivity of the entire organization. This lead to our conclusion that how teams come together to deliver value in large organizations is the

first-order

effect, while how individual teams work was a

second-order

effect. Therefore, this book will primarily focus on how to transform the way the teams come together to provide value to the business by integrating all their changes early and often in an

operation-like

environment. This is one of the most important steps in improving the effectiveness of large organizations, because it forces resolving conflicts between teams early before too much time and effort is wasted on code that won’t work together in production. Then, when that part of the transformation is complete, the organization will have the right framework in place to continue improving and

fine-tuning

how the individual teams work with more traditional Agile methods at the team level.

Executives need to understand that applying Agile and DevOps principles at scale both differs significantly from typical Agile implementations and provides quicker time to value. To help executives understand why, they need to understand the challenges that large organizations experience with traditional approaches. In chapter 2 we will dissect the Waterfall Method, look at the Agile principles that answer Waterfall’s shortcomings, and then uncover key challenges that result from using a more traditional Agile approach in the enterprise.

The first step executives need to understand about our approach is that it is paramount to begin with business objectives. You should never do Agile or DevOps just so you can say you did them. A

large-scale

transformation is too much work and turmoil just to be able to say you are doing Agile. We believe that the key reason executives would be willing to take on this much change is that their current development processes are failing to meet the overarching needs of the business. Executives are in the best position to understand those failings and the needs of the business, so they are best suited to clarify the objectives of the transformation. In chapter 3, we will go into how executives begin to lead the transformation, using these objectives to communicate the vision, prioritize improvements, and show progress.

Once the business objectives have clarified the

long-term

goals of the transformation, executives then will use an

enterprise-level

continuous improvement process to engage the organization throughout the journey. Because it is so hard to measure process improvements with software, executives can’t just manage the change by metrics like they would other parts of their business. They are going to have to engage with the organization to get a more qualitative understanding of what is working and what needs fixing next. This transformation can’t be

top-down

, just like it can’t be

bottom-up

.

The continuous improvement process is designed to engage the broader organization in setting objectives the team feels are important and achievable. Additionally, since a transformation of this size can take years and is going to be such a discovery process, it is designed to capture and respond to what everyone is learning along the way. The executives will use a combination of the business objectives and the continuous improvement process to lead the transformation and prioritize improvements based on what will provide the biggest benefit to the business. We will cover the continuous improvement process in more detail in chapter 4, including setting

short-term

objectives for each iteration, focusing on what everyone is learning, and identifying what is and isn’t working to determine priorities for the next iteration.

In this book we will use the term

enterprise-level

 to describe an organization with software development efforts that require 100 or more engineers to coordinate the development, qualification, and release of their code. It does not refer to a coordinated effort across an organization the size of HP, because that would just be too complex. The plan for transforming the organization should be kept as small as possible to reduce complexity. But if different applications in the enterprise have to be qualified together to ensure they work in production, then they should be included as part of the same

enterprise-level

transformation.

FIGURE 1

Once the business objectives and continuous improvement process