Performance testing in Agile environments

Deliver quality software in less time

Business white paper
Table of contents

Executive summary....................................................3
Why Agile? And, why now?.......................................3
Incorporating performance testing in
Agile processes.....................................................4
Key considerations in Agile performance testing............5
A day in the life of an
Agile performance user story...................................7
How HP supports performance testing
in Agile environments.............................................8
Executive summary Riding the wave of these proven benefits, this software
development has tremendous momentum. If your
Performance testing is an essential activity in all organization isn’t currently using Agile methodologies,
software development projects, including Agile chances are you soon will be.
ones. From initial planning to production analysis,
application performance drives the development of The Agile approach breaks through the barriers
better software iterations and releases. It should never of conventional waterfall approaches to software
be an afterthought. Application owners, stakeholders, development to deliver business value sooner and
developers, and testers should make performance accelerate return on investment (ROI). Instead of
a key consideration in every sprint, or iteration, having siloed teams that use sequential, building-block
in Agile development. methods, an Agile environment puts multifunctional
teams to work developing software in a manner that is
This paper will help you gain a closer understanding incremental, iterative, and adoptive.
of how you can integrate performance testing into
your Agile processes. It summarizes key concepts Agile methods promote close teamwork, frequent
and best practices involved in performance testing, reviews, multiple iterations, self-organization, and
highlights software tools that enable performance accountability. When done right, this best practice
validation in Agile environments, and provides allows for rapid delivery of high-quality software
practical tips that you can put to work today. and enables a business approach that aligns
software development with customer needs and
Why Agile? And, why now? company goals.

Around the world, organizations are working to

produce higher quality software in less time. To
realize this goal, many companies are adopting Agile
software development processes. This approach can
help accelerate time to market, enhance software
quality, increase productivity, and improve alignment
between technology and business objectives—to name
just a few of the many benefits of Agile approach to
software development.

Figure 1. Agile brings ROI back

Agile (blue line) generates faster ROI than the waterfall (grey line) approach

Agile Brings ROI Back

ROI
Release Release Release
1 2 3
$ Time

Go Live

3
 Performance testing in an Agile software development lifecycle
While good application performance is an expected outcome of any release, its foundation is put in
place during the release planning stage. Considerations include user stories that describe performance
requirements and functional user stories that can potentially affect the performance of an application.
Performance testing can help you answer questions such as these:
• Does the application architecture need to be modified to meet the performance guidelines?
• Will the use of newer technologies such as AJAX degrade the performance of the application?
• Does the IT infrastructure of the testing and production sites need to be upgraded?
• Will the user interface designs degrade the performance of the application?
• Can an expected increase in usage impact application performance?
• Will performance suffer when the application is made available in new geographies?

Incorporating performance testing in But what does it take to incorporate performance

testing into your Agile development processes? For
Agile processes starters, you need to think about performance testing
Performance testing is an integral part of Agile in new, more Agile ways. It is not something that
processes, it can help your organization develop happens after an application is pretty much complete.
higher quality software in less time while reducing Performance testing that takes place at the end of the
development costs. The goal is to test performance software development lifecycle (SDLC) in a waterfall
early and often in the development effort, and to process moves to the beginning in an Agile process,
test functionality and performance in the same along with analysis, design, and coding.
sprint. That’s because the longer you wait to conduct
Performance of an application is directly proportional
performance tests, the more expensive it will become
to its design and hence should be addressed at an
to incorporate changes.
early stage of the development lifecycle. Performance
Performance testing becomes more important for testing is taken into consideration throughout the
Web 2.0 applications because mashups that Agile SDLC—from the release planning stage onward.
incorporate content from multiple websites and One way of ensuring that performance testing is
databases into customized Web pages bring in a included early in the SDLC process is by having a
whole new set of performance challenges. clear definition of “done” and its adoption in the Agile
The Web 2.0 applications are not only more dynamic, implementation so that the application is tested for
but also have many small components that could affect functionality, performance, and security by the end
performance and the end-user experience. With of each sprint and hence is available for incremental
Web 2.0 applications, there are a lot more moving deployment, if desired.
parts and everything is interwoven—from client scripts
to network traffic, from Web services to server scripts.

Figure 2. Paradigm shift in SDLC

Sequential activities become simultaneous ones in Agile SDLC.

Analysis Analysis

Design Design

Code Code

Test Test

Time Time

4
 The definition of “done”
In an Agile project, the definition of “done” should include the completion of performance testing within
a sprint. Only when performance testing has been completed can you confidently deliver a successful
application to your end users. The best practices for performance testing within a sprint are:
• Gather all performance-related requirements and address them during system architecture discussions and
planning.
• Work closely with end users and stakeholders to define acceptance criteria for each performance story.
• Involve performance testers early in the project, in the planning and infrastructure stages.
• Make performance testers part of the development (sprint) team.
• Ensure that the performance testers work on test cases and test data preparation while developers are
coding for those user stories.
• Get performance testers to create stubs for any external Web services.
• Deliver each relevant user story to performance testers as soon as it is signed off by the functional testers.
• Provide continuous feedback to developers, architects, and system analysts.
• Share performance test assets across projects and versions.
• Schedule performance tests for off-hours to increase the utilization of time within the sprint.

Figure 3. Definition of “done”

The definition of “done” includes the completion of performance testing within a sprint.

Architecture,
Planning Infrastructure

Analysis

Design

Coding

Testing

Performance

User Acceptance

Pilot

Live

Key considerations in Agile Application owners and stakeholders should

be interested in the performance aspects of the
performance testing application right from the start. They determine the
To gain a closer understanding of how performance priorities for the SLO requirements. For example,
testing is integrated into Agile processes, let us look at those requirements that have the greatest impact on
some of the key considerations in performance testing. the application—and by extension, the business—are
Service-level objectives
given higher priority in the development process.
Service-level objectives (SLOs) drive the planning of Take for example, an application owner might offer
performance requirements in Agile environments. this user story: “As a user, I want all the pages of our
Through SLOs, business and IT managers agree on website to be refreshed within two seconds so that I
requirements for application throughput, response can spend my time more productively on our website.”
times, numbers of simultaneously supported users, and Here’s another example story: “As an application
other factors that affect the end-user experience. These owner, I want the application to support as many
requirements, which become part of the application as 100,000 users at a time without degrading the
backlog, must be met before an application can go performance of the application, so that I can make the
live in a production environment. application available globally to all employees of my
company.”

5
Making incremental
These examples set the requirements for the Trending
shipments possible application, which in turn will define the acceptance In an Agile environment, it’s important for application
criteria and drive the test scripts. owners to see continual improvement in an application
Performance validation is
over the course of successive sprints. They want to
an integral part of every Focused performance testing
software development see a favorable trend, where each iteration of the
SLOs spell out the expected behavior of applications.
project. But in Agile projects, application is better than the last.
it takes on a higher role. It But in a practical sense, development, test teams, and
makes it possible to deliver the business don’t always know the exact requirements This makes it all the more important to monitor
a potentially shippable application performance trends in terms of SLO
for an application until they see how it performs in a
product or application
increment. This means that, production environment. requirements. Trending reports allows you to give
if desired, the work done in stakeholders regular snapshots of performance,
iteration (a sprint) has gone For this reason, Agile processes rely on the production
which should ideally show that performance is getting
through enough checks and availability lifecycle (PAL) to see how an application
balances (including meeting steadily better or at least is not degrading.
actually performs in the real world. The feedback
performance objectives) that
the application could be from the production environment helps developers In addition, by looking at trending reports you do
deployed or shipped to meet and testers focus on specific problem areas of the not necessarily have to study the analysis on every
business needs. application, thereby increasing the resources during test run.
The decision to actually
deploy or ship an application short sprint cycles.
Reusable and shared testing assets
increment is influenced by
This is an important ongoing step in the Agile To accelerate testing and development work, Agile
many other factors, such as
the incremental value added development process. It provides developers and processes should make use of a repository of reusable
to the application in one testers with actual metrics and usage trends. They and shared testing assets. This repository of test scripts
sprint, the effect of an update
can then take those findings into account when they gives everyone on the virtual development team,
on a company’s operations,
and the effect of frequent work on the next version of the application. Real-world including contractors working on an outsourced basis,
updates on customers or end metrics can drive better tests—and better software. access to the same test assets.
users. Therefore, performance
isn’t the only issue—but it’s Test data preparation Among other benefits, the repository provides
an important one. Could you efficiencies that come with “follow-the-sun” testing.
In an Agile environment, sprints are short—just two to
potentially ship an application
increment? This is possible four weeks typically—so testing times are short too. Test scripts can be created over the course of a day in
only if you have carefully This makes it all the more important to bring test data one geography and then made available to testers in
validated its performance. preparation into the planning stage, so that there is another geography, who will run the test during their
enough time for performance testing. business day.

Test data preparation may be done under the The ability to reuse and share test assets becomes
guidance of a line of business (LOB) or some other more important with Agile development when the
area of the business that has a stake in the protection testing cycle is limited. It allows more work to get done
and privacy of the data that will be tested. For within the available time window.
instance, a stakeholder might want to mask a data set
Automated testing
to protect the privacy of a confidential customer or
The use of automated testing tools can speed up
employee information.
the process of performance testing. With the right
The earlier test data preparation takes place, the more software in place, you can create a script, make it
time you have for testing. So performance testers reusable, and then schedule a test to run in the off
should work with stakeholders in the planning stage to hours, when developers are not changing the code
prepare tests and test data. This kind of collaboration you’re testing. This makes it possible to achieve higher
is one of the keys to getting a lot of work done in a levels of software quality in less time.
relatively short sprint. Different types of work must
Automated testing helps you meet your regression
happen simultaneously.
and performance testing objectives within the
In this sense, Agile software development is a bit like tight timeframes of a two- to four-week sprint. This
just-in-time (JIT) manufacturing. You plan carefully for becomes even more important when you consider that
your needs, and then bring in the resources just when developers often hold onto their work till 60 percent of
you need them. the time has passed in a sprint, before handing off the
build for testing. That doesn’t leave a lot of time
for testing.

6
Continual analysis Stubbing simulates parts of an application that are
In Agile processes, continual analysis is important. either not written or not available. If an application
Both contributors (“pigs” in scrum terminology) and uses outside, third-party data sources—a common
stakeholders (“chickens” in scrum terminology) need occurrence in a Web 2.0 world—then performance
to keep a close eye on the progress of the project, testers and quality assurance (QA) specialists are
especially when it comes to application functionality going to need stubs because they cannot add load to
and performance. To give them the view they need, third-party production servers.
performance analysis should be both continual and
By performance testing components inside each sprint,
comprehensive. This ongoing analysis helps pinpoint
you will help enable the development effort yields
any problem areas in the application.
a high-quality application that performs well from
Analysis takes place all the way down to the daily end‑to‑end.
scrums that include IT infrastructure and performance
One last check!
testers as contributors and application stakeholders.
A hardening sprint is an optional last-step in the Agile
Contributors are active members of the sprint
process. This sprint is dedicated to final checks, such
team who participate in daily scrums, which give
as a last round of testing, end-to-end performance
all stakeholders visibility into the current state of
checks, and tests to verify compliance with SLOs.
the development effort. When all interested team
members know the performance of each sprint, they While it can highlight problems that need to be
are in a better position to keep the quality of the entire addressed, the hardening sprint is really there to
application high. The sooner problems are found, the increase everyone’s confidence in the application
sooner they can be fixed. before it goes into production. If you’ve done things
right—and made performance testing part the entire
Component testing
Agile SDLC—you should have no surprises. The
In an Agile environment, you will not have an
hardening sprint should not be the first time that
end‑to‑end application in every sprint. This makes
performance testing is being carried out.
it important to be able to performance test only a
portion or a component of an application. Stubbing
provides a way to test components.

A day in the life of an Agile performance user story

Application Sales lead generation application offered as software as a service (SaaS)

Current user base 30,000 subscribers

Vision Driven by the business:

“Grow total subscribers to 150,000 in the next 12 months by upgrading the application’s user interface to
more modern Web 2.0 for more interactive and engaging process flows and notifications.”

Roadmap requirement High-level and not so specific:

“As an application owner of the sales leads application, I want the application to scale and perform well to
as many as 150,000 users so that new and existing subscribers are able to interact with the application with
no perceived delays.”

Release and sprint requirement Specific:

“As an application owner of the sales lead application, I want the application to perform well to as many
as 150,000 simultaneous users so that when subscribers change their search criteria, an updated count of
leads available is refreshed with two seconds on the screen.”

Sprint execution Each sprint checks the two-second search requirement is maintained.
Multiple sprints found that the two-second requirement was slipping. But each time it was fixed immediately.

Outcome The application is delivered on time with the two-second search requirement because the performance
requirement was included in each sprint. If performance was put off until the hardening sprint, the product
would have been late or the requirement would not have been met.

7
 HP Diagnostics software
Figure 4. A performance user’s story HP Diagnostics extends HP LoadRunner and
A performance user’s story begins as an idea and then becomes
an acknowledged entity that influences the software development process. HP Performance Center software to address the
unique challenges of testing complicated Web 2.0
application architectures and enterprise applications
Product Vision
across the application lifecycle.
Product Roadmap HP Diagnostics gives your testers the tools to pinpoint
the problem area for performance bottlenecks,
Release Plan identified by HP Performance Center software—a
feature that saves precious time in a short
Sprint Plan development sprint. Among other capabilities, it helps
you find and solve problems earlier in the lifecycle,
Daily achieve higher quality by identifying the most common
Standup
application problems before applications go live, and
collect concrete data to support decisions to go live
with an application.
HP Business Availability Center software
How HP supports performance HP Business Availability Center is a comprehensive
testing in Agile environments application management solution that helps you
HP is a leader in performance testing software proactively identify and resolve problems with
for Agile and waterfall environments. Various HP applications quickly and efficiently, before the
performance management products are used widely business is impacted. It allows 360-degree monitoring
in Agile processes. These include HP Performance and management of business services and application
Center software, HP Diagnostics software, and health from the point of view of key stakeholders—the
HP Business Availability Center software. These business, its customers, and its partners.
products help your team manage performance In Agile environments, HP Business Availability
throughout the application lifecycle. Center and its diagnostics capabilities provide
HP Performance Center software
your developers and testers with real-world
HP Performance Center software can be used information on application performance. This
throughout the Agile SDLC to validate performance of information helps your team focus on specific
different iterations of an application. This integrated problem areas of the application and pinpoint
performance validation solution emulates hundreds individual components of the application that are
to tens of thousands of concurrent users to apply causing performance degradation.
simulated production workloads to virtually any client For more information
platform or environment.
To learn more about HP solutions that enable
With its advanced trending, scheduling, central performance testing in Agile environments, visit these
repository, and Web access features, HP Performance online sites:
Center is uniquely equipped to handle performance
testing in an Agile environment. Using Performance HP Performance Validation:
Center, your testing teams can stress an application www.hp.com/go/performancevalidation
from end-to-end—applying consistent, measurable, HP LoadRunner and Performance Center Blog:
and repeatable loads—and then use the data you www.communities.hp.com/online/blogs/
generate to identify scalability issues that could loadrunner/default.aspx
impact users.

Get connected
www.hp.com/go/getconnected
Get the insider view on tech trends, alerts, and
HP solutions for better business outcomes

© Copyright 2010–2011 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only
warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

4AA1-5263ENW, Created January 2010; Updated April 2011, Rev. 1