How to build a better

test strategy:
A layered approach for
Agile, SAFe, and DevOps
environments

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 1

 Introduction

The test strategy document originated in the days fall to Agile may be delivering faster but may not
of waterfall development and testing – often re- be inherently increasing the quality of products. It
ferred to as “waterfail” because it required a huge has also been found that Agile can improve team
commitment of time and people. Many organi- communication, but less so across the entire
zations have abandoned this approach (and the organization. Documenting an organization-wide
strategy document along with it) as they transi- test strategy can address these challenges.
tioned to Agile and DevOps methods. Rather than
While the primary purpose of test strategies is
doing away with test strategy documents alto-
to clearly define what needs to be tested and
gether, Agile teams should revise their approach
how, taking the time to define one can also spark
with a test strategy that is built to accommodate
creative ideas. In this white paper, we outline a
changes in requirements.
creative approach that likens designing a test
Documented test strategies give teams the ability strategy to building a house. Metaphors like
to communicate and work together more effec- this can stop repetitive thinking in its tracks and
tively, regardless of the method they are using. become powerful catalysts for generating new
Teams that have made the transition from water- ideas. Read on to learn more.

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 2

 Three benefits of a unified test strategy
Communication is the number one challenge a By following a unified test strategy, organizations
test strategy document can address for Agile and can avoid these difficulties and benefit from:
DevOps teams. In an environment that prioritizes
1. Quickly identifying the cause when product
short release cycles, an iterative approach, and
defects are found
automation of repetitive steps, it’s critical for all
stakeholders to understand what’s necessary to 2. Less duplication of efforts from siloed test
get the job done. Building a product according to strategies
DevOps principles means that everyone — be it a 3. Improved efficiencies across the entire
developer, operations engineer, or product testing organization
manager — shares a common vision, maintaining
A centralized test strategy may also result in
it via communication. If the testing organization
better collaboration among teams, elimination of
does not share their vision for quality in a test
testing bottlenecks, and improved quality of releases.
strategy, there will be gaps in communication
before the project ever gets off the ground.

“We don’t have to approach big, complex problems like test strategy documents as if we have
to boil the ocean. Much the same way that we break down complex testing problems in our
everyday job, we can use these same skills to approach our test strategies.”
- Adam Satterfield, Senior Director of Test Engineering, Global Payments

Building a modern test strategy:

Wearing many hats: Why communication is
A layered approach
the strongest skill a testing leader can have
The movement to Agile and DevOps has too Testing leaders and their teams know more
often left test strategy documents behind. But about the testing system than anyone else be-
the concept and purpose still apply, and when cause they are in it on a day-to-day basis. The key
approached correctly, they can facilitate to creating a truly successful strategy is to con-
collaboration, speed, and overall success in sider the different ways it may impact developer,
modern delivery environments. operations engineer, or product manager roles,
and to prioritize understanding their perspec-
Taking a modern, layered approach to the test
tives. This will help testers ask the right questions
strategy process helps teams to understand
during strategy sessions and develop a shared
how each testing phase contributes to the over-
vision for quality that everyone can endorse.
all product quality. Each layer builds upon the
last and provides cover and protection — just
Implementing the right tools can help maintain
like with building a house. Here are the steps to
this shared vision by simplifying cross-team
developing an effective test strategy, using this
communication and collaboration. Tricentis
metaphor as a guide.
qTest improves these areas with easy-to-config-
ure workflows that trigger updates across Agile
and DevOps tools, from Jira to Jenkins. Learn
more about leveraging qTest here.

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 3

 The blueprint: The strategy session
Use the strategy session to define the test strategy, 3 tips for working with virtual
scope, and objectives for the project, ensuring align- testing teams
ment to business requirements. It’s best to go into the
session with a clear understanding of the goals: Will Working with virtual teams can make
you use the strategy session to ensure adequate test- strategy sessions more challenging. Here’s
ing coverage or speed, that the final product provides some tips to keep teams engaged:
an improved user experience, or that testers are clear
on scope? Clarify the roles that each team member will 1. Use cloud tools to share information more
have in testing, discuss the impacts of the test strategy easily
on each part of the process, and decide who needs to 2. Set up micro strategy sessions with
be involved to make the project successful. small groups instead of a multi-hour
meeting with all stakeholders
The foundation: Unit testing
3. Communicate clear deadlines for feed-
A strong foundation is a prerequisite of a strong house
back so there is no need to hunt down
in the same way that proper unit testing is the founda-
team members
tion of high quality software. Even though unit testing is
something typically owned by development, the testing
team must understand how it is done, where it is hap-
pening, and what metrics are being tracked. It is crucial Walls, paint, cabinets, and fixtures:
that they know what is being unit tested and what is not. Automation strategy

Automation is the core of any testing strategy

The frame: Integration testing and the piece that everyone from leadership
Almost no part of an application works in isolation. to testing teams are focused on. Like the
Integration testing verifies that various units will work finishes in a house, automation must relate to
together when grouped. By finding defects earlier and in the previous layers of the test strategy and
smaller assemblies, integration testing can reduce the build upon them. If the goal is to find efficien-
cost of defects, provide structure for the rest of the cies, then the testing team must truly under-
testing process, and contribute to faster deployments. stand the strategy in other areas to know what
efficiencies are to be found.
Plumbing and wiring: API and database testing
Often, the biggest vulnerabilities are hidden in the The finishing touches: Exploratory testing
API and database layers, where they can be harder to
Exploratory testing (and any other manual
detect – so don’t be tempted to skip this step. Making
testing the project requires) is the finishing
sure all the internal testing systems are hooked up and
touch that pulls it all together. There will always
working can be a complicated task because it requires
be a need for manual testing: It shows testers
knowledge of both the design of the interfaces and how
what users will see and can help ensure the
they are used. But with the right strategy and tools, tes-
best possible user experience. Like furniture in
ters can pick this up. Enabling testers to focus on the
a house, manual testing ensures users will be
API layer will make them more effective in their roles –
comfortable in all possible scenarios. Manual
by catching defects earlier, where they are easier to
testing does not compete with the other layers
diagnose and less risky to fix.
but works hand-in-hand with them.

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 4

 The neighborhood view: End-to-end testing The final test: Performance and load testing

End-to-end testing is typically performed as part of Sometimes the components work great individually
the deployment phase. Unlike a static test that is but then fail when they come together. It is essen-
just looking at the code, end-to-end testing builds tial to have performance and load testing rolled into
on each previous testing layer by looking at depen- the strategy to ensure the application offers the
dencies and ensuring all integrated pieces work best possible user experience, even under in traffic
together as expected. To make end-to-end testing conditions. This testing doesn’t have to be executed
successful, bring users, business stakeholders, and every release, but should be performed regularly as
product owners into the process to define high part of the testing routine.
business value tests. Ask them:

• What workflows and integrations, if they

failed, would bring business to a halt?

• What features or user flows, if they did not

work, would cause the phones to ring off
the hook?

• Which new features are most important to

you, your team, or our customers?

Knight Capital Group: What happens when the high business value test is skipped

Knight Capital Group was an American global financial services firm. In 2012, Knight lost $440 million
in the first 30 minutes of trading. Overnight they had introduced new code that was a high-frequency
trading algorithm designed to buy and sell massive amounts of stock in a short period of time. Knight
called the incident a “trading glitch,” but more likely it was due to problems with software develop-
ment and testing models.

“They went from a solvent, amazing company overnight to a company that was out of business by the
end of the day. That is a true example of a high business value test that was not done, and the compa-
ny went out of business,” said Adam Satterfield, Senior Director of Test Engineering at Global Pay-
ments, in a recent Tricentis webinar.

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 5

 Tying it all together with modern
test management
With all of these different types of testing going on, GitHub via webhooks, and offering a centralized
often across dozens of tools, it’s critical that testers view of test automation at the team, project, or
have a way to track, organize, and report on testing organization level.
activities across projects, teams, and tools.
Test management also can help put testing into
A scalable, in-sprint test management solution like the context of the business, allowing you to
Tricentis qTest can go a long way towards ensuring evaluate what’s working across projects, teams,
alignment with stakeholders across the delivery and tools. This information will become a valuable
pipeline. qTest’s flexible, open test management asset as you revisit and refine your test strategy
can keep teams and tools connected by integrat- over time, whether you are operating at the team
ing with Agile and DevOps tools, orchestrating or organization level.
common events across tools like Jira, Jenkins, and

THE MOST ASKED SOFTWARE TESTING STRATEGY QUESTIONS

1. Does test driven development (TDD) qualify customer health numbers, or call center numbers
as a test strategy? to capture how things are getting delivered. Capture
details that help to answer this question: “Is our test
It can from the standpoint of unit testing, as TDD
strategy helping us understand our risk better and
means defining tests before writing a single line of
allowing us to deliver better quality to our customers?”
code. In that sense, it is a fantastic albeit difficult
way to approach the overall development. 3. How do you combine tests without duplicating
them?
However, TDD does not necessarily help when put-
ting all the components together or understanding Chaining tests together is often seen with perfor-
how performant the system is. While TDD encap- mance and load testing. Instead of creating a bunch
sulates a great way to define unit and potentially of new scripts, combine existing Selenium scripts and
integration tests, it often falls flat for understanding run them multiple times to create a load environment.
acceptance automation or manual testing. Pair it In end-to-end testing, a chain of automation tests can
with things like behavior driven development or run together to create end-to-end scenarios.
acceptance driven development, but it should not
In both instances, there is naturally going to be some
be the only thing in a test strategy document.
level of duplication. Sometimes these are expensive
2. What are good metrics for understanding and time-consuming tests, and therefore not run on
the effectiveness of a test strategy? every release. However, defining the layers and under-
standing the final aim will help limit the duplication as
It is typically difficult to measure effectiveness from
much as possible.
a document-writing standpoint, so think outside
the box and consider metrics like defect data,

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 6