THE STATE OF

SALESFORCE TESTING
How Elite Teams Test Salesforce
to Drive ROI

2022 Edition
What’s Inside
Copado’s first annual State of Salesforce Testing Report provides a snapshot of the testing
strategies leveraged by today’s most innovative brands. We reached out to 275+ Salesforce
professionals to gain insights on testing and pinpoint common challenges and best practices.

Executive Summary 3

Common Salesforce Testing Challenges 7

Testing Trends 10

Testing 101: Models, Types, Teams 17

Testing Strategies and Solutions 28

The Evolution of Testing 40

Survey Methodology 43
Executive Summary
 Business Transformation Depends on Low-Code

Low-code is essential in today’s digital climate due Key Findings

to the persistent shortage of technical developers
and the time, risk and cost of traditional application
development. In fact, the secret to Salesforce’s
41% of teams don’t have time to test every
change before each release.
success is a low-code model of development that
makes it easier and faster for subject matter experts to 92% experience production issues due to
inadequate testing.
build mission-critical apps.

With rapid innovation comes the need for scalable 84% still rely at least partially on manual
testing.
end-to-end testing — and that’s where many teams
continue to struggle.
51% of organizations have 25+ full-time QA
resources (20% have 100+ resources).

95% higher total cost of ownership for

manual testing vs. commercial test
automation.

© 2022 Copado | Page 4

 Manual, Homegrown or Commercial?
Figure 1

Manual
Testing

Homegrown
Testing Tool

Commercial
Testing Tool

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 110% 120% 130% 140% 150% 160%

Median Cost of Testers as a Percent of Salesforce Cost

© 2022 Copado | Page 5

 Breaking the Bottleneck with Automated Testing

We created this report to achieve three goals: However, we also observed a growing shift from
manual testing to automated solutions — with
Analyze current testing processes
significant outcomes for teams wielding the power of
Measure the impact of quality automation:
Identify key areas for improvement
58% fewer production failures per year
Our findings were clear — testing remains the biggest
bottleneck to digital transformation. Companies of 50% more frequent releases
all shapes and sizes are still using traditional testing more likely to complete testing for each
methods such as manual testing or script-based 50% release
solutions. As a result, teams lack the time to test
changes before deployments and the release quality 50% lower total cost of ownership
suffers.

© 2022 Copado | Page 6

Common Salesforce
Testing Challenges
 Common Salesforce Testing Challenges

From Marketing Cloud to Commerce Cloud,

development teams integrate their cloud suites
through platforms like Customer 360 and MuleSoft.
But even with all this integration, Salesforce remains
just one piece of a company’s portfolio of enterprise
applications.

This makes testing on Salesforce a tricky challenge.

Going the low-code route ramps up development.

And because Salesforce manages the underlying
infrastructure, faster software cycles mean more
changes and updates that cause unexpected
surprises.

© 2022 Copado | Page 8

 Three Roadblocks to Salesforce Testing

1 Salesforce doesn’t exist in a vacuum. 3 Salesforce is highly configurable.

Connections to other apps and platforms require The entire ecosystem is designed to champion
end-to-end testing to validate each integration low coders. This empowers non-technical users
across the AppExchange, SAP, ServiceNow and to customize and configure orgs — from custom
more. If any business processes break, the fix is objects to different languages to page layouts.
likely to be costly. However, more org customization means more
testing complexity.

2 Salesforce apps require constant updates

and feature dynamic content.
Heavily customized apps within the Salesforce
ecosystem — CPQ, nCino, Veeva, etc. — have
technical elements (shadow DOMs and IFrames)
that require specialty skills to accurately test.

© 2022 Copado | Page 9

Testing Trends
 As Salesforce grows in scope and sophistication,
organizations must execute high-quality cross-cloud
customizations and verify third-party integrations
across other platforms and technologies.

© 2022 Copado | Page 11

 Planning and Production Issues

Testing is a proven way to maintain speed and Figure 2

confidence in the face of change. But do organizations Do you have enough time to fully test all changes
have the time and capacity to keep up? before each release?

More than two in five teams we surveyed (41%) told

us they don’t have enough time to sufficiently test all
changes before a release. We picked up on a common Yes, all planned testing is
pattern: these teams feel pressure to deliver features 52% done in time for each release
due to aggressive project timelines. Inevitably,
development takes longer than expected and testing
gets thrown on the backburner.

Even teams that claim to be agile often resort to No, due to time constraints
fixed time boxes like sprints — which function like 41% we often have to reduce the
mini waterfall projects. This approach (called “Water- scope of testing
Scrum-Fall”) squeezes testing to the end of the
process.
8% N/A, or I don’t know

© 2022 Copado | Page 12

 Can you complete all planned testing?
Figure 3

Use
automated
testing

No test
automation

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70%

% of Total Count of Responses

In comparison, teams that use automated testing are 50% more likely to complete their
planned testing on time. Each batch of changes requires new tests — but teams that invest in
automation are able to keep pace with testing needs and derisk the development lifecycle.

© 2022 Copado | Page 13

 Planning and Production Issues
Figure 4

On average, how many production issues arise each Without enough time planned for testing, quality
year due to the quality of the release? is often sacrificed for speed. We discovered that
a whopping 92% of respondents run into critical
business issues each year due to inadequate testing.
Bugs found early in the development cycle are easy to
2% None fix — but defects found later in the pipeline can cost
640X more to repair.

92%
36% 1-5

38% 5-10
Experience Critical
Issues Annually 18% 10+

6% N/A

© 2022 Copado | Page 14

 Lack of testing is the #1 risk to digital transformations.
Figure 5
% DEFECTS INJECTION % DEFECTS FOUND COST TO REPAIR DEFECT

85% 640x

40x

10x

4x
1x
CODING UNIT TEST FUNCTIONAL SYSTEM RELEASE
TEST TEST

Applied Software Measurement: Global Analysis of Productivity and Quality, Caper Jones McGraw-Hill Education, 2008

© 2022 Copado | Page 15

 Manual or Automated Testing?
Automated testing helps teams avoid more than twice as many production failures as teams
with traditional test methods (3 vs. 7 per year). The fallout from just one of these failures can
add up to millions of dollars in lost productivity, revenue and reputation.

Figure 6

Using test
automation

No test
automation

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0

Median 15b. Production Failures Per Year

© 2022 Copado | Page 16

 Testing 101:
Models, Types, Teams
 Testing Models
The Big Four

Software testing strategies go hand in hand with

development methodologies. These four models
approach testing from a cavalcade of angles and
pinpoint quality deficits at different stages in the
Classic Agile
pipeline:
Classic Model
Agile Model
Shift-Left Model
Shift-Left Shift-Right
Shift-Right Model

© 2022 Copado | Page 18

 Classic Model
Centralized QA for Waterfall
Development
The classic testing model involves a centralized
QA team that owns testing and software quality. Requirements
In traditional development, each software project
is carefully planned and scoped in advance in a Analysis
“waterfall” style. Depending on the size of the project,
testing might not happen months after development. Design

Coding
The QA department was created to provide a
quicker way to verify the work of developers and
Testing
meet aggressive release timelines without cutting
corners. QA managers plan testing strategies, allocate
Operations
resources and oversee teams of testers. This model
dominated the landscape before the advent of Agile
and is still widely used today.

© 2022 Copado | Page 19

 Agile Model
Short Sprints and Close
Collaboration
Test
Agile software development is founded on a bedrock loy
of short delivery increments (sprints) that each last a ep

D
few weeks. Shorter cycles require close collaboration

p
between testers, developers and business analysts

e l o
— with new sets of requirements introduced in every

D ev
R evie w
sprint.

Agile requires exploratory testing (simultaneously

learning, designing and testing) in order to rapid-test

n
s

ig
new software releases and weed out feature defects. e
D
When it comes to flow speed and test effectiveness, Plan Launch
agile offers better flexibility and visbility across teams.

© 2022 Copado | Page 20

 Shift-Left Model
Tackling Quality at the Source Eliminate the need for mass inspection by building
quality into the product in the first place.”
The mission of “shift left” is to guarantee quality by
addressing it at the root. Shift-left testing is designed -W. Edwards Deming
to give developers direct feedback via unit testing,
static analysis, pair programming and pull requests.
This model builds quality into the planning and
development process and loops in the QA team as
early as possible to get at the heart of customer
needs.

An early start gives QA enough time to create test

plans and earmark specific software for automated
testing vs. manual testing. Instead of enlisting an army
of business users to execute manual testing late in
the process, shift-left testing teams bring in a handful
of users to assist with test design during the early
stages.

© 2022 Copado | Page 21

 Shift-Right Model
Production Testing for
Unpredictable Systems
The “shift right” principle is all about testing in
production — monitoring software behavior and
testing functionality in real life. While traditional
testing replicates production systems in a sandbox
environment, shift-right testing is necessary for
complex systems whose behavior is impossible to
predict.

Due to feedback loops, delays and an ocean of provide continuous feedback and end-to-end visibility.
possible data points and user behaviors, web-scale Practices like feature flags can toggle capabilities on
systems are often expensive and complicated to or off in production — which allows for testing even
accurately simulate for testing. Shift-right testing after software has been deployed.
monitors these systems once they’re in production to

© 2022 Copado | Page 22

 Testing Types
Driving Developer Feedback and
Release Quality
Why is testing such a challenge? Because an
End-to-End Tests
application can function in many ways — but the
ways it can misbehave are as infinite as outer space.
Regression Tests
To guarantee quality from every angle, testing requires
a multi-pronged approach. Essentially, there are
Integration Tests
two main types of tests: fast tests to ramp up the
developer-feedback loop and comprehensive tests to
UI Tests
ensure release quality.

Acceptance Tests

Unit Tests

© 2022 Copado | Page 23

 Two Types of Fast Tests
Linting and Static Analysis
Automated feedback on the basic code syntax to
identify common flaws.

Unit Tests
Quick tests that execute individual units of code to
ensure functionality. (Apex and Lightning Components
can be unit tested in Salesforce)

© 2022 Copado | Page 24

 Six Types of Comprehensive Tests
Code-based Acceptance Testing Regression Testing
Salesforce includes complex business logic such as Any change to code or configuration could undermine
flows that rely on countless components under the existing functionality. These changes and updates are
hood. Apex code can be used to test these processes often seamless, but failures can remain undetected for
(keep in mind — this type of test takes longer to run). weeks or months without regression testing.

User Interface (UI) Testing Integration Testing

Tools that simulate the behavior of a user on a browser Real end-to-end functionality hinges on multiple
can validate if the system is working as expected. systems working together in harmony. Integration
Selenium is the best-known open source UI test testing makes sure business processes that involve
(although it can be brittle when the underlying UI is multiple systems are firing on all cylinders.
changed). The Robot Framework is a newer and more
robust type of UI test. User Acceptance Testing (UAT)
Actual end users need to verify that new functionality
Application Programming Interface (API) Testing will meet their needs before it’s deployed to
This type of test validates that APIs are responsive and production. UAT is often a final step before rolling out
return the expected information. API testing targets can changes to all users.
provide precise verification of specific network calls.

© 2022 Copado | Page 25

 Testing Teams
Figure 7
What Do They Look Like? Roughly how many people work full-time on
testing or quality assurance at your company?

It takes a village to tackle testing — and no two teams 1-10 27%

look alike. 51% of the organizations we surveyed have
25+ resources dedicated to full-time quality assurance
and 20% have more than 100. What’s more, 26% of 23%
11-25
respondents reported outsourcing their testing.

26-50 24%

51-100 7%

101+ 20%

© 2022 Copado | Page 26

 Who is primarily responsible for testing at your company?
Figure 8

We employ a full-time
49%
in-house testing team

We outsource most testing

26%
to consulting companies

Testing is primarily
23%
done by developers

Other 2%
23%

© 2022 Copado | Page 27

Testing Strategies
and Solutions
 How are companies testing today?

64% of teams still chiefly rely on manual

testing

84% of organizations still partly rely on

manual testing

10% of teams automate all their testing

© 2022 Copado | Page 29

 Do you currently have a test automation strategy in place?
Figure 9

No, we currently have manual testing and are

7% NOT looking into test automation

No, we currently only have manual testing but

17% are looking into test automation

Yes, but all previous test automation efforts have

17%
84%
failed and now we only test manually

Yes, there is some test automation but majority of

Rely on Manual 23% testing is still manual

Testing Yes, majority of our testing is automated with

20% some manual testing

10% Yes, all of our testing is automated

6% N/A or I don’t know

© 2022 Copado | Page 30

 Defining the Right Testing Strategy

An effective testing strategy starts with a clear The key is distinguishing repetitive work (that can
and concise definition of success. It should seek to be automated) from creative work (that can’t be
optimize experiences for both internal and external automated). Automating a test comes with an upfront
customers while accounting for the looming risk of cost in time — which can be paid back by future
end-to-end changes. Ultimately, the right strategy will savings once the automation gets up to speed. Teams
measure user risk against the costs and capabilities need to estimate:
of a testing team to find the most efficient way to
execute objectives. What are the testing dependencies?
How critical to the business is this scenario?
How often could changes undermine that?
How likely is that functionality to break?
How long does it take to manually test?
How likely are we to forget or run out of time to
test?

© 2022 Copado | Page 31

 Defining the Right Testing Strategy

Testing is a protective task. It must be prioritized Easy-to-use test automation solutions help QA
according to a company’s most critical business professionals and developers exercise creative
systems. For instance, a team might estimate that a ways to protect functionality against risks that could
certain functionality takes 15 minutes for a person to undermine the code.
test and has a 5% chance of breaking each month,
with a productivity cost of $10,000 per hour if it
breaks. These metrics are contrasted with the cost of
creating an automated test.
COPADO ROBOTIC TESTING

Web master

None No RU N

Automated testing solutions like Copado Robotic ADD ROBO T

Suite runs Results

Testing enable fast and reliable test scripts via low-

12 3
Case Status

master 27

code platforms and are an effective way to slash the master 16

cost of investment and reduce business risk. These

master 13

18
master

solutions automate repetitive tasks — freeing up

users to focus on creative work: exploratory testing,
designing new test cases, analyzing test plans,
communicating with users and more.

© 2022 Copado | Page 32

 Six Phases of the Software Testing Lifecycle

Requirements Analysis: Establishing what types of Those stages remain the same across both manual
tests needs to be performed. and automated testing. Automation simply allows
teams to be faster and more comprehensive in test
Test Planning: Project plan includes type, method, execution.
efforts and resources.
Requirements
Test Case Development: Creating tests aligned to Analysis

the requirement set.

Test Test
Closure Planning
Environment Setup: Provisioning the environment
and test data.

Test Execution: Running tests and capturing results,

Test Test Case
defects and resolutions. Execution Development

Test Closure: Making sure requirements have been Environment

fulfilled to end the test cycle. Setup

© 2022 Copado | Page 33

 Test Management Techniques

Test management helps users organize, track An effective test management system enables
and analyze their testing activities in one central organizations to:
location. Without effective test management,
organizations are likely to struggle with a lack of Leverage polymorphic test objects to integrate and
centralized test visibility, siloed test roles and limited manage different types of tests and tools
process automation between testing and release Clearly understand the definition of “done” on a
management. user story
Organize all test plans

© 2022 Copado | Page 34

 Security Testing

In the realm of software development, security testing is just as essential as functional

testing. These three best practices help teams execute effective security tests.

1 Define which metrics to track. 2 Run security tests (regardless of company

size).
Capturing data around defects not only
enhances quality — it sheds light on the Running regular tests fuels consistent delivery
importance of security testing. Tracking escaped cycles and helps organizations steer clear of
defects released to production and identifying costly security breaches.
which ones are related to security can help
teams strategize more precisely for future
deployments. 3 Find the right balance.
Security is important — but so is speed. It’s all
about finding the right balance to keep releases
from falling behind schedule. This is where the
“shift-left” mentality comes into play. The earlier
teams can test, the earlier they can address
concerns and reduce risk.

© 2022 Copado | Page 35

 Beyond Testing
DevOps for Salesforce
DevOps for Salesforce has emerged as one of the hottest trends in the low-code development world. This
technical and social methodology solves Salesforce deployment headaches by managing sophisticated XML
file merges and metadata dependencies.

Continuous Integration test requires initial effort, but the cost of re-running
Continuous integration (CI) speeds up development by each test is negligible. This makes test automation an
merging work early and often (at least daily). Rather ideal solution for high-velocity teams who need fast
than letting conflicts accumulate, CI resolves them and consistent testing to prevent backward regression
in small batches. CI prevents teams working in one as they build new capabilities.
part of the system from making changes to shared
components that could cause code to break elsewhere Continuous Delivery
in the system. Continuous delivery (CD) expands on CI by deploying
code to production in small batches on a regular
Automated testing minimizes this risk by providing basis (at least daily). While CI addresses the risks of
a layer of protection — allowing functionality to be parallel development, CD tackles the risks of delaying
checked and rechecked thousands of times with feedback from end users.
minimal human intervention. Creating an automated
© 2022 Copado | Page 36
 Teams using automated testing deploy 50% more
releases each year than teams that rely on manual
testing (34 on average vs. 22)

© 2022 Copado | Page 37

 Building a Culture of DevOps Success

DevOps success comes down to five core elements (CALMS):

Culture Lean
Mature DevOps teams commit to a culture Lean is all about doing more with less by
of continuous learning and experimentation. understanding and optimizing value streams. Toyota’s
This requires a growth mindset that prioritizes lean manufacturing practices have gone on to
psychological safety and information flow. revolutionize healthcare and now IT.

Automation Measurement
The central miracle of technology is automation and Taking a data-driven approach to DevOps and testing
the dramatic efficiencies it enables. But automation gives teams a yardstick to gauge the effectiveness of
is not a panacea — the other four DevOps elements process improvements.
must be developed to drive success.

© 2022 Copado | Page 38

 Building a Culture of DevOps Success

Sharing
Digital leaders share a fountain of information,
practices and resources between individuals, teams
and organizations. Books like Mastering Salesforce
DevOps show how traditional techniques like
modularization and continuous integration can be
adapted to the Salesforce platform. Learn more about
DevOps for Salesforce by reading Copado’s State of
Salesforce DevOps Report.

© 2022 Copado | Page 39

The Evolution
of Testing
 Testing has evolved significantly over the years — from
traditional waterfall practices and siloed tools to collaborative
agile methodologies and interconnected cloud platforms.
Today’s commercial testing solutions meet testing personas
at their skill level (no-code, low-code and pro-code) to quickly
ramp up testing quality and maturity.

© 2022 Copado | Page 41

 Five Key Testing Trends
Artificial Intelligence and Machine Learning: AI Robotic Process Automation: RPA is used to
computer systems carry out activities normally done automate business processes and human tasks.
by humans. AI leverages ML to learn new behaviors This technology complements test automation by
by making sense of datasets and algorithms. With streamlining repetitive tasks at a higher consistency
configurable and dynamic systems like Salesforce, AI/ and quality.
ML can automate mundane maintenance tasks and
predict future states.
By 2024, 75% of large enterprises will be using AI-enabled
test automation tools that support continuous testing
Robotic Testing: This emerging technology marshals across the different stages of the DevOps life cycle.”
software robots and artificial intelligence to execute
testing. The Robotic Framework makes testing more
productive and accessible for all types of users. Hyperautomation: All this talk of automation and
AI can stoke fears of robotic takeover and human
Low-code Testing: Platforms like Salesforce enable obsolescence. But the reality is quite the opposite
non-technical citizen developers to build and deploy — automating repetitive tasks frees up QA teams
apps. But these apps still need to be tested — ideally to focus on more creative initiatives and smarter
with automated testing tools that don’t require coding. processes.

© 2022 Copado | Page 42

Survey Methodology
 Who Did We Hear From?
Figure 10

Copado surveyed 275+ executives, managers Work Role

and members of Salesforce development teams in
2021 to learn about their testing procedures. Most Salesforce Developer Quality Engineer or
Assurance or Testing
Release
Management or
Salesforce Admin

respondents were from companies with 1,000+

41% DevOps 6%
21%
7%
employees — including global system integrators
with hundreds of thousands of employees.

Non-trivial analysis was done on the Tableau

Analytics platform to enable data visualization,
cleansing and cross tab analysis. For some
I manage a team I am a conultant, Sales Product I oversee
who builds coach, or trainer Engineer, Mgr. or BA multiple
directly on Sales, or teams who

numeric questions, we simply asked respondents Salesforce 5% Marketing 3% build on

Salesforce
6%
what range they fell in to keep things simple and
3%
3%

avoid collecting company proprietary information. Proj.

Mgr.
I am a
C-level
IT Ops. Support I oversee
teams,

We used the midpoint of that range to crunch our or

Scrum
exec. 1% 1% mult.
platforms

numbers.
Mstr. 2%
1%
2%

© 2022 Copado | Page 44

 Who Did We Hear From?
Figure 11 Figure 12

Industry Roughly how many Salesforce users are in your

company?
Technology Financial Services Retail Manufacturing

46% 13% 7% 5%

17%

19%
10%

13%

18%
23%
1-100 101-500 501- 1,001- 5,001- 10,001+
1,000 5,000 10,000

Insurance Government Education

Healthcare & Pharmaceuticals Figure 13
3% 3% 3%
12%

Region
Energy Telecom. Media

2% 2% 2%
42% AMER

32% APAC

21% EMEA

5% LATAM

© 2022 Copado | Page 45

 What’s Next?

Low-code platforms like Salesforce will continue to

dominate the digital landscape as every business
with a pulse migrates its infrastructure to the cloud.
The speed and ease of building on these platforms is
unmatched — but they require robust testing to make
sure changes don’t break existing systems.

It remains early days for testing these low-code

platforms. Per our research, the majority of teams
still mainly rely on manual efforts. But the rise of
DevOps and the acceleration of change means that
automated testing is the next frontier for today’s
digital businesses.

© 2022 Copado | Page 46

www.copado.com