Survey

A SANS Survey:
Rethinking the
Sec in DevSecOps:
Security as Code
Written by Jim Bird and Eric Johnson
Advisor: Frank Kim
June 2021

©2021 SANS™ Institute

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 1
Executive Summary
As IT workloads move to the cloud, organizations face a fundamental shift in how to
develop and deliver systems—and in their security practices. Deploying and running
production systems has become abstracted from the underlying hardware and network.
Infrastructure is defined through code, and operations work through cloud service APIs.

Security has moved away from selecting and implementing network

DevSecOps: Security as Code
appliances and writing checklists to Security as Code: reviewing
infrastructure and service configuration templates, understanding how to Deploying and operating services in a
cloud-native environment requires that we
correctly use cloud security services and APIs, and writing automated tests
understand and write code. We abstract
and continuous compliance policies. and present operations functions through
Security professionals need to know how to read and write code. They must services and API calls. System and network
implementation and configuration occurs via
understand and use modern software development tools to catch security
code. Cloud providers, by abstracting the work
vulnerabilities and to build guardrails and secure defaults into software involved in deploying and running services
during code development. They need to understand and use Continuous through a set of service API calls, have made
Integration/Continuous Delivery (CI/CD) build pipelines and programmable operations and security into a coding problem.

configuration management tools to automatically check and enforce Software engineering practices for writing
and building good code extend from software
security and compliance policies on every change. They need to understand
development to system and network
different cloud architectures and platforms, including both their strengths engineering and even to security. Thus, security
and weaknesses. They also need to do all of this at high velocity, without becomes security engineering: writing security
getting in the way of delivery. and compliance policies in code; reviewing,
scanning, and testing application code and
Security as Code represents the future of security. What does this mean service configurations; and understanding
to security professionals, to their priorities, to their training, and to the how application development and system
investments that they make in technology and tooling? engineering teams work and helping them to
find and implement tools to inject security
This survey, the eighth in an annual series that focuses on application testing directly into development.
security and DevOps, examines the following with regard to DevSecOps in
Security as Code requires new skills and
the cloud: new ways of thinking and working: more
collaborative and transparent, faster, and more
• What do security teams need to understand about software
iterative. It requires leaning on automation to
development to meet the demand of high-velocity delivery? solve common problems and to reduce costs
• What skills enable security teams to architect secure cloud services and risks.

and ensure that they catch and fix vulnerabilities as early as

possible?

• What impact do the different cloud architectures and platforms have

on this effort, including risks, strengths, and weaknesses?

SANS surveyed 281 organizations across the world. Figure 1 on the next page provides a
snapshot of the demographics of the survey respondents.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 2

 Top 4 Industries Represented Organizational Size

Banking and Small

finance (Up to 1,000)

Small/Medium
Cybersecurity (1,001–5,000)

Application Medium
development (5,001–15,000)
firm
Medium/Large
Healthcare (15,001–50,000)

Each gear represents 10 respondents. Large

(More than 50,000)

Operations and Headquarters Each building represents 10 respondents.

Ops: 79
Top 4 Roles Represented
HQ: 9 Ops: 76
HQ: 8 Security administrator/
Ops: 96
Security analyst
HQ: 34

Security manager
or director

Ops: 227 Ops: 56 Security architect

HQ: 198 HQ: 12

Ops: 35
Ops: 49 IT manager or director
Ops: 53 HQ: 2
HQ: 12
HQ: 6
Each person represents 5 respondents.

Figure 1. Survey Demographics

How prepared are organizations for these changes?

Key Findings
Cloud platforming:

• More than half (57%) of organizations use three or more cloud platforms. Each cloud
platform is unique: The configuration models differ, as do the APIs and services.
Therefore, operational and security risks differ, making them difficult to understand
and manage. Increasingly, cloud-agnostic tools help to reduce costs and risks.

• Only a third (34%) of organizations have automated cloud configuration through

code and platform APIs. Manual configuration is slow, less traceable, and more
error prone. Organizations need to increase automation of configuration through
programmatic infrastructure as code tooling and build pipelines to keep up with
rapid change—and to help ensure continuous compliance.

Velocity of delivery and security testing:

• More organizations are taking advantage of the cloud, DevOps, and lightweight Agile
practices and tools to deliver features and changes to production faster and more
cost-effectively. The velocity of IT delivery has increased by 14% over the past five
years, but the speed of security assessments has failed to keep up. Security testing
continues to lag.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 3

 • Only 29% of organizations have automated most (75% or more) of their security
testing. Fewer than half of organizations (44%) have included security tests and
reviews as part of coding workflows.

• Foundational software development practices such as CI/CD and test automation

are key to delivery velocity and continuous security testing. If development
teams do not automate their build/test work, they will find it more difficult to
implement automated security testing. Whereas 66% of organizations currently
automate builds, only half of organizations (52%) follow CI and take advantage of
automated testing.

Operations:

• As delivery teams continue to get faster, so do attackers. Only half of organizations
(51%) patch or otherwise resolve critical security vulnerabilities and other critical
security risks within a week of identifying these risks. Organizations need to leverage
DevOps and Agile practices, and automated build chains and automated testing, to
get patches out faster with confidence.

• Organizations, especially organizations that cannot keep up with security testing,

need to consider the value of runtime shielding for cloud platforms, to provide
continuous protection against configuration and deployment mistakes, emerging
vulnerabilities, and evolving threats. Only a third of organizations (33%) rely on
operational runtime protection solutions in the cloud. Using Security as Code
principles, organizations can integrate protection services such as cloud web
application firewalls (WAFs) with serverless functions to parse logs, identify
scanners, and automatically block bad IPs.

Barriers and enablers:

• Successfully implementing DevSecOps is not a technical problem; it is an

organizational problem. Lack of resources, lack of management and developer
buy-in, bureaucracy, poor communication across silos, and poor prioritization hold
organizations back, not technology.

• Training in secure coding practices is a key enabler—not just training for developers,
but also training for operations, and security, and even compliance personnel.
Understanding how to write secure code requires a fundamental change in mindset
for everyone involved in IT in the cloud.

Understanding the Cloud Landscape

Mapping out the cloud landscape—the extent of cloud services adoption, the cloud
platforms’ runtime architectures used, and how applications are built and deployed
to the cloud—helps security teams understand the potential risks that organizations
need to manage.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 4

Cloud Platform Analysis: The Big 3
A majority of survey respondents (63%) indicated that they spend at least half of their
time on public cloud security and operational responsibilities. See Figure 2.

As DevOps teams move their workloads into the cloud,

What percentage of your time is spent on cloud-related architecture,
security teams need to learn how to apply operations, security, or development?
monitoring, and runtime security controls across public
40%
cloud providers. As with previous surveys, the Big 3 33.1%
cloud providers—Amazon Web Services (AWS), Microsoft 30%

Azure, and Google Cloud Platform (GCP)—continue 22.1%

20% 17.4%
to dominate the public cloud space. AWS is in use at 16.4%
86% of respondents’ organizations, with 27% of those
10% 7.5%
organizations hosting 75% or more of their applications 3.6%
in the AWS cloud. See Figure 3. 0%
0% 1–24% 25–49% 50–74% 75–99% 100%
Microsoft Azure continues to close the gap, with 81% Figure 2. Time Spent in the Cloud
of respondents using Azure. However, only 18% of those respondents depend heavily
(75% or more) on Azure for the running their workloads. GCP remains in third, with only
61% adoption. Perhaps the most telling gap in the numbers is that only 7% of the GCP
users depend on it for 75% or more
of their workloads. Almost a quarter What percentage of your cloud-based applications are hosted by:
of respondents (22%) run at least 1–24% 25–49% 50–74% 75–99% 100%
some part of their workloads on cloud 29.2%
30%
platforms outside of the Big 3. 24.9%

Many organizations, especially large 20.9%

20% 19.5% 18.8%
16.6% 17.0% 17.7%
enterprises, work with multiple cloud 14.1% 14.4%
12.3%
platform providers. SANS found that
10% 9.4%
most organizations (97%) use at least 5.8% 5.8% 5.8% 6.1% 6.5%
4.3%
one public cloud provider, with more 0.7%
0.4%
0%
half than (57%) running workloads on Other Microsoft Azure Google Cloud Amazon Web
Platform (GCP) Services (AWS)
three or more public cloud providers.
Figure 3. Cloud Hosting Providers

TAKEAWAY
As the technologies and options multiply in heterogenous cloud environments, so do the risks. Security teams must
learn different security and data privacy models, identity management and access control schemes, data storage and
encryption capabilities, activity auditing functions, configuration languages and defaults, compliance and security
controls and APIs, and the architectural strengths and weaknesses for each provider’s service platforms.1
Having multiple options increases the value of tools and security solutions that work across different cloud service
providers (CSPs). For example, Terraform, an open source platform for managing cloud services, enables teams to
configure and provision services across multiple cloud platforms using the same toolset and one high-level language
syntax (HCL), which reduces development and operational costs and security costs and risks. However, abstraction
comes with a downside. You may run into problems configuring and provisioning services where it is unclear if there
are mistakes in your Terraform code or bugs or limitations in the Terraform providers (API wrapper) you use.

1
For more on multicloud security considerations, see www.sans.org/reading-room/whitepapers/cloud/top-5-considerations-multicloud-security-39505

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 5

Leveraging Cloud Services to Reduce Cost and Risk
With public cloud adoption rising, organizations are transitioning workloads from on-
premises to a mix of cloud-hosted virtual machines (VMs), cloud-hosted container
services, and cloud-hosted serverless platforms. As Figure 4 shows, organizations run
roughly the same amount of work on cloud-hosted VMs and container services as on-
premises. They also use serverless platforms for activities like back-end batch processing
and task scheduling.

What percentage of your applications are running in the following methods:

0% 1–24% 25–49% 50–74% 75–99% 100%

29.9% 30.3%
30% 27.9% 28.3%
26.7% 26.7%
25.1% 24.7%
23.5% 23.5%
20.7%
20% 18.3% 17.5%
14.7% 15.9%
12.4%
10.8% 10.8%
10% 9.2% 9.2%
6.8% 7.6%
6.0% 5.6% 5.6%
2.8% 2.4% 2.0% 2.4%
0.8%
0%
On-premises Cloud-hosted Cloud-hosted Cloud-hosted Functions-as- Other
virtual machine container service a-Service (FaaS) (serverless)

Figure 4. Runtime Platforms

As organizations move from on-premises infrastructure to cloud-based services,
operational complexity and scale, development cost, compliance responsibilities, and
security risk shift to the cloud platform:

• O
 n-premises—The organization is responsible for managing everything.

• C
 loud VMs—The organization is responsible for managing cloud operations, security,
and configuration management. This includes carefully checking the VM platform:
Each cloud provider auto-provisions default network and firewall rules for new VMs,
which may unintentionally expose private resources.

• C
 loud container services—The CSP is responsible for patching the VM and
provisioning and hardening the container runtime. DevOps teams (and security)
can focus on the application and its build/runtime dependencies, packaged and
shipped in container images.

• C
 loud serverless—The CSP is responsible for hardening container images and
patching the underlying application runtime shift to the cloud provider. DevOps
teams need to worry only about writing and testing application code, the
permissions associated with the function’s service account, and major upgrades to
the application runtime environment. However, even careful coding and testing is
not enough to protect your code against common attacks, and a lack of visibility into
the serverless runtime makes it difficult to understand what attacks you are facing.
This is where cloud workload protection platforms (CWPPs) and other runtime
protection solutions come in.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 6

Securing Cloud Container
The Cloud in Code
Services
The concept of treating infrastructure as code was introduced to help manage runtime
Containers enable developers to infrastructure in on-premises data centers. Tools like Ansible, Chef, and Puppet allow
package an application and its operations engineers and developers to configure and provision runtime technology stacks
(i.e., operating systems, VMs, and network and database configurations) in high-level code.
runtime dependencies in a simple,
portable way, and they provide a The cloud takes this further. DevOps teams can provision their own IT infrastructure
without needing to understand the details of the underlying technology (i.e., the
measure of isolation at runtime.
servers, storage, and network) or having to operate it, reducing cost and time to roll
These capabilities make containers out application services. High-level, declarative configuration languages, such as AWS
a core technology in DevOps. But CloudFormation, Azure Resource Manager (ARM), and Google Deploy Manager (GDM),
as container use has exploded, cross-platform tools such as Terraform, and toolsets for containers enable DevOps
teams to model, configure, and deploy cloud technology stacks in the same way that
teams need orchestration tools like
developers build and deploy applications, through automated pipelines. You can make
Kubernetes to deploy, manage, and iterative and incremental changes to runtime configurations safely and roll out the
operate containers at scale. changes across your infrastructure at velocity and scale. All changes are versioned and
audited and repeatable.
Installing, configuring, hardening,
As with application code, DevSecOps teams can leverage static analysis tools to find
and managing an on-premises
problems in cloud configurations at build time, before changes are rolled out. Modern
container cluster is a heavy lift and
scanning tools look beyond syntax checking and basic semantic analysis to check for
introduces a new set of operational common security mistakes and vulnerabilities, and to validate configurations against
costs and security risks. It is easier compliance policies, up and down the stack (e.g., cloud services, containers, and
and less expensive to shift most of Kubernetes configuration), and in some cases across cloud platforms. Open source
examples include the following:
these responsibilities and costs to
• terrascan: https://github.com/accurics/terrascan
a proven CSP, using a containers-
• checkov: https://github.com/bridgecrewio/checkov
as-a-service (CaaS) platform.
• tfsec: https://github.com/tfsec/tfsec
CaaS platforms, such as Google
• cfn_nag: https://github.com/stelligent/cfn_nag
Kubernetes Engine (GKE), Amazon
• kics: https://github.com/Checkmarx/kics
Elastic Container Service (ECS),
The benefits of configuring the cloud in code are obvious. However, a high learning
Azure Kubernetes Service (AKS), and curve applies to understanding cloud architecture and cloud configuration languages
Red Hat OpenShift, abstract the and coding practices and tools at the same time. DevOps teams and security teams
details of deploying and running need time to learn the concepts and run technical proof-of-concept exercises to
evaluate the tools and identify risks.
containers, letting developers focus
on what goes into the container
rather than how the container itself Which container orchestration tools are managing your production workloads?
works. See Figure 5. Select all that apply.

Cloud-hosted Kubernetes
(e.g., EC2, Azure VM, GCE) 47.8%
Cloud-hosted Docker
(e.g., EC2, Azure VM, GCE) 43.9%
Cloud-managed container service
(e.g., AWS ECS, AWS Fargate, Azure Container) 37.6%

On-premise Docker Engine 30.6%

On-premise Kubernetes 30.6%

Cloud-managed Kubernetes service
(e.g., AWS EKS/Fargate, Azure AKS, Google GKE) 30.2%

Docker Swarm 7.5%

Open Shift 6.3%

Other 4.3%
0% 10% 20% 30% 40% 50%

Figure 5. Container Orchestration Platforms in Use

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 7

While the most popular platforms for managing containers are now based in the cloud,
a still significant number of on-premises hosted container platforms exist, likely due to
investments made before cloud-hosted CaaS became available. Simple Docker Engine
implementations are common in development and testing—and for security—because
security tools are often packaged in Docker containers.

TAKEAWAY
Managed container service platforms still require careful review by DevSecOps engineers:
• D
 evSecOps engineers need to evaluate container platforms for misconfigurations and
insecure defaults. For example, CaaS resources created through the cloud web console
might run workloads in default cloud-managed virtual private cloud (VPC) networks without
traditional network security controls such as network logging and egress traffic firewall rules.
• C
 ontainer images stored in cloud-managed registries are not automatically scanned for
vulnerabilities. Cloud security teams need to understand each cloud provider’s image
scanning capabilities and know how to enable this.
• C
 ontainer workloads can be assigned an identity (or service account) with excessive
permissions that can allow a compromised container to move laterally through the cloud
account and perform data exfiltration.
• R
 untime detection and incident response tools do not Which languages and platforms in your application portfolio have
automatically monitor compromised container workloads. been the greatest source of risk or exposure to your organization?
If a workload is in fact compromised, having the ability to Select your top three.
immediately alert (if a malicious actor actually performed
data exfiltration) and to capture a detailed activity record Python 29.4%
are critical.
C/C++ 27.4%
Managing these risks requires security engineers to carefully
JavaScript 26.6%
review the container service configuration, create secure by-
default templates (e.g., Terraform modules) for development PHP 26.2%

teams, and have runtime threat detection and response Java 26.2%
capabilities in place.
.NET 23.4%

COBOL 21.4%
Programming Environments and Risks C# 18.3%
As development teams move to the cloud, they adopt HTML 17.5%
new programming languages and tools. With these new
Android 16.7%
capabilities come new risks, as shown in Figure 6.
Go 14.7%
The security risk of any programming language generally
Perl 13.9%
tracks its popularity of use:2
Groovy 11.1%
• S
 cripting languages—Languages such as JavaScript
Ruby 9.1%
and Python are easy to learn and use, making
them increasingly popular in cloud environments, Objective C/Swift 7.9%

especially with nonprofessional programmers, who Other 4.4%

are more likely to make dangerous programming Rust 4.0%
mistakes. These languages also make extensive use 2.0%
Scala
of open source software packages (e.g., npm, pip), 0% 5% 10% 15% 20% 25% 30%
which introduce another class of vulnerabilities that
Figure 6. Risk in Programming Languages and Platforms
we must manage.

2
A useful indication of popularity of use is GitHub’s “State of the Octoverse,” which ranks development languages used in open source projects:
https://octoverse.github.com

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 8

 • M
 emory-unsafe languages—Languages such as C (especially) and C++ will continue,
by definition, to present a major source of security risk. Static analysis checking
and manual code reviews are essential to catch coding vulnerabilities and enforce
practices such as the Computer Emergency Readiness Team (CERT) secure coding
standards.3

• Java and PHP—These have been mainstays of web development, and although
the use of these languages has declined over time, many organizations still have
significant legacy investments in this code, as well as code written in legacy
languages like COBOL.

Security practices and tooling must be adapted to the needs of development teams as
well as the development environments, languages, and frameworks that these teams use.

TAKEAWAY
Newer coding languages and frameworks, and special-purpose languages (including configuration template languages), come with
new risks: a lack of secure coding guidelines, limited integrated development environment (IDE) support, and limited tooling for static
analysis and software component analysis (SCA). In these cases, developers need to fall back on general defensive coding practices and
rely on code reviews, including third-party expert reviews, and other kinds of testing to find security problems. Security teams should
identify the risks of working with new or niche languages early, in upfront risk assessments, as they evaluate development tools and
architecture candidates.

Security at Velocity
High-velocity, incremental delivery enables development teams to innovate and run
experiments at low cost, collect feedback, respond, and change. Founded on these
priorities, modern tooling and lightweight Agile practices minimize friction and cost and
encourage teams to learn and change as they go. However, the relentless focus on velocity
of delivery can force DevOps teams to take shortcuts, putting speed ahead of everything
else, taking lightweight too far. Teams build up all kinds of technical debt: code that is not
clean and safe to change, lack of test coverage, and outdated dependencies. The faster
that teams move, the more debt they build up.4

Continuous change means that the targets and risks for security and compliance
constantly change. Security must become agile and iterative:

• R
 uthlessly automate security testing to keep pace with delivery. However, the
emphasis on speed and fidelity/accuracy means that you will not catch some
problems. You need to recognize these risks, and add deep scanning, fuzzing, and
penetration testing or bug bounties to find problems that fast, incremental scanning
and testing can’t find.

3
https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=494934
4
“2017 SANS State of Application Security: Balancing Speed and Risk,” October 2017,
www.sans.org/reading-room/whitepapers/analyst/2017-state-application-security-balancing-speed-risk-38100

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 9

 • C
 ontinuously enforce compliance on every change. You cannot rely on manual
checklists and audits if you change code and infrastructure every day; you will find
it too difficult to go back and review so many changes by hand. Encode compliance
rules and test them automatically.

• B
 ecause code and third-party dependencies and cloud configurations constantly
change, you need to track and understand all of this code and how it is built,
and think of code and the code build infrastructure as part of the organization’s
attack surface.

• W
 orking in an iterative, incremental way means that everyone (including the security
teams) “learn on the go.” Look out for incorrect assumptions, and for holes in
knowledge and understanding on the part of the delivery teams and on the part of
the security team (you don’t know what you don’t know). Be prepared to respond
quickly to new knowledge and to new risks.

To understand how organizations manage these risks and challenges, we asked a series
of questions:

• H
 ow often do organizations deliver changes to production?

• H
 ow do they deliver these changes? What technical practices and toolchains do
they use?

• H
 ow often do they test or assess security?

• H
 ow much of this testing is automated?

• W
 hat security tools and practices do they rely on to manage security risks?

We describe the survey results and our findings related to these questions in the
following sections.

Delivery Velocity Is Accelerating

The velocity at which organizations deliver IT changes continues to accelerate. In the early
days of Agile development, Scrum teams and XP teams delivered working software every
month, which most considered radical at the time. Today, 75% of organizations deliver
changes more than once per month, an increase of 14% over the past 5 years. See Figure 7.

Frequency of delivery to production

2017 2018 2020 2021

30%

20%

10%

0%
Other/Ad hoc More than Annually Quarterly Monthly More than Weekly Daily Continuously
1x year 1x month

Figure 7. Velocity of Delivery to Production 2017–2021

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 10

DevOps Foundational Practices
To deliver changes at high velocity, DevOps teams follow a core set of common technical
practices and automation technologies, including those shown in Figure 8. More than half
of this year’s respondents indicated that their organizations follow iterative, incremental
design and development using core
Which of the following practices does your organization follow?
Agile and DevOps technical practices,
Select all that apply.
such as automated builds (66%) and CI
(52%). See Figure 8. Build automation 66.2%

However, close to half of respondents’ Automated testing 51.6%

organizations do not use automated Continuous Integration (CI) 51.6%

testing. This implies that they still rely Automated deployment 43.1%
on manual testing (which does not Microservices-based architecture 40.9%
scale and cannot keep up with rapid, Continuous monitoring and measurement 35.6%
iterative development and delivery)
Continuous Deployment (CD) to production 35.6%
or, worse, that they do not test at all. Programmable configuration
management/Infrastructure as code 33.8%
And only 34% are provisioning and
Immutable infrastructure provisioning 21.7%
configuring infrastructure using modern
Blameless retrospectives 11.7%
programmatic tools, such as Chef,
Terraform, or AWS CloudFormation, Chaos engineering 5.0%

instead of point-and-click admin Other 2.1%

0% 10% 20% 30% 40% 50% 60% 70%
console interfaces.
Figure 8. Technical Practices
A lot of room exists to improve automation, by moving manual work into code. Consider and Automation Technologies
the following foundational practices and what they mean from a security perspective: in Use

• I ncremental design and development—Modern Agile development imposes

constraints on how security is practiced: must be lightweight, performed in-phase,
iterative, and continuous.

• M
 icroservices-based architecture—A common architectural pattern for cloud
applications, where small teams work independently, designing and delivering
small services with each offering discrete responsibilities through APIs. This trades
off development simplicity for operational complexity and greatly increases the
system’s attack surface. Common security risks include holes in API authentication
and access control. Protecting these applications requires close collaboration
between development and runtime security approaches, including next-generation
web application firewalls (NGWAFs), Runtime Application Self-Protection (RASP),
CWPP, and API fuzzing.

• B
 lameless postmortems—Retrospectives extended from development to operations
to learn from outages and other operational problems and near misses. Teams work
together to walk through incidents and incident response and to conduct root cause
analysis to identify improvements. These practices can (and should) be extended to
security breaches, compliance violations, and near misses.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 11

 • A
 utomated testing—Developers write their own automated tests that execute on
each check-in, to provide confidence in builds and iterative changes, ensuring that
code is not broken by accident (regressions). Automated testing is a key prerequisite
for continuous security testing. If developers have not already widely adopted
automated testing, introducing automated security testing into their workflows will
prove problematic.

• C
 ontinuous monitoring—Infrastructure and culture to detect and respond to
exceptions can be used to enlist DevOps teams in monitoring for evidence of attacks
on application services and infrastructure and on other security events.

• P
 rogrammatic configuration/infrastructure as code—Configuring cloud
infrastructure in code creates a control structure for security and compliance. We
can check configuration changes into version control, and automatically scan and
test using CI/CD build pipelines to catch configuration errors and enforce hardening
policies. It also provides transparency into runtime architecture and configuration,
making it easier to understand and assess risks.

• I mmutable infrastructure—Changes are made by tearing down and rebuilding

runtime instances (for example, using containers). Prevents configuration drift,
prevents tampering with runtimes, and ensures that infrastructure configuration is
known and traceable.

• A
 utomated deployment—Make deployment steps repeatable, testable, automated,
auditable, and safe. Enforces change control and reduces the cost and risk of
making changes, including deploying patches.

• B
 lue/green deployment—An incremental deployment technique to reduce the cost
and risk of rolling out changes.

• C
 haos testing—Operational failure injection in running systems, either in test or
in production. Netflix made this concept famous through their “Chaos Monkey”5
service. Effective at uncovering problems in technical architecture and for incident
detection and response. Chaos testing ensures that security controls remain
resilient to DOS attacks and other types of attacks.

• B
 uild automation/CI/CD—Moving from manual builds to automated build chains,
and then to Continuous Integration and Continuous Delivery or Continuous
Deployment, is a prerequisite for iterative and incremental development.
Standardizing and streamlining these steps, making them repeatable and
transparent, reduces costs and risks for the delivery teams (and for security).

Security and compliance can build on and leverage these enabling practices and tools—if
we have the practices and tools in place and working effectively.

5
https://netflix.github.io/chaosmonkey

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 12

CI/CD
Iterative, Agile development requires a repeatable, automated build process.
Continuous Integration (CI) is one of the foundational practices in modern
software development. With CI, each time code changes are checked in, the
code is automatically scanned, built, and
Which continuous Integration tools are you using to automate your build
tested, providing developers with immediate
and release workflows? Select all that apply.
feedback on its fitness. DevOps teams
Cloud hosted
use Continuous Delivery (CD) to automate (e.g., GitHub Actions, GitLab CI) 55.1%
packaging, configuration, and deployment, On-premises open source
49.4%
(e.g., Jenkins)
so that each change can be automatically On-premises commercial
46.4%
(e.g., TeamCity, Bamboo, Travis, CircleCI)
pushed directly to production. Cloud native (e.g., AWS CodePipeline,
Microsoft Azure DevOps) 46.1%
The CI/CD pipeline provides a control plane Other 2.6%
for both software engineering teams and 0% 10% 20% 30% 40% 50% 60%
security and compliance. As shown in the Figure 9. Continuous Integration
survey results in Figure 9, a roughly 50/50 split exists between on-premises Tools in Use
CI/CD and cloud-hosted CI/CD services. Cloud-hosted platforms enable
DevOps teams to offload the responsibilities and costs for provisioning,
configuring, hardening, monitoring, and managing these tool sets onto
the CSP, allowing teams to focus on the work of coding instead of on the
undifferentiated heavy lifting involved
in setting up and managing the build Protecting the CI/CD Pipeline
infrastructure. Automated build and deployment pipelines provide a direct path from
development to production, making them a tasty target for attackers. If
GitHub and GitLab offer managed source
attackers can compromise the repositories or build chain, they can inject
code control and CI/CD as a service. These malware into production systems without penetrating them. The CI/CD
services offload the work of managing environment all need to be protected: the source and artifact repositories, the
code repositories and check-in workflows CI/CD toolchain configuration and runtime, the keys and other secrets used,
and the infrastructure for running these services.
and of creating, integrating, and running
The recent attack on SolarWinds’s customers through that company’s build
automated build chains. Their enterprise-
environment, where attackers compromised SolarWinds’s automated build
tier services include built-in code scanning
process to insert a backdoor that was then distributed to customers, has
for most common programming languages, proven the seriousness of the risks. According to a statement from SolarWinds’s
secrets scanning, dependency analysis, new CEO, the company is taking the following steps to improve the security of
and other security testing capabilities. its development and build environments:6
• I ncreased hardening of the development and build environment and
Development teams, and security and
build pipelines
compliance teams, can take advantage
• Auditing and surveillance of the development and build processes
of these capabilities out of the box,
• S
 canning and analysis of source code and build artifacts to ensure that
lowering the bar of entry for security
they match
testing, and incrementally add additional
• E
 nforcing multifactor authentication (MFA) and strict access controls across
testing solutions for higher coverage using the development and build environment
standard APIs.
These steps offer a roadmap to all organizations on how to protect their own
CI/CD environments.

6
https://orangematter.solarwinds.com/2021/01/07/our-plan-for-a-safer-solarwinds-and-customer-community

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 13

Let’s now look at speed of delivery from the security perspective.

Is Security Keeping Up with the Velocity of Change?

The faster that engineering and development teams deliver changes, the faster security
teams need to identify and assess risks. By comparing the velocity of delivery to the
velocity of security testing, as shown in Figure 10, we see that most organizations cannot
keep up with the pace of delivery.

Velocity of Delivery vs. Velocity of Security Assessment

Security Assessment/Test Delivery

30%

20%

10%

0%
Other Ad hoc Annually Quarterly Monthly More than once Weekly Daily Continuously
per month

Figure 10. Velocity of Delivery vs.

Without testing, we leave vulnerabilities undetected. The earlier in the delivery life cycle Velocity of Security Assessment
that we identify vulnerabilities, the easier and cheaper we can fix them (along with a
higher likelihood that we will do so)—the whole point of shifting security left.

When Is Security Testing When do you perform security testing in your build and release pipeline workflow?
Performed in DevOps? Select all that apply.

As shown in Figure 11, security testing Architecture/Design

(risk assessments, threat modeling) 48.8%
occurs multiple times during the Code commit/Pull request
(code reviews, scanning) 43.8%
development cycle, from upfront
Coding
43.8%
requirements to design, through coding (IDE security plug-ins)

and developer/QA testing workflows, Unit testing in build/CI 39.5%

Requirements/Use case
and into the predeployment and (security stories, misuse cases) 38.4%

deployment stages. QA/Acceptance testing 38.0%

Periodic testing outside of release cycle
However, fewer than half of (e.g., scheduled pen testing) 32.9%
Operational run-time protection/shielding
organizations (49%) have shifted (e.g, WAF, RASP, CWPP) 32.6%
Release gate review/approval
security reviews into requirements, by security/compliance
29.1%
design, and coding workflows, and 29% Automated/continuous compliance
19.4%
policy enforcement
still rely on release gate reviews, which Continuous testing outside of the release cycle
6.6%
(e.g., fuzz testing, chaos engineering)
cannot keep up with high-velocity CD. None 2.3%
0% 10% 20% 30% 40% 50%

Figure 11. When Security Testing Is Done in Development/Deployment

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 14

How Much of Security Testing Is What percentage of security testing is automated?
Automated? 40% 37.5%
Obviously, because manual reviews and pen testing
30%
cannot keep up with continuous changes in high-
24.2%
velocity developments, organizations require some
20%
16.8%
level of automated testing. Test automation enables 14.1%
continuous testing, increasing confidence on each 10%
5.1%
change. Unfortunately, only 29% of respondents 2.3%
0%
indicated that they have automated the majority (75% 0% 1–24% 25–49% 50–74% 75–99% 100%
or more) of their security testing, as shown in Figure 12. Figure 12. Automation of
Security Testing
A contributing factor to this is that only 52% of
organizations automate testing of any kind. Before
Hijacking Developer Testing
DevOps teams will accept automated security testing,
As development teams adopt test automation, you can hijack the
organizations need testing discipline, automated test
work that developers are already doing, the tests that they are
infrastructure, and pipeline workflows in place.
already writing, for security purposes. For example, unit tests are
Test coverage determines your level of confidence in automated tests written by developers to exercise code logic, which
generally make up most of the tests in an automated test suite.
a team’s ability to quicky and safely make changes,
These tests can play an important role in security testing. Ensure
including rolling out patches. Tracking test coverage that you have good coverage for security-critical code, including
also highlights risks associated with areas in code authentication, encryption, logging, and especially access control/
that cannot be trusted as correct. permissions (which is driven by business logic and organizational
structure). Also, encourage developers to get off the happy path7
and write negative tests (tests that should always fail) to exercise
error/exception handling code. Doing so makes the code more
robust and better protected from attackers.

TAKEAWAY
Lack of automation is a key reason that security does not keep up with the pace of delivery. Automation needs
commitment from development/engineering teams and their managers. Such commitment proves easier with
modern CI/CD platforms that include some security testing capabilities and that make it easy to add more testing.

Some of the different test techniques and tools also build on each other:

• D
 ynamic Application Security Testing (DAST)—We can run testing as a separate
testing stage (inside or outside of the build pipeline) or by proxying automated
functional tests (e.g., Selenium WebDriver test suites) through an attack scanner to
include passive and dynamic scanning of web interfaces. The coverage of these tests
relies on the functional tests (and the amount of time we allow the tests to run).

• I nteractive Application Security Testing (IAST)—Results also depend on the

coverage of the automated test suite to catch security vulnerabilities, by analyzing
code paths while we exercise the code.

7
https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=29002

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 15

The type and success of testing techniques and tools depends on who holds responsibility
for security testing:

• D
 evelopment teams—Static Application Security Testing (SAST) and SCA code
scanners (and container scanners) prove a natural fit provided that the tools are
fast and accurate. Automated unit testing is also a developer responsibility, although
security teams can help by identifying areas of code that need coverage and by
writing negative tests.

• S
 ecurity teams—DAST and fuzz testing prove popular with security professionals,
who tend to think more like pen testers and less like developers.

This leaves testing technology like IAST in an awkward, in-between situation—one reason
why IAST represents a niche solution. Implementing IAST requires a higher level of
maturity on the part of DevOps teams. It also calls for cooperation between development,
operations, and security to set up the runtime, manage the tests, and own the results.

Who Is Responsible for Who is responsible for conducting security testing in your organization?
Security Testing? Select all that apply to your organization.

Who owns writing the tests, running

Internal security team 58.0%
them, and keeping them up to date
External consultants 42.4%
within most organizations? Have
QA/Test teams 39.3%
organizations shifted responsibility for
Developers/software engineers 35.8%
testing left onto DevOps teams, or do
they still silo testing in the security team External cloud-based security testing platforms 26.8%

or outsource it to security consultants or Cross-functional DevSecOps teams 24.5%

bug bounty testers? As shown in Figure Customers/users 9.3%

13, in most organizations the security Bug bounty hunters 8.9%
team—whether an internal security team Other 1.6%
(58%) or external security consultants 0% 10% 20% 30% 40% 50% 60%
(42%)—still owns the responsibility for Figure 13. Responsibility for
security testing. Even with increasing automation, however, these small teams cannot keep Security Testing
up with the increased speed of delivery.

TAKEAWAY
The key to scaling security testing is to shift this work to development or DevOps teams. However, only 36% of
organizations have done this. The more that security teams understand the code, and the build and delivery
processes and technologies, the more they can help DevOps teams take ownership for testing their own code.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 16

Compliance as Code What percentage of compliance policies are
checked/enforced automatically?
Just as security is moving to code, so is compliance.
Instead of following paper checklists and enduring 31.5%
30%
manual audits, organizations are taking advantage of 25.2%
configuration in code, automated build and deployment 20% 17.3%
15.7%
pipelines, and automated testing to reduce the time and
cost required for compliance reviews and audits, while 10%
5.1% 5.1%
at the same time providing a higher level of compliance
0%
assurance. DevOps teams can demonstrate their staying 0% 1–24% 25–49% 50–74% 75–99% 100%

continuously onside of compliance, automatically Figure 14. Percentage of Compliance

checking and enforcing policies on every change. See Policies Enforced Automatically

Figure 14.

Surprisingly, more than half of organizations

Compliance in Code Tooling
(62%) automatically enforce 50% or greater of
In addition to the tests and automated build infrastructure that DevOps
their compliance policies, taking advantage of teams already work with, special-purpose tools make it easy to express
their investments in automation to perform compliance requirements and policies into code, at a level of abstraction that
continuous compliance. both developers and auditors can understand. Developers and auditors can
write asserts, or guardrails. They can add these concrete pass/fail checks to
Moving compliance into code means build and delivery pipelines, with metadata that tie the tests back to specific
that DevOps teams can have more direct policies or compliance standards, making them traceable for auditors.
ownership of compliance responsibilities. Organizations now have expanding choices of high-level toolsets for authoring
Rather than reading through generic compliance tests. Using open source tools like AWS CloudFormation Guard,
guidelines and responding to auditor Chef InSpec, Conftest, Dev-sec.io, or Terraform Compliance, developers can
write simple tests, and walk auditors through the code and the automated test
requests, developers can add tests and
results, to prove that they follow policies consistently and continuously.
asserts into their build pipelines and can take
advantage of the built-in audit trails in version control
and CI/CD to provide compliance evidence. This turns
On average, how long does it take for your organization
bureaucratic overhead into concrete, straightforward to patch/resolve critical security risks/vulnerabilities for
coding work. systems already in use?

Speed remains an important factor in security testing,

Same day 14.1%
and its criticality is even higher when it comes to
Next day 11.0%
resolving vulnerabilities and other security problems
2–7 days 25.9%
when found in production.
8–30 days 20.0%

Is Vulnerability Remediation Keeping Up? 31 days to 3 months 11.4%

When teams find serious vulnerabilities, they need 3–6 months 5.1%

to respond and patch quickly to close the window of 6 months to 1 year 2.0%

exposure. As Figure 15 shows, 71% of respondents said More than a year 0.0%

they patch or remediate critical vulnerabilities within 30 Other 1.6%

days (a common cutoff for regulatory compliance). 0% 5% 10% 15% 20% 25% 30%

Figure 15. Time to Fix and Patch Critical Vulnerabilities

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 17

In the same way that development teams have become more agile and much faster,
however, so have attackers. A 30-day patching cycle that was state of the art 20 years
ago is not good enough today. Attackers exploit critical vulnerabilities within days and
even hours of a vulnerability becoming known.8 Only half (51%) of organizations patch or
remediate critical security risks within a week of identifying the risk.

TAKEAWAY
To outrace attackers, organizations need to take advantage of the high-velocity delivery capabilities of modern Agile development
and DevOps teams, infrastructure in code, automated builds, and CI/CD pipelines to patch code and infrastructure faster and with high
confidence. Teams must also take advantage of visibility into code repos and configuration, and continuous vulnerability scanning at
multiple levels, including SCA for open source dependencies, container and image scanning, scanning applications and configuration
code, and cloud security posture management (CSPM), to catch vulnerabilities as early as possible.
Organizations should use virtual shielding—that is, use runtime protection solutions such as a web application firewall (WAF), next-
generation WAF (NGWAF), RASP, CSPM, or CWPP service—to help minimize the risk of exploits, especially those organizations that cannot
keep up with vulnerability patching.

Vulnerability management Table 1. Usefulness of Security Testing Practices/Tools and Position in Development Cycle
plays a vital role in security
Practice/Technique Very Useful Useful Not Useful Total Useful
programs, but Security as Security training for developers/engineers 29.3% 53.1% 7.8% 82.4%
Code involves much more. Upfront risk assessments before development starts 27.0% 54.3% 5.1% 81.3%
Automated scanning of code for security 27.7% 51.2% 10.2% 78.9%
vulnerabilities and other defects (SAST)
DevSecOps Tools Periodic vulnerability scanning 24.2% 48.0% 15.2% 72.3%
and Practices: What Open source/third-party dependency analysis 24.6% 46.5% 13.7% 71.1%
Works? Third-party penetration testing 25.0% 46.1% 13.3% 71.1%
Network detection and response (NDR)/ 21.9% 49.2% 11.7% 71.1%
Organizations can use a network traffic analysis (NTA)
wide range of security tools Manual code review 24.6% 45.7% 18.4% 70.3%

and practices to understand Threat modeling, attack surface analysis, 22.3% 48.0% 12.1% 70.3%
or architecture/design reviews
and manage risks, both Continuous vulnerability scanning 27.0% 43.0% 11.3% 69.9%
early in development and WAF 29.7% 39.5% 17.6% 69.1%
at runtime. We asked
9 Internal penetration testing 25.8% 41.4% 13.3% 67.2%
Container/image security scanning 25.4% 40.6% 12.5% 66.0%
respondents to identify
Compliance reviews or audits by a third party 19.5% 45.7% 17.6% 65.2%
which tools and techniques NGAF 23.8% 41.0% 10.5% 64.8%
they believe are most useful. File integrity monitoring/host-based intrusion 21.5% 43.4% 13.7% 64.8%
detection system (HIDS)
Table 1 shows the results.
Security stories, abuser stories, or evil-user stories 15.2% 47.3% 12.5% 62.5%
to inject security into requirements backlog
DAST 21.1% 40.2% 10.9% 61.3%
CSPM 17.6% 38.7% 10.9% 56.3%
Virtual patching 15.6% 40.2% 13.7% 55.9%
Fuzz testing 18.8% 35.5% 11.7% 54.3%
IAST 15.2% 38.7% 11.7% 53.9%
RASP 16.4% 35.9% 11.7% 52.3%
CWPPs 15.6% 35.9% 10.9% 51.6%
Bug bounties 16.0% 34.8% 14.5% 50.8%
Other 6.6% 12.1% 5.1% 18.8%

8
www.fireeye.com/blog/threat-research/2020/04/time-between-disclosure-patch-release-and-vulnerability-exploitation.html
9
 ANS has mapped security practices and open source tools in a secure DevOps toolchain, found at
S
www.sans.org/security-resources/posters/cloud-security-devsecops-practices/200/download

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 18

Let’s look at these practices and techniques as they fit into DevSecOps, in order of ranking:

Secure Coding Training (82%) Training developers/engineers in defensive coding and

secure programming concepts, risks, and techniques is key to
shifting responsibilities for security early into the design and
coding life cycle. Developers also need security training to be
effective participants in threat modeling, perform code reviews,
understand and accept SAST tools, and so on.

Risk Assessments (82%) Lightweight risk assessments identify applications/services that

have compliance or security or operational risks early in design/
concept stages. Development teams work with security engineers
to identify sensitive information, map threat scenarios, and
identify data and services that need to be protected. See Mozilla’s
Rapid Risk Assessment as an example.10

Static Analysis (SAST) (79%) Organizations can perform automated code scanning at multiple
points in engineering workflows to catch common coding
mistakes and vulnerabilities:

• High-fidelity, immediate checks in IDE using integrated plug-ins

• Custom rules in precommit hooks (high-level open source

frameworks like semgrep and CodeQL make it easier to write
your own checks)

• High-fidelity, fast, incremental checks in CI (time-bound)

• Checks for embedded secrets in code and repositories, later

in build (e.g., git-secrets, OWASP Sedated, truffleHog)

• Deeper scans of the entire code repo on a regular frequency

(e.g., nightly), outside of the build pipeline.

Different engines catch different problems, so you need multiple

scanners to get high levels of confidence.

SAST should prove an easy sell to DevOps teams because it

works at a level that they care about and understand. SAST also
has a training effect. Over time, developers will change their
coding practices in response to tool findings. But to be accepted,
the tools must be fast, accurate (focused on problems that
developers and engineers will actually fix), and integrated early
into coding workflows.11

Periodic Vulnerability Scanning Regular, periodic (e.g., monthly) automated vulnerability scans are
(72%) a fundamental part of vulnerability management programs. Like

10
https://infosec.mozilla.org/guidelines/risk/rapid_risk_assessment.html
11
F or a good overview of success factors in implementing SAST, see “How Google and Facebook do code analysis,”
https://techbeacon.com/devops/how-google-facebook-do-code-analysis

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 19

 other point-in-time assessments, periodic assessments leave a

SCA Dependency Analysis window of exposure open in rapidly changing environments.

(71%) Software dependency or SCA tools automatically identify

licensing problems and known vulnerabilities in code
dependencies (libraries and frameworks). Organizations can
perform this scanning as a gating process on pull requests to
identify added dependencies, and later during the build process
to check for newly reported vulnerabilities.

Third-Party Penetration The effectiveness of periodic manual testing by outside experts

Testing (71%) depends heavily on testers’ expertise, their familiarity with the
domain and environment, and the time they have available to
conduct the tests. Testing is generally done to meet compliance
requirements.

Point-in-time assessments like this offer limited value in

continuously changing, Agile environments, but they are one
of the most effective ways to find real security vulnerabilities.
Good pen testers find problems that scanners and other tests
cannot. Although organizations cannot use penetration tests to
verify that a system is secure, they can and should use them as a
health check on the state of the secure software development life
cycle (SDLC). Teams should swarm on the results, conducting a
postmortem review to understand what they missed and improve
their training and testing.

NDR/NTA (71%) Network detection and response (NDR) and network traffic
analysis (NTA) capabilities are important for understanding,
managing, and securing east-west network communications in
hybrid cloud architectures and container-based microservices.
They provide deep visibility into network traffic and use machine
learning to identify real-time threats and attacks. Public
cloud provider traffic-mirroring services have made it easy to
deploy these technologies, as part of a shift right approach to
operational security.

Manual Code Reviews (70%) Lightweight peer code reviews, done through pull requests, have
become a common practice in modern software development,
but their effectiveness in finding vulnerabilities depends on the
reviewers’ skills and knowledge, the time they have available,
and their priorities. Research at organizations like Microsoft12 and
Google shows that reviewers spend more time on readability

12
www.cabird.com/pubs/bacchelli2013eoc.pdf

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 20

 and maintainability problems than looking for real defects. While
readability is important (difficult-to-read code likely contains more
defects and proves more risky to change), reviewers need to be
trained to look for more fundamental problems—including training
in secure coding and defensive coding. Manual code reviews are
more effective when combined with SAST: Automated scans can
find subtle problems that reviewers may not be aware of.

Threat Modeling (70%) Threat modeling or design reviews focused on identifying

security threats and risks are difficult to implement in iterative,
incremental development, where the design is under continuous
refinement.

Continuous Vulnerability Continuous scanning for changing threats and vulnerabilities

Scanning (70%) allows organizations to rapidly identify new risks and priorities
and to track the status and success of patching programs. It
involves frequent scanning and incremental analysis of results to
surface changes in security posture and risks.

WAF (70%) A WAF is a plug-in or appliance that filters out specific signatures
in HTTP traffic. The major cloud platforms offer native WAF
services that provide basic protection against common HTTP
attacks and DDoS.

Internal Penetration Internal testers are generally more familiar with the domain and
Testing (67%) environment, but most organizations lack the technical expertise
required for effective penetration testing.

Container/Image Scanning Organizations can and should do static scanning of containers at

(66%) different points for different threats:13

• Scanning container images or code templates (e.g.,

Dockerfiles) for common configuration mistakes and checks
against best practices for setup and use of containers (e.g,
DockerBench)

• L ooking inside container image layers to identify dependencies

and to identify known vulnerabilities in this software

• Scanning containers in registries for vulnerable images (now

provided by most registry services)

Third-Party Compliance Audits and assessments of control programs such as SOC 2, ISO
Reviews (65%) 27001, and PCI are required for regulated industries. In DevOps
environments, these assessments present challenges around
continuous incremental change, CD, and “you build it, you run it,”
which can violate requirements for separation of responsibilities.

13
Container scanning at these different points could be included in CWPPs.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 21

NGAF/NGWAF (65%) Next-generation application firewalls provide higher-fidelity,
cloud-based runtime protection for web applications/APIs, which
consolidates threat information with application context. The boundary
between NGWAF and RASP can be blurry.

HIDS (65%) File integrity monitoring and host-based intrusion detection systems
such as Tripwire and OSSEC track changes to files and configuration
data. In rapidly changing Agile environments, this results in lots of
noise, making it difficult to separate out authorized changes from
suspicious activity.

Security Stories (63%) Developers can be asked to identify personas for “bad users” (e.g.,
fraudsters, hackers, insider threats) and write requirements (user
stories) from a negative perspective, to identify threats to the system
or service that need to be protected against. “As a hacker, I want to….”
A variation of standard Agile practices, this encourages development
teams to think outside of functional user stories and to address
security requirements and risks in design and development.

DAST (61%) Organizations can perform dynamic scanning of web applications/APIs

later in the CI/CD pipeline, after the code is built and the system is
provisioned for functional testing, integration testing, and acceptance
testing. They can proxy these tests through a scanner to automatically
execute passive and active vulnerability scans.

CSPM (56%) CSPM tools/services automatically scan cloud instances to detect

and catch common configuration mistakes and known threats and
to enforce security and compliance policies. Open source examples
include the following:

• Cloud Custodian:
https://github.com/cloud-custodian/cloud-custodian

• Prowler: https://github.com/toniblyx/prowler

• Cloudsploit: https://github.com/aquasecurity/cloudsploit

In any cloud environment, we have too many configuration options

and too many compliance and security policies to review manually.

Virtual Patching (56%) Virtual patching refers to applying protection in WAFs, RASP, and CWPP
at runtime to block known attack signatures as a rapid response
to newly discovered vulnerabilities. Organizations can use virtual
patching as a temporary stopgap until they can apply and roll out a
patch to the software.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 22

Fuzzing (54%) Fuzzing refers to automated negative testing for APIs, file-based
applications, and protocols. Security researchers use fuzzing because it
can find real vulnerabilities with few false positives. It proves especially
useful and important for applications written in memory-unsafe
languages like C/C++ (although fuzzing is also available for languages
such as Go, Rust, Java, and Python). Fuzzing can be difficult to add into
CI/CD, because good fuzzing takes time to set up, run, and interpret
the test results. However, modern cloud-based fuzzing services make
this easier, using limited test cases in the build pipeline (for example,
GitLab now offers fuzzing options) or continuously outside of CI/CD.

IAST (54%) IAST instruments the application runtime and detects application
vulnerabilities as the system or service runs. This passive testing
technique depends on other tests (e.g., functional tests) to exercise
the code.

RASP (54%) RASP instruments the application runtime (e.g., JVM or .NET CLR)
to provide operational visibility into attacks and to defend against
common application security attacks and language/framework-specific
vulnerabilities. RASP adds some runtime overhead and operational
complexity in return for runtime protection.

CWPP (52%) CWPPs provide runtime protection for containers and cloud instances.
This broad technology category covers services that offer different
levels and types of runtime shielding, including network segmentation,
system integrity protection, application control/whitelisting, behavioral
monitoring, host-based intrusion prevention, and optional anti-
malware protection. Open source examples include the following:

• Falco for runtime security: https://github.com/falcosecurity/falco

• Open source sysdig for troubleshooting and incident response:

https://github.com/draios/sysdig

Bug Bounties (51%) In these wider-scale programs, organizations invited outside white
hat testers (security researchers, ethical hackers) to continuously test
systems and report vulnerabilities, on a reward basis. Bug bounty
programs at organizations like Google, Apple, Microsoft, and Facebook
attract a lot of attention, but they also require a lot of work to set up
and manage.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 23

Keys to DevSecOps Success
Finally, we want to look at the key drivers, the enablers and barriers to success, of
DevSecOps programs and how to measure a program’s success and risks.

KPIs
What are the major KPIs you use to measure the success of your DevSecOps
Measurement and feedback loops, using data to activities? Select all that apply.
drive decisions, represent a fundamental part of
Number of open security
52.3%
DevSecOps. See Figure 16. vulnerabilities
Time-to-fix security vulnerabilities 37.4%
DevSecOps programs have two broad categories
False positive rates of
reported vulnerabilities 35.8%
of metrics:
Build time delay imposed by
security testing/reviews 35.0%
• V
 ulnerability management—The number Number of builds failed due to
security vulnerabilities found 33.3%
of open vulnerabilities, how long they stay
Time-to-detect security vulnerabilities 31.3%
open, where vulnerabilities are found (early
in development and build, or later), false Automated test coverage 28.4%
Number of security vulnerabilities
positive rates, how long it takes to detect discovered after deployment 28.0%
Change lead time (cycle time to deploy
vulnerabilities, and how much it costs to fix code changes/fixes to production) 20.6%

them. Organizations rely on these metrics to Cost to remediate audit findings 8.6%
understand security risks and trends. Other 6.6%

• D
 evOps delivery—How often builds are 0% 10% 20% 30% 40% 50%

delayed or fail due to security vulnerabilities, Figure 16. KPIs in Use to Measure
DevSecOps
automated test coverage, and change lead time for delivery. These metrics can
help the organization to understand the costs of their security program and how to
address security risks in DevOps.

Organizations still have a long way to go in implementing comprehensive metrics

programs. Only half (52%) use basic vulnerability measurements to understand and
manage risks. Roughly a third of respondents collect and use data on delivery (false
positives, delays, and failures).

TAKEAWAY
Optimizing DevOps efficiency and velocity not only reduces IT costs and improves an organization’s agility, it
also helps improve the organization’s security posture:
3 C
 hange lead time/cycle time—The speed at which teams deliver changes is the heartbeat of DevOps. It also
determines the organization’s ability to push out security patches.
3 B
 uild time delays caused by security testing and number of builds failed due to vulnerabilities found—
Organizations can use these to improve the effectiveness of security controls added to delivery (to find
vulnerabilities early) without degrading the team’s velocity of delivery.
3 F alse positive rates of reported vulnerabilities—This important measure of efficiency improves the fidelity
of scanning and reporting, minimizing noise, which allows security and DevOps teams to focus on real risks.
3 A
 utomated test coverage—This involves identifying how much automated tests in the build pipeline protect
the code base (or not). As discussed earlier, this metric helps you understand where your blind spots are in
testing and compliance. Test coverage also determines the organization’s confidence in its ability to roll out
patches successfully and how challenging it will be to add security testing into build and delivery pipelines.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 24

 The success of DevSecOps programs involves both
What are your top five challenges (e.g., barriers) in implementing
DevSecOps at your organization? technical and organizational factors. See Figure 17.

Organizational silos between While tools could be faster and more accurate and
development, operations, security 50.0%
Insufficient budget/funding for
easier to use, technology (that is, the capabilities
security programs and tools 48.8%
and speed of tools and the capabilities of cloud
Shortage of application
37.8%
security personnel/skills service providers) does not hold organizations
Continuously changing
36.6%
requirements and priorities back. As shown in Figure 17, the major challenges
Lack of developer/engineer buy-in 34.3% to implementing DevSecOps successfully continue
Lack of transparency into
development/operations 28.7% to be organizational: siloed specialties, insufficient
Shortage of cloud engineering funding for security programs, shortage of skills,
personnel/skills 28.3%
Shortage of cloud security
26.4% continuously changing priorities and requirements,
personnel/skills
Lack of management buy-in 25.2%
and lack of buy-in from the different stakeholders.
Inadequate/ineffective security
training for developers 24.4%

Lack of buy-in from security team 20.5%

Lack of coding skills in security teams 19.7%

Technical debt and security debt
in legacy system environments 18.1%
Compliance risks or lack
16.5%
of compliance buy-in What do you consider the top five factors that have contributed to your
Lack of security tool support for security program’s success?
languages, frameworks, platforms 15.0%
Security testing/scanning tools
are inaccurate/unreliable 14.2% Training developers/engineers
in secure coding 51.8%
Security testing/scanning
tools are too noisy 13.4% Improving communications
Inadequate test automation/ across Dev, Ops, and security into
13.0% developer/engineering tool chains 51.4%
over-reliance on manual testing
and build/deploy workflows
Supply chain risks in third-party/open
source components, APIs, containers 12.6%
Securing management buy-in 47.8%
Security testing/scanning
tools are too slow
7.1%
Securing developer/engineering buy-in 46.2%
Security capabilities of cloud
platforms are inadequate 6.3%
Integrating automated security testing
into developer/engineering tool 45.0%
Other 2.4% chains and build/deploy workflows
0% 10% 20% 30% Sharing
40%goals and
50%measurable success
factors across Dev, Ops, and security
Figure 17. Top Challenges in Implementing DevSecOps into developer/engineering tool 43.4%
chains and build/deploy workflows
Automating build/test/deploy/
Management commitment and organizational provisioning workflow, and
thereby minimizing time/ 43.4%
changes also serve as key drivers to success (see cost to fix vulnerabilities
Figure 18). Improving communications across Developing “security champions”
in Dev/Ops teams 42.2%
organizational silos, securing management buy-in Creating cross-functional
35.1%
DevSecOps teams
and buy-in from developers and engineers, and Reusing “secure by default” frameworks,
31.5%
libraries, templates, services
sharing goals and KPIs across teams, are all key Clear definition of success and
ability to measure (metrics, CSFs) 26.7%
nontechnical success factors.
Enforcing security/compliance
policies in code using programmable/ 20.7%
Success starts with training developers and immutable infrastructure
engineers in security concepts and secure Following a common compliance
framework (Please specify the 10.0%
coding. This needs buy-in from management and framework(s) you use)

developers, and improved communications across Other 4.8%

specialties, before you can get to the heavy-lifting 0% 10% 20% 30% 40% 50%

work: integrating automated testing into build Figure 18. DevSecOps Success Factors
chains, automating builds, and enforcing security
and compliance policies in code.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 25

Conclusions/Moving Forward
Security as Code consists of three parts:

• S
 ecurity in coding—Application security practices and techniques to secure the
SDLC, training developers on secure coding practices, building secure by default
frameworks and libraries, and integrating security checks inside the workflow and
build pipelines

• C
 oding in security—Understanding the SDLC workflows and tools and how to use
them for security and compliance purposes, moving from guidelines and checklists
to guardrails and automated tests; as security teams mature, leveraging version
control and CI/CD to deploy detection, alerting, and response solutions

• S
 ecuring the code—Treating the code repos and build and test toolchains and
infrastructure as part of the organization’s attack surface; securing the code and
toolchains and infrastructure from end to end; getting visibility and control over
what code is changing and who can change it

Code becomes the common language and

Fundamentals of Security as Code
currency between dev, ops, security, and
compliance. It brings together these different Moving security into code involves a significant learning curve. Security
engineers need to understand a long and constantly changing list of things,
specialties, enabling collaboration and
including multiple cloud service platform services and APIs, containers and
improving transparency. VMs, serverless architectures, templating languages, and scanning tools.
With Security as Code, training requirements However, to work with developers and engineers, and to move security into
change. We have focused on addressing code, you need to start with understanding the mechanics of modern software
development, including version control concepts, branching and merging, Git
developer skill gaps around security. Now we
workflows, automated build pipelines, and CI and CD. This process offers a key
need to address coding skill gaps for security opportunity to work with developers—to learn and to build bridges.
specialists.

An ongoing transformation of software development and operations engineering focuses

on speed, iteration, and confidence. The same transformation needs to occur for security:
achieving speed and agility through coding and automation; becoming more iterative, risk
based, and pragmatic; and continuously learning and adapting.

Putting security into code allows organizations to accelerate and scale. Organizations
can reuse policies and controls across services and teams (and even outside of the
organization). The communities built up around open source toolsets like semgrep,
CodeQL, and Open Policy Agent attest to their success.

To achieve this, security teams need to collaborate with developers and operations
engineers and share their tools and practices. Security engineers need to learn to think
and work like developers and apply the same Agile, incremental methods to continuously
improve the security program. Organizations must recognize and reconcile this
fundamental tension between agility and control.

With Security as Code, organizations face enormous challenges but will also benefit from
the associated potential to accelerate, learn, and scale.

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 26

About the Authoring Team
Jim Bird, SANS analyst and co-author of SEC540: Cloud Security & DevOps Automation,
is an active contributor to the Open Web Application Security Project (OWASP) and an
author of books on agile security and secure DevOps. He has worked at major technology
organizations and financial institutions around the world in management, software
development, IT and business operations, and security.

Eric Johnson is Principal Security Engineer at Puma Security and Senior SANS Instructor
focusing on cloud security, DevSecOps automation, and building static analysis tools.
His experience includes cloud infrastructure and security automation, cloud security
assessment, static source code analysis, web and mobile application penetration testing,
security development lifecycle consulting, and secure code review assessments.

Frank Kim leads the management and software security curricula for SANS, developing
courses on strategic planning, leadership, and application security. He is also a SANS
certified instructor, helping to shape, develop, and support the next generation
of security leaders. Previously, Frank served as CISO at the SANS Institute, leading
its information risk function, and as executive director of cybersecurity at Kaiser
Permanente, where he built an innovative security program to serve one of the nation’s
largest not-for-profit health plans and integrated healthcare provider. Currently, as
founder of ThinkSec, a security consulting and CISO advisory firm, Frank helps leaders
develop business-driven security programs.

Sponsor

SANS would like to thank this paper’s sponsor:

A SANS Survey: Rethinking the Sec in DevSecOps: Security as Code 27