Unclassified

Pre-Decisional/DRAFT
UNCLASSIFIED
Pre-Decisional/DRAFT
CLEARED
For Open Publication

Oct 19, 2021

Department of Defense
OFFICE OF PREPUBLICATION AND SECURITY REVIEW

DoD Enterprise
DevSecOps Reference
Design:
Multi-Cluster CNC F Kubernetes

September 2021

Version 0.9

This document automatically expires 1-year from publication date unless revised.

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

UNCLASSIFIED
Unclassified 1
Pre-Decisional/DRAFT
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Document Set Reference

UNCLASSIFIED i
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Document Approvals

Approved by:

________________________________________
Paul B. Puckett III
Director, Enterprise Cloud Management Agency
HQDA CIO

________________________________________
Jason R. Weiss
DoD CIO – Information Enterprise Directorate

UNCLASSIFIED ii
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Trademark Information
Names, products, and services referenced within this document may be the trade names,
trademarks, or service marks of their respective owners. References to commercial vendors and
their products or services are provided strictly as a convenience to our readers, and do not
constitute or imply endorsement by the Department of any non-Federal entity, event, product,
service, or enterprise.

UNCLASSIFIED iii
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Contents
1 Introduction............................................................................................................... 1
1.1 Background ........................................................................................................ 1
1.2 Purpose.............................................................................................................. 1
1.3 DevSecOps Compatibility .................................................................................. 2
1.4 Scope................................................................................................................. 3
1.5 Document Overview........................................................................................... 3
2 Assumptions and Principles...................................................................................... 4
3 Software Factory Interconnects ................................................................................ 5
3.1 Immutable Infrastructure .................................................................................... 7
3.2 Continuous Reconciliation of Components ........................................................ 9
3.3 Multi-Cluster Kubernetes.................................................................................. 11
3.4 Secure Supply Chain Provenance ................................................................... 13
3.5 Production Ready Kubernetes, even in Development...................................... 15
4 Additional Tools and Activities ................................................................................ 17

UNCLASSIFIED iv
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Figures
Figure 1: Multi-Cluster Kubernetes Reference Design Interconnects.............................. 7
Figure 2: Cluster Creation and Enforcement via Continuous Reconciliation ................. 10
Figure 3: Multi-Cluster Kubernetes relies on a Global Control Plane ............................ 12

Tables
Table 1: Tools Spanning all Phases.............................................................................. 18
Table 2: Security Activities Summary and Cross-Reference ......................................... 19
Table 3: Build Phase Tools ........................................................................................... 21
Table 4: Build Phase Activities ...................................................................................... 23
Table 5: Test Phase Tools ............................................................................................ 24
Table 6: Test Phase Activities ....................................................................................... 25
Table 7: Release and Delivery Phase Tools ................................................................. 26
Table 8: Release and Delivery Phase Activities ............................................................ 26
Table 9: Deploy Phase Tools ........................................................................................ 27
Table 10: Deploy Phase Activities ................................................................................. 29

UNCLASSIFIED v
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

1 Introduction
1.1 Background
The Department of Defense (DoD) is at a defining moment as the services pivot to deliver
cloud-based services, adopt cloud-native software development practices, and DevSecOps
methodologies that serve to modernize the DoD in preparation for the digital battlefield and
competition with near-peer threats. The CNCF Multi-Cluster Kubernetes Reference Design
provides the DoD with a model and guidance on building and deploying cloud-native software
on Kubernetes to deliver measurable software-based outcomes to the mission with resilience
and at the speed of relevance.

1.2 Purpose
The CNCF Multi-Cluster Kubernetes Reference Design is part of the DoD Enterprise
DevSecOps Reference Design family and adheres to the structure of other DoD Reference
Design documents. This reference design is intended to serve as architectural and design
guidance to DoD organizations that intend to build and deploy cloud-native software on
Kubernetes. This reference design describes the verifiable attributes and lists the preferred and
required set of tools and activities across all phases of the DevSecOps lifecycle.

For brevity, the use of the term ‘Kubernetes’ or ‘K8s’ throughout the remainder of this
document must be interpreted as a Kubernetes implementation that properly submitted
software conformance testing results to the CNCF for review and corresponding
certification. The CNCF lists over 90 Certified Kubernetes offerings that meet software
conformation expectations. 1

The target audiences for this document include:

• DoD Enterprise DevSecOps capability providers who build DoD Enterprise DevSecOps
hardened containers and provide a DevSecOps hardened container access service.
• DoD Enterprise DevSecOps capability providers who build DoD Enterprise DevSecOps
platforms and platform baselines and provide a DevSecOps platform service.
• DoD organization DevSecOps teams who manage (instantiate and maintain)
DevSecOps software factories and associated pipelines for its programs.
• DoD program application teams who use DevSecOps software factories to develop,
secure, and operate mission applications.
• Authorizing Officials (AOs).

1Cloud Native Computing Foundation, “Software conformance (Certified Kubernetes,” [ONLINE] Available:
https://www.cncf.io/certification/software-conformance/. [Accessed 8 February 2021].

UNCLASSIFIED 1
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

This reference design aligns with these reference documents:

• DoD Digital Modernization Strategy. 2
• DoD Cloud Computing Strategy. 3
• DISA Cloud Computing Security Requirements Guide. 4
• DISA Secure Cloud Computing Architecture (SCCA). 5
• Presidential Executive Order on Strengthening the Cybersecurity of Federal Networks
and Critical Infrastructure (Executive Order (EO) 1380). 6
• National Institute of Standards and Technology (NIST) Cybersecurity Framework. 7
• NIST Application Container Security Guide. 8
• Kubernetes STIG. 9
• DISA Container Hardening Process Guide. 10

1.3 DevSecOps Compatibility

This reference design asserts version compatibility with these supporting DevSecOps
documents:
• DoD Enterprise DevSecOps Strategy Guide, Version 2.1.
• DevSecOps Tools and Activities Guidebook, Version 2.1.
• U.S. Army DevSecOps Playbook, Version 1.1. 11

2 DoD CIO, DoD Digital Modernization Strategy, Pentagon: Department of Defense, 2019.
3 Department of Defense, "DoD Cloud Computing Strategy," December 2018.
4Defense Information Systems Agency, “Department of Defense Cloud Computing Security Requirements Guide,
Version 1, Release 3,” March 6, 2017
5Defense Information Systems Agency, "DoD Secure Cloud Computing Architecture (SCCA) Functional
Requirements," January 31, 2017.
6 White House, "Presidential Executive Order on Strengthening the Cybersecurity of Federal Networks and Critical

Infrastructure (EO 1380)," May 11, 2017.

7National Institute of Standards and Technology, Framework for Improving Critical Infrastructure Cybersecurity,
2018.
8National Institute of Standards and Technology, "NIST Special Publication 800-190, Application Container Security
Guide," September 2017.
9 Defense Information Systems Agency, “Kubernetes STIG, Version 1, Release 2,” July 26, 2021.
10Defense Information Systems Agency, “Container Hardening Process Guide, Version 1, Release 1,” October 15,
2020
11 Army Enterprise Cloud Management Agency, “Army DevSecOps Playbook, v1.1,” [Online] Available at:
https://www.milsuite.mil/book/docs/DOC-1050076.

UNCLASSIFIED 2
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

1.4 Scope
The CNCF Multi-Cluster Kubernetes Reference Design is a blue-print for designing a secure
application platform for DoD organizations. This platform will allow these organizations to
interconnect and share infrastructure services while maintaining the independence and flexibility
to enact on their unique missions. While this design provides options to centralize management
services, such management plane points can be forward deployed based on the needs of the
truly unique and organic composition of DoD networks, which includes disconnected, hard to
access, and highly secured networks. By allowing flexibility for Authorizing Officials (AO) and
mission owners to own, share, or borrow components of this design, the design naturally
enables progressive adoption for workloads that might be in a state of migration, both in their
hosting locations and technical architectures.
All components of this design can be elastically provisioned and dynamically scaled in their
installation while also taking advantage of building upon existing installations, either in service
capabilities or workload hosting. The design accommodates for workload mobility, cloud
migration, on-premises migration, and continued portability across various hosting locations
(with special consideration including edge and tactical networks).
Each recommendation in this design relies heavily on the concept of cloud native computing,
which “empowers organizations to build and run scalable applications in modern, dynamic
environments such as public, private, and hybrid clouds.” 12 The platform utilizes containers, a
software packaging abstraction that focuses on application components and their
dependencies. To run distributed applications at scale using containers, Kubernetes is used as
a foundational substrate across the environments. In addition to providing a consistent
abstraction for computing resource allocation, Kubernetes has become the industry standard for
orchestration and operation of modern distributed applications. Alternative form factors to
containers can also be hosted if required.
This document does not address strategy, policy, or acquisition.

1.5 Document Overview

The documentation is organized as follows:
• Section 1 describes the background, purpose and scope of this document.
• Section 2 identifies the assumptions relating to this design.
• Section 3 describes the DevSecOps software factory interconnects unique to a multi-
cluster Kubernetes architecture.
• Section 4 describes the software factory design.
• Section 5 captures the additional required and preferred tools and activities, building
upon the DevSecOps Tools and Activities Guidebook as a baseline.

12 CNCF Technical Oversight Committee, “CNCF Cloud Native Definition, v1.0,” [Online] Available at:
https://github.com/cncf/toc/blob/main/DEFINITION.md.

UNCLASSIFIED 3
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

2 Assumptions and Principles

This reference design makes the following assumptions:
• Network boundaries are within an existing accreditation boundary with a private IP
space.
• Communication through network boundaries, especially on DoD networks, might require
additional cybersecurity, coordination and configuration.
• Existing accreditation boundaries should have an accredited Virtual Data Center
Managed Services (VDMS), this is also referred to as delegated authentication and this
VDMS providing industry standard authentication is assumed. Alternative technologies
that leverage industry standard authentication can also be used if they meet Defense
Information Systems Agency’s (DISA) Secure Cloud Computing Architecture (SCCA)
requirements. 13
• Adoption of the ecosystem design can in some instances be achieved through different
alternative choices of software tools. The reference design makes an effort to
recommend the most mature and suitable tools for the core design principles as of the
time of this writing, along with tradeoffs and decision making criteria when evaluating a
particular technology selection over the other.
• This reference design will be updated as tools and design principles evolve in industry.
• The Software Factory Interconnects and Detailed Descriptions of Core Principles are not
fully prescriptive about tool selection and are presented as the means to adopt the
model. The later section on Creating an Ecosystem will illustrate how the Army set about
building out an implementation that is compliant with this reference design, and offer the
reader recommendations of some open source tooling that is not required for adoption of
the design.
• Common shared services within an existing network boundary can be leveraged to
reduce the overall burden of accreditation and maintenance. This is encouraged for
Foundational Services and Enterprise Services (see the Army Cloud Plan, Appendix A
for examples of some of these services). 14
• The selected Kubernetes implementation must pass the Kubernetes conformance test
suite and support lifecycle management through the Kubernetes SIG Cluster Lifecycle
Cluster API project in order to fully benefit from certain control mappings.

13 Defense Information Systems Agency, “Secure Cloud Computing Architecture,” [Online] Available at:
https://storefront.disa.mil/kinetic/disa/service-catalog#/forms/secure-cloud-computing-architecture.
14 US Army Chief Information Officer, “U.S. Army Cloud Plan,” [Online] Available at:
https://www.army.mil/standto/archive/2020/10/09/.

UNCLASSIFIED 4
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

• Product lock-in into the Kubernetes API and its overall ecosystem is openly recognized.

It is critically important to avoid the proprietary APIs that are sometimes added
by vendors on top of the existing CNCF Kubernetes APIs. These APIs are not
portable and may create vendor lock-in at the application level, not just at the
software factory level!

3 Software Factory Interconnects

The DoD Enterprise DevSecOps Fundamentals describes a DevSecOps platform as: “a
multi-tenant environment consisting of three distinct layers: Infrastructure,
Platform/Software Factory, and Application(s). Each reference design is expected to
identify its unique set of tools and activities that exist at the boundaries between the
discrete layers, known as Reference Design Interconnects. Well-defined interconnects
in a reference design enable tailoring of the software factory design, while ensuring that
core capabilities of the software factory remain intact.”

The core principles of the platform for the CNCF Multi-Cluster Kubernetes Reference
Design are designed to be further enhanced with future reference designs for higher
levels of abstraction on top of Kubernetes, and therefore this design focuses on the
foundational layers rather than those closer to the application level.

A DevSecOps Ecosystem on Kubernetes must demonstrate that the following verifiable

attributes have been programmatically implemented:

● Immutable Infrastructure: Create infrastructure that is scaled up, destroyed,

and recreated rather than persisted.
● Continuous Reconciliation: Microservices and software components are
installed to enable rapid patching, prevent configuration drift, and alignment with
approved baselines. The configuration is policy-driven to enable guardrails and
self-service layering.
● Multi-Cluster Kubernetes: The principle of multiple relatively small clusters
compared to a single super cluster further reduces blast radiuses, allows better
isolation, and enables better distribution of resources. Distributed resources allow
better collaboration across organizations and disparate environments. Multiple
clusters thrive on composable toolchains; when setup and configuration of the
environment is done declaratively through intent driven APIs, additional
management burden is not created. Clusters effectively communicate workloads
running on top to coordinate more complex deployment requirements.
○ Smaller production locations and minimalist edge environments: The
ability to distribute workloads means production environments can be

UNCLASSIFIED 5
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

disconnected, in a different location from development, and still have high

confidence on consistent environments.
○ Secure inter-service communications: Communications between and
within clusters are authenticated and encrypted.
● Provenance in Secure Supply Chain: Third party software is verified and
documented; containers and software originating from upstream locations meet
DoD requirements that include vendor-support where applicable, continually
maintained from upstream, updated repeatedly, and strive to have binding
agreements that software is maintained. Stewardship of this supply chain is
assured to meet DoD ongoing maintenance needs.
○ Scanned and verified images: No in-place patching. All artifacts to be
deployed with special consideration to container images are to be
modularly built with layers that can be individually updated, scanned, and
verified.
Production Ready Kubernetes, even in Development: Complete operating
environments: clusters that possess the necessary additional components deployed in a
thoughtful way to achieve compliance needs, even when deployed into development or
lab environments, to create a consistent environment with the necessary oversight for
authorizing officials and security teams. Minimize operational overheads and complexity
while going above and beyond.
Figure 1: Multi-Cluster Kubernetes Reference Design Interconnects visually depicts these
specific interconnects.

UNCLASSIFIED 6
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Figure 1: Multi-Cluster Kubernetes Reference Design Interconnects

Each of these interconnects will be described fully next.

3.1 Immutable Infrastructure

In a modern platform there are numerous layers of infrastructure and abstraction that
teams can operate at - from all the way down to the bare metal to virtual constructs like
Kubernetes services. Shrinking this sphere of concern is of utmost importance for
operations teams to be able to effectively maintain and harden their environments.
Immutable infrastructure creates the basis on which consistent underlying environments
can be provisioned for the platform to operate on and deliver mission value.
Kubernetes runs on top of worker machines, called nodes, that run traditional operating
systems to provide the underlying capabilities containerized applications need. Since
containers are not operating systems, an environment still has to take care of proper
hardening, compliance, and maintenance on these underlying nodes. Cluster API is a
Kubernetes sub-project focused on providing declarative APIs and tooling to simplify
provisioning, upgrading, and operating multiple Kubernetes clusters. Cluster API
facilitates provisioning the underlying nodes in a declarative, immutable manner that
greatly facilitates immutable approaches to infrastructure management.

Mutating operating systems in place creates risk that changes will impact running
workloads, allow configuration drift, or permit advanced persistent threats. Leveraging
Kubernetes ability to cordon and remove underlying nodes, along with proper

UNCLASSIFIED 7
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

accountability for disruption budgets across tenants, enables delegating of upgrading

underlying systems more readily to the environment’s control. As a result, operating
systems are more easily patched while providing necessary assurances that any drift in
configuration is periodically reconciled.

New Virtual Machines are instantiated from a known good state template that has been
cleaned, including the latest updated packages, General Purpose Operating System
Security Requirements Guide (SRG, and any relevant Security Technical
Implementation Guides (STIG) baselines, and brought in to replace the old virtual
machines. 15 This operation occurs prior to removing the old virtual machines, creating a
scale up operation prior to scaling down, so running workloads are given the necessary
time to schedule over to a new machine. By managing this from a central source, even
when multiple Kubernetes clusters are deployed, every Virtual Machine across the
entire network can be properly updated on a prearranged scheduled basis to meet the
requirements of a given Authorizing Official’s posture.

Kubernetes running on top of these nodes follows a similar pattern with the higher level
abstractions, like Deployments, StatefulSets, Jobs, and DaemonSets. These constructs
leverage immutable container images instantiated with similar security controls applied
to prevent mutability of running workloads. By mounting configuration into these
ephemeral environments as read-only there is significantly better control over what is
visible from the requester and the resources are given access to the information on a
per-session basis. Nodes that do not require access to secrets or other environment
variables relevant to the workloads the node is running are not able to access that
information.

As virtual machines are rotated in this fashion, the underlying authentication credentials
that allow nodes to communicate with the Kubernetes control plane are similarly
refreshed. Newly scheduled pods on these nodes authenticate through these refreshed
credentials. The node's credentials provide access to the underlying storage for the
configuration necessary for all workloads.

This process also greatly enhances the security posture of Configuration Management
(CM) and Contingency Planning (CP) in the NIST SP 800-53 family. Combining this
capability with continuous reconciliation of components gives the organization fully
patched systems on regular intervals. A lot of the concepts mentioned in this section are

15Defense Information Systems Agency, “General Purpose Operating System Security Requirements
Guide Technology Overview, v2.1” [Online] Available at: https://dl.dod.cyber.mil/wp-
content/uploads/stigs/zip/U_General_Purpose_Operating_System_V2R1_SRG.zip.

UNCLASSIFIED 8
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

also requirements for proper implementations of Zero-Trust Architectures based on

NIST SP 800-207. 16, 17

3.2 Continuous Reconciliation of Components

Modern distributed systems, especially those leveraging microservices, typically have a

larger deployment topology than a single binary. For example, a Source Code
Repository may include databases, caching mechanisms, proxy servers, etc. Installation
and configuration of these components is a complicated and error prone process. A
successful deployment (and maintenance) process needs to address the following
concerns:

● There must be an obvious, single, trusted process to run the

deployment/updates, complete with appropriate access controls. This can be
called the Control Plane.

● There must be a means to validate the processes and artifacts used in the
deployment.

● There must be a method to allow granular configuration on where the

components are deployed.

● There must be a method to allow per-deployment customization of the installed

software beyond vendor-provided templates.

Recent trends in industry guide organizations towards concepts such as Infrastructure

as Code (IaC) or Configuration as Code. These trends, while useful, fail to consider the
end user. For example, a playbook, written in some automation framework, that stands
up a given environment is of little use to an end user who doesn’t know where or how to
run that playbook (i.e. there is no obvious answer to the question “How do I run this?”).
It is also difficult to ascertain the level, and duration of, permissions required to run the
playbook.

A better approach is one in which we are continuously enforcing the state of the system,
a process known as Continuous Reconciliation. Figure 2 visualizes cluster creation and
enforcement using continuous reconciliation. If we place the Kubernetes component
configurations in a central repository (considered as a “source of truth”), we can

16 National Institute of Standards and Technology, “Security and Privacy Controls for Information Systems and
Organizations (SP 800-53, Revision 5),” [Online] Available at: https://csrc.nist.gov/publications/detail/sp/800-53/rev-
5/final.
17 National Institute of Standards and Technology, “Zero Trust Architecture (SP 800-207),” [Online] Available at:
https://csrc.nist.gov/publications/detail/sp/800-207/final.

UNCLASSIFIED 9
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

configure each cluster to refer to this central configuration repository. Each cluster,
independently, will work to ensure that its own configuration is the same as that defined
in the central repository. When clusters are created, the standard baseline deployment
and configuration is pulled from the central repository. After creation, the same central
repository is continuously referenced and the central configurations are enforced on the
cluster. This arrangement means troubleshooting and customization is easier to
achieve, while still maintaining compliance with a central set of instructions. Such an
approach also provides a more reliable mechanism to facilitate upgrades and ownership
of those components' health while preventing unauthorized modification.

Figure 2: Cluster Creation and Enforcement via Continuous Reconciliation

Continuous Reconciliation accounts for the deployment and maintenance steps better
than a repository that simply holds configuration (i.e. a standalone playbook), and forces
an organization to centralize and automate the process for running their configuration
on the environment. By automating this process, it can be repeated and leveraged to
prevent configuration drift.

Configuration of components can be bundled with the Kubernetes manifests necessary

to run those configurations, whether they are custom built, vendor provided, or open
source. Additionally, these bundles can be validated, signed, and stored in a repository.
The repository can be relocated to different environments while maintaining the
signature for validation in a decentralized manner.

Finally, while vendor-provided hooks exist for customization, the author of a given
Kubernetes component cannot predict the needs of every end user. There will be cases
where customization is needed but the author-supplied manifest does not allow for this.
In order to address this concern, the same workflow mentioned above can inject, or
overlay, additional configurations on top of the manifests. This per-site customization

UNCLASSIFIED 10
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

strikes the balance of honoring central management while providing an agile method for
responding to unknown circumstances in the field.

Continuous Reconciliation of Components can occur from multiple focal points in an

environment, as by design it is decentralized in nature. For individual teams or areas of
responsibility within this environment, the centralizing of these components naturally
becomes a regional control plane, or a team centered control plane. If a single team
has responsibility for the majority of the components in a given environment (for small
footprint, or small teams) then that centralized control plane would form a regional
control plane.

3.3 Multi-Cluster Kubernetes

DoD environments are numerous and complicated: some enclaves have limited or zero
connectivity to egress out to the public internet. These restrictions are emphasized for
enclaves on the Department of Defense Information Network (DODIN). Due to the
sensitive nature of the data that resides inside of these environments, there is higher
regulation for any requests to egress or ingress into the network, further complicated by
the confusing Ports, Protocols, and Services Management (PPSM) process. 18
Additional regulations create friction and barriers to entry for new organizations trying to
stand up development environments, operational environments, and create additional
hurdles for timely patching and maintenance on environments.

Multi-Cluster Kubernetes is a design paradigm to provide reduced friction on bringing

the correct tools to the correct environments. Figure 3 depicts the use of a Global
Control Plane alongside a secure supply chain process to continuously reconcile one or
more regional management clusters. By creating a global view across all of the
enclaves, workloads and developer tooling can be instantiated into the areas that make
the most sense, and establish peering and connectivity around the complexity of the
DoD environments. Leveraging industry standard caching and information transfer for
artifacts and fully secured connections when possible, workloads are able to follow a
path to production that might result in workloads running in disconnected or limited
connectivity environments. Developers are able to develop code, visualize testing, and
push workloads through numerous services that might be hosted in cloud environments,
data centers, or tactical networks. Using a common substrate across these
environments enables elasticity and better efficiency, while at the same time enabling
Authorizing Officials (AO) to maintain full control over their enclaves.

18Defense Information Systems Agency, “Enterprise Connections Ports, Protocols, and Services
Management,” [Online] Available at: https://public.cyber.mil/connect/ppsm/.

UNCLASSIFIED 11
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Kubernetes is a distributed system, meaning that elasticity and on-demand capabilities

are possible in a localized fashion (in a single cloud environment or a single
datacenter). Kubernetes provides an Application Programming Interfaces (API) that
enables consumption of these on-demand capabilities. In addition, the declarative and
intent driven nature of the Kubernetes API means that work can be described at rest
more efficiently, so that when a cluster becomes available it can be equipped with the
information needed to perform a task. More on this is described in the previous section
on Continuous Reconciliation of Components. While environments that have
nonstandard forms of communication cannot be continuously reconciled against, a
fallback pattern is to relocate this information effectively (see next section on
Provenance in Secure Supply Chain) so that the environments can still stand up and
participate in a larger ecosystem of capabilities.

Figure 3: Multi-Cluster Kubernetes relies on a Global Control Plane

Kubernetes APIs are exposed to different users based on Role-Based Access Control
(RBAC) through the delegated authentication, enabling a self-service and API driven
ecosystem. This enables a larger classification of users, rather than just privileged and
non-privileged. For example, the Army DevSecOps Playbook asserts enough controls
and guardrails can be implemented to enable non-privileged developers appropriate
access to their source code and the path to production to reach all security and
compliance requirements. Engineers are able to push new changes to subsequent
environments because there is enough visibility, testing, and predictability for security
and authorizing officials to understand the risk posture each push to the next
environment would create.

UNCLASSIFIED 12
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

This design structure posits that the individual secure components of a software factory
can be scaled and instantiated independently if those services are able to maintain
secure connectivity. For example, containers require a hosting cluster for various stages
in a software development cycle, but the creation of those containers could occur in a
central location. This would allow certain mission owners that operate in a Denied,
Disrupted, Intermittent, and Limited (DDIL) environments for production to build and
develop their containers in a cloud environment while deploying production workloads
onto a cluster hosted in that DDIL environment. As a direct result, mission owners
reduce duplication of software factory development areas and gain the benefit of
flexibility. This also contributes to a much smaller footprint for production environments,
which is sometimes crucial when that environment is DDIL.

Secure Inter-Service Communication

It is critically important to avoid the proprietary APIs that are sometimes added by vendors on
top of the existing CNCF Kubernetes APIs. These APIs are not portable and may create
vendor lock-in at the application level, not just at the software factory level.

3.4 Secure Supply Chain Provenance

One of the critical aspects of modern application security is understanding all of the
components that come together to create a distributed application. Recent security
breaches have been caused by everything from misconfigured data service interfaces to
maliciously altered code injected into an application build. DoD applications are
especially sensitive to security vulnerabilities, and require stringent guardrails against
risk that do not stifle developer productivity and innovation. NIST has a large effort on
informing guidance on the entirety of supply chain securit and this section is meant as
supplemental guidance in the frame of DoD DevSecOps methodologies 19. Provenance
in Secure Supply Chain is meant to address specifically the origins of components when
taken in an immutable environment and making sure that what is being written earlier in
the process is preserved, scanned, and verified through to the later steps in the
software development process.

Most existing software environments rely on a combination of developer discipline and

“review by committee” to create a safe supply chain. Unfortunately, even with the best
of intentions, this approach runs counter to the needs of agile software development

19National Institute of Standards and Technology, “Improving the Nation’s Cybersecurity: NIST’s
Responsibilities under the Executive Order,” [Online] Available at: https://www.nist.gov/itl/executive-order-
improving-nations-cybersecurity.

UNCLASSIFIED 13
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

approaches. In many large organizations, security acceptance reviews can be—by far—
the biggest constraint in the software release process.

The CNCF Multi-Cluster Kubernetes Reference Design provides additional secure

supply chain guidelines beyond existing guidance:

● Source curation: Components such as application source code, libraries, and

shared services, must be vetted before the component is integrated into the
application. Curated repositories of software build components and approved
services (such as database or messaging services) can be combined with
application code and image scanning to help automatically shift key security
activity to the beginning of the application development process. Furthermore, an
advanced software supply chain can make sure security patches are applied
quickly and automatically across a large and complex application portfolio.

● Artifact protection: Elements moving through the supply chain must be

immutable and verifiable, through techniques such as source code and container
image signing. The supply chain system must protect against malicious parties
modifying code or images, without the application team’s knowledge. The supply
chain should also protect against alteration of software process attestations (see
next bullet), configuration files, and anything else that might make the application
more vulnerable to a breach or attack.

● Attestation: One of the really powerful things a modern secure software supply
chain can do is provide verifiable documentation of application composition,
process steps taken to build and deploy the application, and test and vulnerability
scanning output. These artifacts are then always instantly available for audit and
review at any point during or after the application lifecycle. In fact, some aspects
of auditing and verification can be automated using supply chain attestation
artifacts.

To address these needs, additional guidelines to the software supply chain will consist
of:

● Curated inputs from commercial vendors, open source communities, and even
the DoD itself (and its agency partners).
● Process automation for both development and operations processes, including
building, testing, scanning, and deploying software components.
● Observability and verification tools (including security tools such as the
aforementioned vulnerability scanners, governance automation, etc.)

UNCLASSIFIED 14
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Curated inputs can be handled a number of ways, but the selective use of cloud native
buildpacks, service brokers, and private source code management repositories (such as
git) enable the DoD to mitigate risk without having to manage every single input
individually. When managing these inputs consideration should be taken for
compliance needs like NIST SP 800-53 and NIST SP 800-190. 20 There are other DISA
references that apply NIST 800-53 controls to containers, such as section 2 and 3 in the
DevSecOps Enterprise Container Image Creation and Deployment Guide, but this
reference design does not expect adherence to the other examples in that guide and
supports following any process that aligns with the intended outputs of OCI
containers. 21 Similar guidance can be found in the FedRAMP Vulnerability Scanning
Requirements for Containers 22.
Process automation is typically applied to the source code build-test-deploy cycle, but
the larger supply chain can include everything from buildpack management to
vulnerability patching in production. Tools that orchestrate steps can be combined with
a good supply chain platform to compose an optimal flow for any given application.

Finally, verifying the artifact integrity of the software lifecycle requires collecting data on
both the software supply chain tools and the application itself. Analysis and visualization
of this data should be as close to real-time as possible, enabling developers and
operators alike to respond to anomalies quickly. This allows faster recovery times for
issues discovered as early in the software lifecycle as possible.

These tools also allow the software development process to remain largely self-service,
which is critical to developer productivity and experimentation. The DoD strives to
minimize developer toil while also removing as much risk as possible from the software
development lifecycle.

3.5 Production Ready Kubernetes, even in Development

Kubernetes offers a primitive for building platforms within an organization. Most

organizations within the DoD prefer to operate at a higher turn-key solution level and

20National Institute of Standards and Technology, “Application Container Security Guide (SP 800-190),”
[Online] Available at: https://csrc.nist.gov/publications/detail/sp/800-190/final.
21 Defense Information Systems Agency, “Container Image Creation and Deployment Guide, Version 2,
Release 0.6,” [Online] Available at: https://dl.dod.cyber.mil/wp-
content/uploads/devsecops/pdf/DevSecOps_Enterprise_Container_Image_Creation_and_Deployment_G
uide_2.6-Public-Release.pdf.
22 General Services Administration & FedRAMP, “FedRAMP Vulnerability Scanning Requirements for
Containers, Version 1.0, [Online] Available at:
https://www.fedramp.gov/assets/resources/documents/Vulnerability_Scanning_Requirements_for_Contai
ners.pdf.

UNCLASSIFIED 15
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

need compliance, security, and stability more readily available to growing organizations.
By injecting additional capabilities into each Kubernetes cluster as it is created, and
continually reconciling those capabilities to prevent configuration drift, clusters can be
spun up on-demand to meet the needs of a growing ecosystem. Applying these
capabilities into Kubernetes in thoughtful places, like as additional workloads (such as
Daemon Sets), Service API extensions, or Admission Controllers, Kubernetes can meet
all compliance needs from a security technical implementation guide (STIG) or security
requirements guide (SRG) need.

These additional capabilities consist of:

● Storage Integrations, making sure requisitioned storage is properly encrypted

for data at rest for both platform components, the underlying VMs, and critical
supporting databases (like etcd). This is also expandable to secrets at rest being
properly encrypted and in a readily rotatable position.
● Container Networking, making sure the container overlay networking complies
with necessary encryption at transit requirements and conforms to the underlying
network topology. This networking also integrates with policies at the cluster level
to isolate workloads and prevent unnecessary routing internally (allow listing,
internal cluster traffic, and egress).
● Service Routing, used for exposing workloads securely for ingress while making
sure proper certificates and TLS are employed.
● Observability, visibility into logging and metrics of running workloads, but also
forwarding necessary audit logging to the enclave’s audit and logging solution.
● Identity, Credential, and Access Management, providing secured, credentialed
access to users, services, applications, and interaction with the underlying
infrastructure. Limiting this access to the least privileged and making sure
sessions are validated.
● Admission and Validation, scheduling new workloads should be validated
against best practices, and mutating those workloads when possible to meet
those requirements. This provides additional gates prior to entry into a cluster,
some non-exhaustive examples: enforcing only signed images, validating images
against vulnerabilities, applying DoD issued certificate authorities, or inserting
proxy settings. These processes, implemented as controllers running on the
clusters, can provide additional security, governance, and configuration
management.

Implementing these capabilities makes adhering to the compliance guidance laid out in
the Kubernetes STIG distinctly more manageable. Additionally, these capabilities have
a high overlap with the guidance in the Kubernetes Hardening Guidance provided by
the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security

UNCLASSIFIED 16
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Agency (CISA), which all result in a more secure, authorizable, production ready
Kubernetes platform. 23

4 Additional Tools and Activities

The DevSecOps Tools and Activities Guidebook, part of the DevSecOps Fundamentals,
establishes common DevSecOps tools and activities. The guidebook recognizes that specific
reference designs may elevate a specific tool from PREFERRED to REQUIRED, as well as add
additional tools and/or activities that specifically support the nuances of a given reference
design. The following sections identify those tools and activities unique to this reference design
across the Deploy and Monitor phases of the DevSecOps lifecycle.

23 National Security Agency, “Kubernetes Hardening Guidance,” [Online] Available at:

https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETESHARDENINGGUIDANCE.PDF.

UNCLASSIFIED 17
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 1: Tools Spanning all Phases

Tool Features Benefits Inputs Outputs Baseline

CI/CD Creation and execution of The underpinning to A set of A visible set of REQUIRED
Orchestrator automated tasks, jobs, and the CNCF Multi Cluster processes, out outputs to
workflows. Does not have Kubernetes phase workflows, include, but not
to be a single orchestrator, operations. and/or stages limited to:
but can be. that are run - Status
based on some - Resultin
Preferred to be managed trigger - a code g
by the Global Control push, timeline,
outcome
Plane, but not a hard etc. - Logs
requirement.

Identity, Provide RBAC and other Allows for A notion of roles Decisions on REQUIRED
Credential, identity/authorization segmentation and and/or allowing or
and Access services to the cluster and scoping of capabilities. permissions. permitting an
Management its operations action
Requests from
users.

Common The concept here is that Creates mechanisms A request for a Access to a PREFERRE
service/bund there is a shared interface for the various clusters running provisioned, D
le repository for organizations to request and organizations to service/connecti hardened,
common services and/or interface on common on/appliance. authorized
“bundles” - examples can abstractions, instance of the
be applications, network minimizing the requested
connections, databases, workload of any given entity.
etc. team and reducing
redundant work.

UNCLASSIFIED 18
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 2: Security Activities Summary and Cross-Reference

Activities Table Tool Table

Activities Phase Tool Dependencies
Reference Reference

Enforce node immutability Build Table 4 Node mutation prevention tool Table 3

Artifact signing/validation Build Table 4 Release packaging tool Table 3

Container image security Test Table 6 Test tool suite Table 9

scan (DevSecOp
s Tools and
Activities
Guidebook)

Bundle compliance scan Test Table 6 Test tool suite Table 9

(DevSecOp
s Tools and
Activities
Guidebook)

Admission control Test Table 6 Admission controller Table 5

Node rotation Deploy Table 10 Rotation Table 9

automation tools

UNCLASSIFIED 19
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Activities Table Tool Table

Activities Phase Tool Dependencies
Reference Reference

Cluster network Deploy Table 10 Networking service Table 9

management

UNCLASSIFIED 20
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 3: Build Phase Tools

Tools Features Benefits Inputs Outputs Baseline

Container builder Build a container Container image Source code, OCI compliant REQUIRED
image based on build automation language container image
minimal and hardened runtimes,
base images and minimal base
appropriate language images
runtimes.

Bundler Build a relevant Pairs specific Kubernetes OCI compliant REQUIRED

deployment bundle for runtime configuration container artifact:
Kubernetes as a requirements in and a bundle.
container is not a Kubernetes dependent
sufficient mechanism alongside the OCI container
to deploy standalone associated image
and most deployments containers - also
require more than a provides a
single container to mechanism to
run. indicate how
containers should
be run.

Node mutation Packages the Prevents Underlying Read Only PREFERRED

prevention tool underlying operating mutations to the operating operating
systems Kubernetes underlying system for the system/runtime
runs on as Read Only. operating deployed on the deployed
systems making nodes. nodes.
escalations of

UNCLASSIFIED 21
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Tools Features Benefits Inputs Outputs Baseline

privilege and

Release Extends the Signing of OCI compliant Signed bundle. PREFERRED

packaging packaging tool Kubernetes bundle of
referenced in Table 11 deployable deployable
tool of the DevSecOps artifacts. Kubernetes
Tools and Activities objects.
guidebook to strongly
encourage signing
other deployed
artifacts beyond VM’s.

UNCLASSIFIED 22
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 4: Build Phase Activities

Activities Description Inputs Outputs Tool Dependencies

Containerize Packages all required Source code, OCI compliant Container builder
OS components, language runtimes, container image
developed code, minimal base images
runtime libraries, etc.
into a hardened
container

Bundle the Build the deployable Kubernetes OCI compliant Bundler

deployment artifact for configuration and container artifact in a
Kubernetes. dependent OCI bundle.
container image

Enforce node Harden the base Underlying operating Read Only operating Node mutation
immutability nodes so as to system for the system/runtime on the prevention tool
prevent any deployed node deployed node
inadvertent or
malicious mutations to
the shared resource.

Artifact Sign built release Raw deployable Signed deployable Release

signing/validation bundles for non- Kubernetes bundle Kubernetes bundle packaging
repudiation and
attestation. tool

Table 5: Test Phase Tools

UNCLASSIFIED 23
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 5: Test Phase Tools

Tools Features Benefits Inputs Outputs Baseline

Admission Automatically Prevents Policies and Decision on REQUIRED

controller facilitates policy unsuitable, rules. whether the
checking for improperly Bundles to be bundle is
Kuberenetes bundles. configured, or deployed permissible to be
outright malicious deployed
bundles/compone
nts from entering
the Kubernetes
environment.

UNCLASSIFIED 24
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 6: Test Phase Activities

Activities Description Inputs Outputs Tool Dependencies

Container image Scans all layers of Container image Security issues for Test tool suite
security scan container images remediation
down to the OS for
known
misconfigurations and
CVEs

Bundle compliance Often a custom built Bundle and/or it’s Compliance issues for Test tool suite
scan or configured set of components remediation
tests that validate for
adherence to
organizationally
defined controls and
known STIGs/SRGs.

Admission control Checks against Policies and rules. Decision on whether Admission controller
policies and rules to Bundles to be
the bundle is
allow or deny bundle deployed.
deployable.
deployment.

UNCLASSIFIED 25
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 7: Release and Delivery Phase Tools

Tools Features Benefits Inputs Outputs Baseline

Kubernetes bundle Enables the automated Connects the Raw Deployed bundles REQUIRED
reconciliation tool flow of bundle/component source of truth Kubernetes on running
deployment and updating. of declarative bundles (be Kubernetes
An example Kubernetes they clusters.
implementation can be a bundles/compon deployed
GitOps flow. ents via helm,
automatically to yaml, etc.)
the various
running
environments

Table 8: Release and Delivery Phase Activities

Activities Description Inputs Outputs Tool Dependencies

Kubernetes bundle An automated loop for Stored declarative Running bundles on Kubernetes bundle
reconciliation reconciling Kubernetes bundles Kubernetes clusters in reconciliation tool
Kubernetes the appropriate
declarative sources of environment
truth.

UNCLASSIFIED 26
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 9: Deploy Phase Tools

Tools Features Benefits Inputs Outputs Baseline

CNCF Certified Features Benefits described Kubernetes Running cluster(s) with REQUIRED
Kubernetes described in the in the “Container deployment conformant application
“Container Deployment” model configuration deployments.
Deployment” of the DevSecOps
model of the Tools and Activities
DevSecOps Guide
Tools and
Activities Guide

Global control Ability to Enables a form of Commands or Provisioned hardened REQUIRED

plane communicate centralized requests from resources and services
with child management and child for consumption by child
deployed planning while environments environments and their
environments encouraging applications/systems.
and provided decentralized
shared execution - all from
resources like a similar base
services and substrate.
images

Rotation Automatically Allows for Node baseline Baselined and patched REQUIRED
cycles the continuous configuration underlying nodes
automation
underlying reconciliation of
tools
nodes beneath Node state, while
Kubernetes reducing the risks of
potential advanced

UNCLASSIFIED 27
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Tools Features Benefits Inputs Outputs Baseline

persistent threats.

Service routing Ability to Creates a more Requests for Pathways for services to REQUIRED
tool administer secure means for certificates, communicate using the
mutual TLS systems, services, routing policy, provided mTLS/TLS
(mTLS) and and applications to and pathway.
TLS across communicate authentication.
environments across various
network
environments.

Networking Facilitate the Make a secure and Requests for Configuration of the REQUIRED
service communication configurable allow listing or connection and
of Kubernetes mechanism to new connection determination on
pods with both control network creation permissibility of the new
the internal communications of connection request
cluster and pods.
outside world.
Ideally this is
Kubernetes
Container
Networking
Interface (CNI)
compliant. 24

24 Cloud Native Computing Foundation, “CNI – the Container Network Interface,” [Online] Available at: https://github.com/containernetworking/cui.

UNCLASSIFIED 28
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Table 10: Deploy Phase Activities

Activities Description Inputs Outputs Tool Dependencies

Automated service Requesting Request to global The requested Global control plane
and resource requests Kubernetes control plane for some resource to be acted
components, form of resource upon
reconciliation, and/or
usable services from
a centralized location

Node rotation Repave the Time trigger (bi- Baselined and Rotation automation
underlying nodes in weekly at a minimum) patched underlying tool
Kubernetes on a nodes
regular interval to
reprovision all
relevant secrets,
perform updates, and
reconcile drift.

Service to Service Varying network Requests for mTLS Successful Service routing tool
communication services communication. communication across
communicating diverse network
securely with one boundaries.
another.

UNCLASSIFIED 29
Pre-Decisional/DRAFT
 UNCLASSIFIED
Pre-Decisional/DRAFT

Cluster network Ensure that pod and Creation of new A decision on the Networking service
management cluster communication bundles/components permissibility of the
pathways are and/or requests for new connection
secured, known, and new interconnections request.
controlled. within or outside the
cluster.

UNCLASSIFIED 30
Pre-Decisional/DRAFT