Kubernetes

Security Guide.
Adopting a cloud-native architecture and a DevOps
approach to development demands changes to your
IT architecture. Inside you’ll find best practices for
implementing Kubernetes security in your organization.

eBOOK
Kubernetes Security Guide.

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Understanding Kubernetes RBAC and TLS certificates

Chapter 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Implementing security at the pod level: Kubernetes Security
Context, Kubernetes Security Policy and Kubernetes Network Policy

Chapter 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Securing Kubernetes components (kubelet, etcd or your registry)

Chapter 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Hardening kube-system components with
Sysdig Secure security policies
Kubernetes Introduction
Security In this Kubernetes security guide we cover Kubernetes underpin deployment and

Guide.
the most significant aspects of implement- enable dynamic scale of those apps. Of
ing Kubernetes security best practices. course, with all these moving parts, orga-
nizations must take a more dynamic se-
Kubernetes security, like monitoring or curity approach.
building a CI/CD pipeline is now a must
as the Kubernetes platform is quickly be- Many organizations are adopting new se-
coming the defacto standard for modern curity best practices and implementing
containerized deployments. DevSecOps processes, where everyone
is responsible for security and security
To keep up with this rapid change, tra- is implemented from early development
ditional security processes need to be stages into production through the entire
updated for Kubernetes; legacy security software supply chain. This also known as
tools that were not built for containers Continuous Security.
cannot keep up with cloud-native trans-
formation. Containers provide immense Moving Kubernetes to production comes
benefit in the development and deploy- with new infrastructure layers, new com-
ment of applications, enabling substan- ponents, new procedures and therefore
tial portability and isolation. However, new security processes and tools. This
many of the same attributes that make security guide will help you to implement
containers so valuable also make it hard- Kubernetes security, focused on Kuber-
er to understand what’s happening inside, netes-specific security features and con-
monitor, and secure them. At higher level, figuration. We also highlight some ad-
microservices allow for faster application ditional tools that will go beyond what
development and orchestration tools like Kubernetes can do.

Sysdig | REPORT 3
Kubernetes Chapter 1
Security Understanding Kubernetes RBAC
Guide. and TLS certificates
—
Two fundamental components of a Kubernetes Kubernetes RBAC uses the "rbac.authorization.
security authentication and authorization in- k8s.io" API group to drive authorization deci-
volve RBAC and TLS certificates. In this chap- sions. Before getting started, it is important to
ter you’ll learn how to enable and configure understand the API group building blocks:
RBAC permissions, manage credentials for
users and services, and rotate TLS certificates
Namespaces: Logical segmentation
and security tokens.
and isolation, or "virtual clusters"
•• Correct use of Kubernetes
Kubernetes RBAC Overview namespaces is fundamental for
Role-based access control (RBAC) is a method security, as you can group together
of regulating access to computer or network re- users, roles and resources according
sources based on the roles of individual users. to business logic without granting
global privileges for the cluster.
With RBAC you define the actions (i.e. get, up- Typically you use a namespace to
date, delete) that Kubernetes subjects (human group a project, an application, a team
users, software, kubelets) are allowed to perform or a customer.
over Kubernetes entities (pods, secrets, nodes).

Sysdig | REPORT 4
Kubernetes Subjects: The security "actors"
•• Regular users: Humans or other
from a pod ("pod/log") or the
particular resource name like an
Security authorized accesses from outside
the cluster. Kubernetes delegates the
Ingress: "ingress-controller-istio",
including custom resources your

Guide. user creation and management to the deployment defines.

administrator. In practice, this means •• Resources can also refer to Pod
that you can "refer" to a user, as we will Security Policies or PSP.
see on the RBAC examples below, but
there is no User API object per se. Role and ClusterRole: A set of
•• ServiceAccounts: Used to assign permissions over a group of resources
permissions to software entities. similar to a security profile. A Role is
Kubernetes creates its own default always confined to a single namespace,
serviceAccounts, and you can
create additional ones for your while a ClusterRole is cluster-scoped.
pods/deployments. Any pod run by •• Before designing your security policy,
Kubernetes gets its own privileges take into account that Kubernetes
through its serviceAccount, applied to RBAC permissions are explicitly
all processes run within the containers additive, there are no "deny" rules.
of that pod.
•• Some resources only make sense at
•• Groups of users. Kubernetes user the cluster level (i.e. nodes), you need
groups are not explicitly created, to create a ClusterRole to control
instead, the API can implicitly group access in this case.
users using a common property, like
•• Roles define a list of actions that
the prefix of a serviceAccount or the
can be performed over the resources
organization field of a user certificate.
or verbs: GET, WATCH, LIST, CREATE,
As with regular Linux permissions, you
UPDATE, PATCH, DELETE.
can assign RBAC privileges to entire
groups. RoleBindings and ClusterRoleBindings:
Resources: The entities that will Grants the permissions defined in
be accessed by the subjects. a Role or ClusterRole to a subject or
group of subjects. Again, RoleBindings
•• Resources can refer to a generic
are bounded to a certain namespace,
entity ("pod", "deployment", etc.),
subresources like the logs coming ClusterRoleBindings are cluster-global.

Sysdig | REPORT 5
Kubernetes Getting Started
The first step is to define a Role that grants the verbs ["get", "watch", list"] over any pod
Security resource, only in the "default" namespace. Then, create a RoleBinding that grants the per-
missions defined in "pod-reader" to the user "jane".
Guide.
kind: Role
apiVersion: rbac. authorization.k8s.io/v1
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]

# This role binding allows "jane" to read pods in the "default" namespace.
kind: RoleBinding
apiVersion: rbac. authorization. k8s.io/v1
metadata:
name: read-pods
namespace: default
subjects:
- kind: User
name: jane
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac. authorization. k8s.io

Sysdig | REPORT 6
Kubernetes Configuring Kubernetes
RBAC Security
them, you can use this parameter to
just bind to localhost. Make sure this
Security Start by making sure your cluster config-
parameter is not set, or at least not set
to a network-reachable IP address.

Guide.
uration supports RBAC. The location of the •• --anonymous-auth: Enables
configuration file is your kube-apiserver anonymous requests to the secure
manifest, and this depends on the de- port of the API server.
ployment method but it’s usually inside
/etc/kubernetes/manifests in either Creating Kubernetes Users
the master node(s) or the apiserver pod. and ServiceAccounts
You need to look for this flag: ServiceAccounts are used to provide
--authorization-mode=Node,RBAC an identity to the processes that run in
your pods (similar concept to the sshd
While the API server includes of flag op- or www-data users in a Linux system). If
tions, some are best avoided when taking you don’t specify a serviceAccount, these
a best practices approach to security: pods will be assigned to the default ser-
vice account of their namespace.
•• --insecure-port: Opens up access
to unauthorized, unauthenticated By using service-specific service ac-
requests, if this parameter is equal to counts rather than default serviceAc-
0, it means no insecure port. counts you can have granular control the
•• --insecure-bind-address: Ideally, API access granted to any software entity
you should avoid insecure connections inside your cluster.
altogether, but in case you really need

When it comes to users, the best approach is one that applies

the principle of least privilege, which promotes minimal user
profile privileges based on users’ job necessities.
•• Grant the minimum •• Select Role and RoleBinding •• Avoid the use of wildcards
required access privileges to their cluster counterparts, ["*"] when defining access
for the task that a user or when possible. It is much to resources or actions over
pod need to carry out. easier to control security these resources.
when is bounded to
independent namespaces.
Sysdig | REPORT 7
Kubernetes Step-by-Step Instructions for Creating a Kubernetes serviceAccount
Imagine, for example, that your app needs to query the Kubernetes API to retrieve pod in-
Security formation and state changes because you want to notify and send some updates using
webhooks.
Guide. You just need ‘read-only’ access and just want to monitor one specific namespace. Using
a serviceAccount you can grant these specific privileges (and nothing else) to your soft-
ware agent.

The default serviceAccount (the one you will get if you don’t specify any) is not able to
retrieve this information:

$ kubectl auth can-i list pods -n default--as=system:serviceaccount:default:default

no

We have created an example deployment to showcase this Kubernetes security feature.

Sysdig | REPORT 8
Kubernetes Take a look at the rbac/flask.yaml file:

Security apiVersion: v1
kind: Namespace

Guide. metadata:
name: flask
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flask-backend
namespace: flask
---
kind: Role
apiVersion: rbac.authorization. k8s.io/v1
metadata:
name: flask-backend-role
namespace: flask
rules:
- apiGroups: ("")
resources: ("pods"] verbs: ["get", "list", "watch")
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flask-backend-role-binding
namespace: flask

Sysdig | REPORT 9
Kubernetes
Security subjects:
- kind: ServiceAccount

Guide.
name: flask-backend
namespace: flask
roleRef:
kind: Role
name: flask-backend-role
apiGroup: rbac.authorization.k8s.io
---
kind: Deployment
apiVersion: extensions/v1betal
metadata:
name: flask
namespace: flask
spec:
replicas: 2
template:
metadata:
labels:
app: flask
spec:
serviceAccount: flask-backend containers:
- image: mateobur/flask:latest
name: flask
ports:
- containerPort: 5000

Sysdig | REPORT 10
Kubernetes This will create a serviceAccount ("flask backend"), a Role that grants some permissions
over the other pods in this "flask" namespace, a RoleBinding associating the serviceAc-
Security count and the Role, and finally a deployment of pods that will use the serviceAccount:

Guide. $ kubectl create -f flask.yaml

If you query the secrets for the flask namespace, you can verify that an API access token
was automatically created for your serviceAccount:

$ kubectl get secrets -n flask

NAME TYPE DATA AGE

default-token-fjfgn kubernetes.io/service-account-token 3 5m
flask-backend-token-68b6q kubernetes.io/service-account-token 3 5m

Sysdig | REPORT 11
Kubernetes Next, you can check that the permissions are working as you expect with the kubectl
auth command, which can query access for verbs, subjects and impersonate other ac-

Security counts:

Guide. $ kubectl auth can-i list pods -n default --as=system:serviceaccount:flask:flask-backend

no

$ kubectl auth can-i list pods -n flask --as=system:serviceaccount:flask:flask-backend

yes

$ kubectl auth can-i create pods -n flask --as=system:serviceaccount:flask:flask-backend

no

You will need to configure a serviceAccount and its related Kubernetes RBAC permissions
if your software needs to interact with the hosting Kubernetes cluster. Other examples
might include the kubelet agents or a Kubernetes Horizontal Pod Autoscaler.

Sysdig | REPORT 12
Kubernetes Step-by-Step Instructions for Creating a Kubernetes User
As we mentioned before, Kubernetes users do not have an explicit API object that you can
Security create, list or modify.

Guide. Users are bundled as a parameter of a configuration context that defines the cluster
name, (default) namespace and username:

$ kubectl config get-contexts

CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubernetes-admin@kubernetes kubernetes kubernetes-admin

If you take a look at the current context, you will note that the user has client-certificate-da-
ta and client-key-data attributes (omitted in the output by default for security reasons).

$ kubectl config view

...
users:
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED

If you have access to the Kubernetes root certification authority, you can generate a new
security context that declares a new Kubernetes user.

So, in order to create a new Kubernetes user, let's start creating a new private key:

$ openssl genrsa -out john.key 2048

Sysdig | REPORT 13
Kubernetes Then, you need to create a certificate signing request containing the public key and other
subject information:
Security
Guide.
$ openssl req -new -key john.key -out john.csr -subj "/CN=john/O=examplegroup"

Please note that Kubernetes will use the Organization (O=examplegroup) field to deter-
mine user group membership for RBAC.

Now, you have to sign this CSR using the root Kubernetes CA, found in /etc/kubernetes/pki
for this example, the file location in your deployment may vary:

# openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out john.crt
Signature ok
subject=/CN=john/O=examplegroup
Getting CA Private Key

You can inspect the new certificate:

# openssl x509 -in john.crt -text

Certificate:
Data:
Version: 1 (0x0)
Serial Number: 11309651818125161147 (0x9cf3f46850b372bb)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Apr 2 20:20:54 2018 GMT
Not After : May 2 20:20:54 2018 GMT
Subject: CN=john, O=examplegroup
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)

Sysdig | REPORT 14
Kubernetes Let's repeat this process for a second user, so we can show how to assign RBAC permis-
sions to a group:
Security $ openssl genrsa -out mary.key 2048

Guide.
$ openssl req -new -key mary.key -out mary.csr -subj "/CN=mary/O=examplegroup"
# openssl x509 -req -in mary.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out mary.crt

You can now register the new credentials and config context:

$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt

--client-key=/home/newusers/john.key
$ kubectl config set-context john@kubernetes --cluster=kubernetes --user=john
Context "john@kubernetes" created.

$ kubectl config get-contexts

CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubernetes-admin@kubernetes kubernetes kubernetes-admin
john@kubernetes kubernetes john

If you want this file to be portable between hosts you need to embed the certificates in-
line. You can do this automatically appending the --embed-certs=true parameter to
the kubectl config set-credentials command.

Let's use this new user / context:

$ kubectl config use-context john@kubernetes

$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "john" cannot list pods in the namespace "default"

Sysdig | REPORT 15
Kubernetes Ok, this is expected because we haven't assigned any RBAC permissions to our "john" user.

Security Let's go back to our root admin user and create a new clusterrolebinding:

Guide. kubectl config use-context kubernetes-admin@kubernetes

$ kubectl create clusterrolebinding examplegroup-admin-binding --clusterrole=cluster-admin --group=examplegroup
clusterrolebinding "examplegroup-admin-binding" created

$ kubectl config use-context john@kubernetes

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
flask-cap 1/1 Running 0 1m

Please note that we have assigned these credentials to the group rather than the user, so
the user 'mary' should have exactly the same access privileges.

Sysdig | REPORT 16
Kubernetes Kubernetes TLS Certificates
Rotation and Expiration
It is worth mentioning that the current
TLS implementation in the Kubernetes API
Security Modern Kubernetes deployments and
has no way to verify a certificate besides
checking the origin. Neither CRL (Certif-
Guide.
managed cloud Kubernetes providers icate Revocation List) nor OCSP (Online
will properly configure TLS so the com- Certificate Status Protocol) are imple-
munication between the API server and mented. This means that a lost or exposed
the kubelets, users and pods is already certificate will be able to authenticate to
secured, that's why we are going to just the API as long as it hasn't expired.
focus on the maintenance and rotation
aspects of these certificates. There are a few ways to mitigate the im-
pact:
Setting a certificate rotation policy from
the start will protect you against the usu- •• issue (very) short lived certificates to
al key mismanagement or leaking that keep the period of potential exposure
is bound to happen over long periods of small
time. This is often overlooked and nev- •• remove the permissions in RBAC. You
er-expiring tokens are shared between cannot re-use the username until the
administrators for convenience reasons. certificate has expired
We are going to cover 3 scenarios: •• recreate the certificate authority and
issue new certificates to all active
•• kubelet TLS certificate rotation & users
expiration •• consider OIDC (OpenID Connect) as an
•• serviceAccount token rotation alternative authentication method
•• Kubernetes user cert rotation &
expiration

Sysdig | REPORT 17
Kubernetes Kubernetes Kubelet TLS Certificate Rotation
The kubelet is a critical component from the security point of view since it serves as the
Security bridge between the node operating system and the cluster logic.

Guide. By default the kubelet executable will load its certificates from a regular directory that is
passed as argument:

--cert-dir=/var/lib/kubelet/pki/

/var/lib/kubelet/pki# ls
kubelet-client.crt kubelet-client.key kubelet.crt kubelet.key

You can regenerate the certs manually using the root CA of your cluster, however, start-
ing from Kubernetes 1.8 there is an automated approach at your disposal.

You can instruct your kubelets to renew their certificates automatically as the expiration
date approaches using the config flags:

•• --rotate-certificates

and

•• --feature-gates=RotateKubeletClientCertificate=true

By default, the kubelet certificates expire in one year, you can tune this parameter pass-
ing the flag − experimental-cluster-signing-duration to the kube-controller-man-
ager binary.

Sysdig | REPORT 18
Kubernetes Kubernetes ServiceAccount Token Rotation
Every time you create a serviceAccount, a Kubernetes secret storing its auth token is
Security automatically generated.

Guide.
$ kubectl get serviceaccounts
NAME SECRETS AGE
default 1 26d
falco-account 1 18d
sysdig-account 1 12d

$ kubectl get secrets

NAME TYPE DATA AGE
default-token-f2lmn kubernetes.io/service-account-token 3 26d
Falco-account-token-jvgtz kubernetes.io/service-account-token 3 18d
Sysdig-account-token-9sjgd kubernetes.io/service-account-token 3 12d

Sysdig | REPORT 19
Kubernetes You can request new tokens from the API and replace the old ones:

Security $ kubectl delete secret falco-account-token-jvgtz

$ cat > /tmp/rotate-token.yaml <<EOF

Guide. apiVersion: v1
kind: Secret
metadata:
name: falco-account-token
annotations:
kubernetes.io/service-account.name: falco-account
type: kubernetes.io/service-account-token
EOF

If you describe the new secret, you will be able to see the new token string. Please note that
existing pods using this serviceAccount will continue using the old (invalid) token, you
may want to plan a rolling update over the affected pods to start using the new token.

Updating serviceAccount tokens is not as common as updating user certs, passwords,

etc. There is no fully automated way of doing this other than using the Kubernetes API at
the moment. Consider whether or not this security artifact rotation makes sense for your
use cases.

Sysdig | REPORT 20
Kubernetes Kubernetes User TLS Certificate Rotation
As we have seen in the 'How to create a Kubernetes user' example, you can assign a cer-
Security tificate to a user, but there is no User API object per se.

Guide. When you sign the user certificate using Kubernetes root CA, you can assign an expira-
tion date using the -days parameter to enforce routinary rotation:

openssl x509 -req -in john.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -days 365 -out john.crt

$ openssl x509 -in john.crt -text

Certificate:
Data:
Version: 1 (0x0)
Serial Number: 11309651818125161149 (0x9cf3f46850b372bd)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=kubernetes
Validity
Not Before: Apr 3 10:43:25 2018 GMT
Not After : Apr 3 10:43:25 2019 GMT

Then you can replace the old user certificate using the config set-credentials command:

$ kubectl config set-credentials john --client-certificate=/home/newusers/john.crt --client-key=/ho

Summary
In this section we have learned how to enable and configure RBAC permissions in Kuber-
netes, use credentials for users and services and rotate TLS certificates and tokens, cov-
ering the authentication and authorization part of Kubernetes security. Next will continue
with Kubernetes Security Context, Pod Security Policies and Kubernetes Network Policy.

Sysdig | REPORT 21
Kubernetes Chapter 2
Security Implementing security at the pod level:
Guide. Kubernetes Security Context, Kubernetes
Security Policy and Kubernetes Network Policy
—

Configuring Kubernetes Security with Admission Controllers

An admission controller is a piece of code that intercepts requests to the Kubernetes
API server prior to persistence of the object, but after the request is authenticated and
authorized. Admission controllers pre-process the requests and can provide utility func-
tions (like filling out empty parameters with default values), and can also be used to
enforce further security checks.

Admission controllers are found on the kube-apiserver conf file:

--admission-control=Initializers, NamespaceLifecycle, LimitRanger, ServiceAccount,

PersistentVolumeLabel, DefaultStorageClass, DefaultTolerationSeconds, NodeRestriction, ResourceQuota

Sysdig | REPORT 22
Kubernetes Here are the admission controllers that
can help you strengthen security:
PodSecurityPolicy
Acts on creation and modification of the
Security DenyEscalatingExec
pod and determines if it should be admit-
ted based on the requested Security Con-
Guide. Forbids executing commands on an "es- text and the available Pod Security Poli-
cies. The PodSecurityPolicy objects define
calated" container. This includes pods
that run as privileged, have access to the a set of conditions and security context
host IPC namespace, and have access that a pod must declare in order to be ac-
to the host PID namespace. Without this cepted into the cluster, we will cover PSP
admission controller, a regular user can in more detail below.
escalate privileges over the Kubernetes
node just spawning a terminal on these ValidatingAdmissionWebhooks:
containers. Calls any external service that is imple-
menting your custom security policies to
NodeRestriction decide if a pod should be accepted in your
Limits the node and pod objects a kubelet cluster. For example, you can pre-validate
can modify. Using this controller, a Kuber- container images using Grafeas, a con-
netes node will only be able to modify the tainer-oriented auditing and compliance
API representation of itself and the pods engine, or validate Anchore scanned im-
bound to this node. ages. Check out this recommended set of
admission controllers to run depending
on your Kubernetes version.

Sysdig | REPORT 23
Kubernetes Configuring Kubernetes Security Context
When you declare a pod/deployment, you can group several security-related parameters,
Security like SELinux profile, Linux capabilities, etc, in a Security context block:

Guide. ...
spec:
securityContext:
runAsUser: 1000
fsGroup: 2000
...

You can configure the following parameters as part of your security context:

Privileged
Processes inside of a privileged container get almost the same privileges as those out-
side of a container, such as being able to directly configure the host kernel or host net-
work stack.

User and Group ID for the processes, containers and volumes

When you run a container without any security context, the 'entrypoint' command will
run as root as you see here:

$ kubectl run -i --tty busybox --image=busybox --restart=Never -- sh

/ # ps aux
PID USER TIME COMMAND
1 root 0:00 sh

Sysdig | REPORT 24
Kubernetes Using the runAsUser parameter you can modify the user ID of the processes inside a con-
tainer. For example:
Security
Guide.
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
securityContext:
runAsUser: 1000
fsGroup: 2000
volumes:
- name: sec-ctx-vol
emptyDir: {}
containers:
- name: sec-ctx-demo
image: gcr.io/google-samples/node-hello:1.0
volumeMounts:
- name: sec-ctx-vol
mountPath: /data/demo
securityContext:
allowPrivilegeEscalation: false

If you spawn a container using this definition you can check that the initial process is
using UID 1000.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1000 1 0.0 0.0 4336 724 ? Ss 18:16 0:00 /bin/sh -c node server.js

And any file you create inside the /data/demo volume will use GID 2000 (due to the fs-
Groupparameter).

Sysdig | REPORT 25
Kubernetes Security Enhanced Linux (SELinux)
You can assign SELinuxOptions objects using the seLinuxOptions field. Note that SELinux
Security module needs to be loaded on the underlying Linux nodes for this to take effect.

Guide. Capabilities
Linux capabilities break down root full unrestricted access into a set of separate permis-
sions. This way, you can grant some privileges to your software, like binding to a port <
1024, without granting full root access.

There is a default set of capabilities granted to any container if you don't modify the se-
curity context. For example, using chown to set file permissions or net_raw to craft raw
network packages.

Using the pod security context, you can drop default Linux capabilities and/or add
non-default Linux capabilities. Again, applying the principle of least-privilege you can
greatly reduce the damage of any malicious attack taking over the pod.

If you spawn a shell, you can verify that these capabilities have been dropped:

$ kubectl create -f flask-cap.yaml

apiVersion: v1
kind: Pod
metadata:
name: flask-cap
namespace: default
spec:
containers:
- image: mateobur/flask
name: flask-cap
securityContext:
capabilities:
drop:
- NET_RAW
- CHOWN

Note that some securityContext should be applied at the pod level, while other labels are
applied at container level.
Sysdig | REPORT 26
Kubernetes If you spawn a shell, you can verify that these capabilities have been dropped:

Security $ kubectl exec -it flask-cap bash

root@flask-cap:/# ping 8.8.8.8

Guide. ping: Lacking privilege for raw socket.

root@flask-cap:/# chown daemon /tmp
chown: changing ownership of '/tmp': Operation not permitted

AppArmor and Seccomp

You can also apply the profiles of these security frameworks to Kubernetes pods. This
feature is in beta state as of Kubernetes 1.9.

AppArmor, Seccomp or SELinux allow you to define run-time profiles for your containers,
but if you want to define run-time profiles at a higher level with more context, Sysdig Fal-
co and Sysdig Secure can be better options. Sysdig Falco monitors the run-time security
of your containers according to a set of user-defined rules, it has some similarities and
some important differences with the other tools we just mentioned.

Sysdig | REPORT 27
Kubernetes AllowPrivilegeEscalation
The execve system call can grant a newly started program privileges that its parent did
Security not have, such as the setuid or setgid Linux flags. This is controlled by the AllowPrivi-
legeEscalation boolean and should be used with care and only when required.
Guide.
ReadOnlyRootFilesystem
This controls whether a container will be able to write into the root filesystem. It is com-
mon that the containers only need to write on mounted volumes that persist the state,
as their root filesystem is supposed to be immutable. You can enforce this behavior us-
ing the readOnlyRootFilesystem flag:

$ kubectl create -f https://raw.githubusercontent.com/mateobur/kubernetes-securityguide/master/readonly/flask-ro.yaml

$ kubectl exec -it flask-ro bash
root@flask-ro:/# mount | grep "/ "
none on / type aufs (ro,relatime,si=e6100da9e6227a70,dio,dirperm1)
root@flask-ro:/# touch foo
touch: cannot touch 'foo': Read-only file system

Sysdig | REPORT 28
Kubernetes Kubernetes Pod Security Policy
Pod Security Policies or PSP are implemented as an admission controller. Using security
Security policies you can restrict the pods that will be allowed to run on your cluster, only if they
follow the policy we have defined.
Guide. You have different control aspects that the cluster administrator can set:

Control Aspect Field Names

Running of privileged containers privileged

Usage of the root namespaces hostPID, hostIPC

Usage of host networking and ports hostNetwork, hostPorts

Usage of volume types volumes

Usage of the host filesystem allowedHostPaths

White list of FlexVolume drivers allowedFlexVolumes

Allocating an FSGroup that owns the fsGroup

pod's volumes

Sysdig | REPORT 29
Kubernetes Requiring the use of a read only root file
system
readOnlyRootFilesystem

Security
Guide. Control Aspect Field Names

The user and group IDs of the container runAsUser, supplementalGroups

Restricting escalation to root privileges allowPrivilegeEscalation, defaultAllow-

PrivilegeEscalation
Linux capabilities defaultAddCapabilities, requiredDrop-
Capabilities, allowedCapabilities
The SELinux context of the container seLinux

The AppArmor profile used by containers annotations

The seccomp profile used by containers annotations

The sysctl profile used by containers annotations

There is a direct relation between the Kubernetes Pod Security Context labels and the
Kubernetes Pod Security Policies. Your Security Policy will filter allowed pod security con-
texts defining:

•• Default pod security context values (i.e. defaultAddCapabilities)

•• Mandatory pod security flags and values (i.e. allowPrivilegeEscalation: false)
•• Whitelists and blacklists for the list-based security flags (i.e. list of allowed host
paths to mount).

Sysdig | REPORT 30
Kubernetes For example, to define that container can only mount a specific host path you would do:

Security allowedHostPaths:
# This allows "/foo", "/foo/", "/foo/bar" etc., but

Guide. # disallows "/fool", "/etc/foo" etc.

# "/foo/../" is never valid.
- pathPrefix: "/foo"

You need the PodSecurityPolicy admission controller enabled in your API server to enforce
these policies.

If you plan to enable PodSecurityPolicy, make sure first that you configure (or have present
already) a default PSP and the associated RBAC permissions, otherwise the cluster will
fail to create new pods.

If your cloud provider / deployment design already supports and enables PSP, it will come
pre-populated with a default set of policies, for example:

$ kubectl get psp

READONLY
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP ROOTFS VOLUMES
gce.event-exporter false [] RunAsAny RunAsAny RunAsAny RunAsAny false [hostPath secret]
gce.fluentd-gcp false [] RunAsAny RunAsAny RunAsAny RunAsAny false [configMap hostPath secret]
gce.persistent-volume-binder false [] RunAsAny RunAsAny RunAsAny RunAsAny false [nfs secret]
gce.privileged true [*] RunAsAny RunAsAny RunAsAny RunAsAny false [*]
gce.unprivileged-addon false [] RunAsAny RunAsAny RunAsAny RunAsAny false [emptyDir configMap secret]

Sysdig | REPORT 31
Kubernetes In case you enabled PSP for a cluster that didn't have any pre-populated rule, you can
create a permissive policy to avoid run-time disruption and then perform iterative ad-
Security justments over your configuration.

Guide. For example, this policy below will prevent the execution of any pod that tries to use the
root user or group, allowing any other security context:

apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: example
spec:
privileged: true
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: 'MustRunAs'
ranges:
- min: 1
max: 65535
volumes:
- '*'

$ kubectl create -f psp.yaml

podsecuritypolicy "example" created

$ kubectl get psp

READONLY
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP ROOTFS VOLUMES
example true [] RunAsAny RunAsAny MustRunAs RunAsAny false [*]

Sysdig | REPORT 32
Kubernetes If you try to create new pods without the runAsUser directive you will get:

Security $ kubectl create -f https://raw.githubusercontent.com/mateobur/kubernetes-securityguide/master/readonly/flask-ro.yaml

Guide.
$ kubectl describe pod flask-ro
...
Failed Error: container has runAsNonRoot and image will run as root

Kubernetes Network Policies

Kubernetes also defines security at the pod networking level. A network policy is a spec-
ification of how groups of pods are allowed to communicate with each other and other
network endpoints.

Kubernetes supports several third-party plugins that implement pod overlay networks.
You need to check your provider documentation (these for Calico or Weave) to make sure
that Kubernetes network policies are supported and enabled, otherwise, the configura-
tion will show up in your cluster but will not have any effect.

To show how these network policies work, let's use the Kubernetes example scenario
guestbook:

$ kubectl create -f https://raw.githubusercontent.com/fabric8io/kansible/master/vendor/

k8s.io/kubernetes/examples/guestbook/all-in-one/guestbook-all-in-one.yaml

This will create 'frontend' and 'backend' pods:

$ kubectl describe pod frontend-685d7ff496-7s6kz | grep tier

tier=frontend
$ kubectl describe pod redis-master-7bd4d6ccfd-8dnlq | grep tier
tier=backend

Sysdig | REPORT 33
Kubernetes You can use these logical groupings to configure your network policy, abstracting away
concepts like IP address or physical node, that wouldn't work here as Kubernetes can
Security change those dynamically.

Guide. Let's apply the following network policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-backend-egress
namespace: default
spec:
podSelector:
matchLabels:
tier: backend
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
tier: backend

That you can also find in the repository:

$ kubectl create -f netpol/guestbook-network-policy.yaml

Then you can get the pod names and local IP addresses using:
$ kubectl get pods -o wide [...]

Sysdig | REPORT 34
Kubernetes In order to check that the policy is working as expected, you can 'exec' into the 'redis-master'
pod and try to ping first a 'redis-slave' (same tier) and then a 'frontend' pod:
Security
Guide. $ kubectl exec -it redis-master-7bd4d6ccfd-8dnlq bash
$ ping 10.28.4.21
PING 10.28.4.21 (10.28.4.21) 56(84) bytes of data.
64 bytes from 10.28.4.21: icmp_seq=1 ttl=63 time=0.092 ms
$ ping 10.28.4.23
PING 10.28.4.23 (10.28.4.23) 56(84) bytes of data.
(no response, blocked)

As we mentioned before, note that this policy will be enforced even if the pods migrate to
another node or they are scaled up/down.

Sysdig | REPORT 35
Kubernetes Kubernetes Resource Allocation Management
Resource limits are most typically established to avoid unintended saturation due to de-
Security sign limitations or software bugs, but they can also protect you from malicious resource
abuse. Unauthorized resource consumption that tries to remain undetected is becoming
Guide. much more common due to cryptojacking attempts.

There are two basic concepts: requests and limits.

Requests
The Kubernetes node will check if it has enough resources left to fully satisfy the request
before scheduling the pod. Kubernetes makes sure that the actual resource consump-
tion never goes over the configured limits.

You can run a quick example from the resources/flask-resources.yamlrepository file:

apiVersion: v1
kind: Pod
metadata:
name: flask-resources
namespace: default
spec:
containers:
- image: mateobur/flask
name: flask-resources
resources:
requests:
memory: 512Mi
limits:
memory: 700Mi

$ kubectl create -f resources/flask-resources.yaml

Sysdig | REPORT 36
Kubernetes Limits
Use the stress load generator to test the limits:
Security
Guide. root@flask-resources:/# stress --cpu 1 --io 1 --vm 2 --vm-bytes 800M
stress: info: [79] dispatching hogs: 1 cpu, 1 io, 2 vm, 0 hdd
stress: FAIL: [79] (416) <-- worker 83 got signal 9

The resources that you can reserve and limit by default using the pod description are:
•• CPU
•• Main memory
•• Local ephemeral storage

There are some third party plugins and Cloud providers that will extend the Kubernetes
API to allow defining requests and limits over any other kind of logical resources using
the Extended Resources interface. You can also configure resource quotas bounded to a
namespace context.

Summary
This chapter explained how to configure security at the pod level using Kubernetes or-
chestration capabilities, as well as manage Kubernetes resource allocation. Now you
know how to use Kubernetes security context, pod security policy and network policy
resources to define the container privileges, permissions, capabilities and network com-
munication rules.

The next chapter covers securing Kubernetes components.

Sysdig | REPORT 37
Kubernetes Chapter 3
Security Securing Kubernetes components
Guide. (kubelet, etcd or your registry)
—
This chapter explains how to secure The main task of a kubelet is managing
sensitive Kubernetes components and the local container engine (i.e. Docker)
common external resources such as the and ensuring that the pods described in
Docker registry. You will learn how to se- the API are defined, created, run, and re-
cure the Kubelet, the etcd cluster and pull main healthy; and then destroyed when
from trusted Docker repositories. appropriate.

There are two different communication

Kubelet Security interfaces to be considered:
The kubelet is a fundamental piece of
any Kubernetes deployment. It is often •• Access to the Kubelet REST API from
described as the "Kubernetes agent" soft- users or software (typically just the
Kubernetes API entity)
ware, and is responsible for implement-
ing the interface between the nodes and •• Kubelet binary accessing the local
the cluster logic Kubernetes node and Docker engine

Sysdig | REPORT 38
Kubernetes Kubernetes node/
Security
Kubernetes API Kubelet
Docker daemon

Guide. These two interfaces are secured by default using:

•• Security related configuration (parameters) passed to the kubelet binary -

•• NodeRestriction admission controller
•• RBAC to access the kubelet API resources

Below are descriptions of each and tips for verifying they are working as expected.

Kubelet Security: Access to the Kubelet API

The kubelet security configuration parameters are usually passed as arguments to the
binary exec. For newer Kubernetes versions (1.10+) you can also use a kubelet configura-
tion file. Either way, the parameters syntax remain the same.

Let's use this example configuration as reference:

/home/kubernetes/bin/kubelet –v=2 –kube-reserved=cpu=70m,memory=1736Mi –allow-

privileged=true –cgroup-root=/ –pod-manifest-path=/etc/kubernetes/manifests –
experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter –
experimental-check-node-capabilities-before-mount=true –cert-dir=/var/lib/kubelet/
pki/ –enable-debugging-handlers=true –bootstrap-kubeconfig=/var/lib/kubelet/
bootstrap-kubeconfig –kubeconfig=/var/lib/kubelet/kubeconfig –anonymous-auth=false –
authorization-mode=Webhook –client-ca-file=/etc/srv/kubernetes/pki/ca-certificates.crt
–cni-bin-dir=/home/kubernetes/bin –network-plugin=cni –non-masquerade-cidr=0.0.0.0/0 –
feature-gates=ExperimentalCriticalPodAnnotation=true

Sysdig | REPORT 39
Kubernetes Verify the following Kubernetes security settings when configuring kubelet parameters:

Security •• anonymous-auth is set to false to disable anonymous access (it will send 401
Unauthorized responses to unauthenticated requests).

Guide. •• The kubelet has a `--client-ca-file flag, providing a CA bundle to verify client
certificates.
•• The `--authorization-mode is NOT set to AlwaysAllow, as the more secure Webhook
mode will delegate authorization decisions to the Kubernetes API server.
•• The --read-only-port is set to 0 to avoid unauthorized connections to the read-
only endpoint (optional).

Kubelet Security: Kubelet Access to Kubernetes API

As mentioned in Chapter #1, the level of access granted to a kubelet is determined by the
NodeRestriction Admission Controller for RBAC-enabled versions of Kubernetes (stable
in 1.8+).

Kubelets are bound to the system:node Kubernetes clusterrole.

If NodeRestriction is enabled in your API, your kubelets will only be allowed to modify their
own Node API object, and only modify Pod API objects that are bound to their node. You
can check whether you have this admission controller from the Kubernetes nodes exe-
cuting the apiserver binary:

$ ps aux | grep apiserver | grep admission-control

--admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,Persistent
VolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota

Sysdig | REPORT 40
Kubernetes Accessing the Kubelet API with Curl
Typically, just the Kubernetes API server will need to use the kubelet REST API. This inter-
Security face needs to be protected as you can execute arbitrary pods and exec commands on the
hosting node.
Guide. You can try to communicate directly with the kubelet API from the node shell:

# curl -k https://localhost:10250/pods
Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy)

Kubelet uses RBAC for authorization and it's telling you that the default anonymous sys-
tem account is not allowed to connect.

# curl --cacert /etc/kubernetes/pki/ca.crt --key /etc/kubernetes/pki/apiserver-

kubelet-client.key --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt -k https://
localhost:10250/pods | jq .

{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"metadata": {
"name": "kube-controller-manager-kubenode",
"namespace": "kube-system",
...

Your port numbers may vary depending on your specific deployment method and initial
configuration.

Sysdig | REPORT 41
Kubernetes Securing Kubernetes etcd
etcd is a key-value distributed database recommended to grant permission to
Security that persists Kubernetes state. The etcd only those nodes that require access to
configuration and upgrade guide stresses etcd clusters."
Guide. the security relevance of this component:
These restrictions can be enforced using
"Access to etcd is equivalent to root per- regular Linux firewalling (iptables/netfil-
mission in the cluster so ideally, only the ter, etc), with run-time access protection,
API server should have access to it. Con- or with PKI-based authentication and pa-
sidering the sensitivity of the data, it is rameters.

Sysdig | REPORT 42
Kubernetes Run-time Access Protection
An example of run-time access protection could be making sure that the etcd binary only
Security reads and writes from a set of configured directories or network sockets, any run-time
access that is not explicitly whitelisted will raise an alarm.
Guide. Using Sysdig Falco it will look similar to this:

- macro: etcd_write_allowed_directories
condition: evt.arg[1] startswith /var/lib/etcd

- rule: Write to non write allowed dir (etcd)

desc: attempt to write to directories that should be immutable
condition: open_write and not etcd_write_allowed_directories
output: "Writing to non write allowed dir (user=%user.name command=%proc.cmdline file=%fd.name)"
priority: ERROR

- macro: etcd_write_allowed_directories
condition: evt.arg[1] startswith /var/lib/etcd

- rule: Write to non write allowed dir (etcd)

desc: attempt to write to directories that should be immutable
condition: open_write and not etcd_write_allowed_directories
output: "Writing to non write allowed dir (user=%user.name command=%proc.cmdline file=%fd.name)"
priority: ERROR

Sysdig | REPORT 43
Kubernetes PKI-based authentication for etcd
Ideally, you should create 2 sets of certif- •• --peer-client-cert-auth
Security icate and key pairs that are going to be •• --peer-trusted-ca-file=<etcd-
used exclusively for etcd. One pair will ca.cert>
Guide. verify member to member connections
and the other one Kubernetes API to etcd Kubernetes API to etcd cluster TLS
connections. Conveniently, the etcd proj-
ect provides these scripts to help you To allow Kubernetes API to communicate
generate the certificates. with etcd, you will need:

Once you have all the security artifacts •• etcd server parameters:
(certificates, keys and authorities), you •• --cert-file=<path>
can secure etcd communications using •• --key-file=<path>
the following configuration flags: •• --client-cert-auth
•• --trusted-ca-file=<path> (can
etcd peer-to-peer TLS be the same you used for peer to
This will configure authentication and en- peer)
cryption between etcd nodes. To configure •• Kubernetes API server parameters:
etcd with secure peer to peer communi-
•• --etcd-certfile=k8sclient.
cation, use the flags: cert
•• --peer-key-file=<peer.key> •• --etcd-keyfile=k8sclient.key
•• --peer-cert-file=<peer.cert>

Sysdig | REPORT 44
Kubernetes Using a Trusted Docker Registry
If you don't specify otherwise, Kubernetes will pull Docker images from the public regis-
Security try Docker Hub. This is fine for testing or learning environments, but not ideal for produc-
tion, since most organizations want to keep images and content private.
Guide. Allowing users to pull images from a public registry is essentially giving access inside
your Kubernetes cluster to any random software found on the Internet. Most of the pop-
ular Docker image publishers curate and secure their software, however you don't have
any guarantee that your developers are going to pull from trusted authors only.

Providing a trusted repository using cloud services (Docker Hub subscription, Quay.io,
Google/AWS/Azure also provide their own service) or locally rolling your own (Docker reg-
istry, Portus or Harbor, etc.) are two ways to solve this problem.

You will pre-validate and update every image in your registry. Appart from any QA and
testing pipeline you regularly apply to your software, this usually means scanning your
Docker images for known vulnerabilities and bad security practices.

Assuming you already have a pre-populated trusted repository, you need to tell Kuberne-
tes how to pull from it and ideally, forbid any other unregistered images.

Configure private Docker registry in Kubernetes

Kubernetes provides a convenient way to configure a private Docker registry and store
access credentials, including server URL, as a secret:

$ kubectl create secret docker-registry regcred --docker-server=<your-registry-server>

--docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Sysdig | REPORT 45
Kubernetes This data will be base64 encoded and included inline as a field of the new secret:

Security {
"apiVersion": "1",

Guide. "data" : {


".dockercfg": "eyJyZWdpc3RyeS5sb2NhbCI6eyJ1c2VybmFtZ
SI6ImpvaG5kb3ciLCJwYXNzd29yZCI6InNlY3JldHBh
}
"kind": "Secret",
"metadata": {
"creation Timestamp":"2018-04-08T19:13:52Z",
"name": "regcred",
"namespace": "default",
"resourceVersion": "1752908",
"selfLink": "/api/v1/namespaces/default/secrets/regcred",
"uid": "f9d91963-3b60-11e8-96b4-42010a800095"
},
"type": "kubernetes.io/dockercfg"
}

Then, you just need to import this secret using the label imagePullSecrets in the pod definition:

spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: regcred

You can also associate a serviceAccount with imagePullSecrets, the deployments / pods
using such serviceAccount will have access to the secret containing registry credentials.

Sysdig | REPORT 46
Kubernetes Kubernetes Trusted Image
Collections: Banning Non-
sha256 hash codes. Any non-whitelisted
image will fire an alarm and container ex-
Security trusted Registry ecution will be immediately stopped.

Guide. Once you have created your trusted im-

age repository and Kubernetes pod de- Summary
ployments are pulling from it, the next This chapter provided tips on how to se-
security measure is to forbid pulling from cure sensitive Kubernetes components
any non-trusted source. and common external resources such as
the Docker registry. Now you know how to
There are several, complementary ways secure the Kubelet, the etcd cluster and
to achieve this. You can, for example, use pull from trusted Docker repositories.
ValidatingAdmissionWebhooks. This way,
the Kubernetes control plane will delegate In the next chapter will offer plenty of
image validation to an external entity. practical examples and use case scenar-
ios covering Kubernetes runtime threat
Using Sysdig Secure, you can also cre- detection.
ate an image whitelist based on image

Sysdig | REPORT 47
Kubernetes Chapter 4
Security Hardening Kube-system Components with
Guide. Sysdig Secure Security Policies
—
In this chapter we cover best practices for Much of Kubernetes security content
implementing run-time security on the available focuses tuning cluster configu-
kube-system components (kubelet, api- ration parameters, restricting user privi-
server, scheduler, kubedns, etc.) deployed leges, and secret management. But what
in Docker containers. if somebody finds a way to bypass those?
Or a software component does something
One of the main sources of concern for unexpected because it suffers from a bug
companies approaching the container or security vulnerability that your static
paradigm has traditionally been security. scanning will never catch?
It’s a radical infrastructure switch after
all, and certain level of caution is perfect- Run-time security provides an extra lay-
ly healthy. er of protection for those times malicious
users or software behave in a way you
Kubernetes devs are aware of this and the didn’t prepare for. Now let’s create and
platform has improved leaps and bounds test Kubernetes run-time security poli-
in this respect. Work on the RBAC API, in- cies using Sysdig Secure, our container
tegrated secrets vault or certificate rota- native security and forensics product.
tion mechanisms are the latest examples
of this effort.

Sysdig | REPORT 48
Kubernetes kube-system Security:
Core Components
no node assigned, and selects a node for
them to run on. From version 1.6 onwards,
Security Below is a list of the relevant kube-system
you can plug in your own custom Kuber-
netes scheduler.
Guide.
components discussed in this chapter:
kube-dns
kube-apiserver
An internal cluster DNS server. Kube DNS
The central communications hub of the will automatically configure the registers
cluster. Provides REST endpoints to in- for Kubernetes namespaces, services,
teract with the other cluster entities and and pods. It enables the pods to easily lo-
stores the distributed state in the etcd cate other services in the cluster.
backend.
kubelet
etcd
The cluster agent that runs on every Ku-
The database backend where the cluster bernetes node. The kubelet launches the
configuration, state, and related informa- pods using the available container engine
tion persists. (Docker, rkt, etc) and periodically checks
and reports pod status.
kube-controller-manager
This component implements the main kube-proxy
control loo. In other words, it observes Another service that runs on every node,
the differences between the current and providing the necessary network transla-
desired cluster states and performs the tion between service endpoints and pods.
changes needed to move towards desired
state. When you launch a ReplicationCon- The kubelet and kube-proxy run as pro-
troller, it is included in this component. cesses directly in the Kubernetes nodes,
while the other components typically run
kube-scheduler as cluster Docker containers inside the
Watches newly created pods that have kube-system namespace.

Sysdig | REPORT 49
Kubernetes Kubernetes Security by Default with Sysdig Secure
We are going to produce a security ruleset for the components listed previously, using a
Security whitelisting approach (explicitly declaring valid entities, banning everything else). The
container/microservice paradigm goes very well with this approach because containers
Guide. are already minimal and predictable by design.

Sysdig Secure
Sysdig Secure, part of the Sysdig Container Intelligence Platform, is designed to provide
container run-time security and forensics for enterprises. Sysdig Secure's deep contain-
er visibility provides insight into what’s happening inside the containers and leverages
key orchestration technologies such as Kubernetes, Docker, OpenShift, Amazon ECS to
bring in metadata and apply rules from a service and application perspective.

The Sysdig Secure rule syntax is the same used in Sysdig Falco, and is documented here.

Sysdig | REPORT 50
Kubernetes kube-system Process Security
One of the easiest and most effective whitelists that you can configure is the list of al-
Security lowed processes. This task that would be somewhat tedious and error prone in a classi-
cal all-purpose server, but it's quite straightforward for microservices.
Guide. For example, let's see the processes on our etcd service container:

$ kubectl exec -it etcd --namespace=kube-system sh

/ # ps aux
PID USER TIME COMMAND
1 root 34:42 etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd
41 root 0:00 sh
45 root 0:00 ps aux

Note that anything beyond the etcd process would be extremely suspicious.

Let's take a look to the API server:

$ kubectl exec -it kube-apiserver --namespace=kube-system sh

/ # ps aux
PID USER TIME COMMAND
1 root 117:34 kube-apiserver --insecure-port=0 --allow-privileged=true --requestheader-username
-headers=X-Remote-User --service-cluster-ip-range=10.96.0.0/12 --proxy-client
40 root 0:00 sh
44 root 0:00 ps aux

Similarly, we see a single process running here.

Sysdig | REPORT 51
Kubernetes Thus, you can write a Sysdig Secure list and rule similar to this one:

Security - list: etcd_authorized_processes

items: [etcd]

Guide. - rule: Etcd allowed processes

desc: Whitelist of authorized etcd processes
condition: spawned_process and not proc.name in etcd_authorized_processes
output: Unauthorized process (%proc.cmdline) running in (%container.id)
priority: ERROR

And the following Kubernetes security policy:

Sysdig | REPORT 52
Kubernetes Here you are restricting the scope of this rule to kubernetes.namespace.
name=kube-system and Docker containers with the label component=etcd. This secu-

Security rity violation is critical enough to stop the container immediately (see Actions) and of
course, in a real scenario you would configure several notification channels such as
Guide. email or Slack, and you can also go for webhooks, VictorOps, PagerDuty, etc.

Let's trigger it the policy we just created, opening a shell in the etcd container and run-
ning any process:

/ # ls
bin dev etc home proc root sys tmp usr var
/ # user@localhost:~/kubernetes$

Notice the automatic expulsion from container. The container where the command was
running has been inmediately killed by Sysdig.

Sysdig | REPORT 53
Kubernetes You will get the event in your Sysdig Secure stream and clicking on it you will be able to
see the detail (just including the sections relevant for this example below):
Security
Guide.

For the sake of brevity, we are only going to include the YAML rules for the next sections,
as you have seen, creating and testing the corresponding Sysdig Secure policies is a
straightforward process.

Sysdig | REPORT 54
Kubernetes Trusted containers in Kubernetes kube-system
In conjunction with other tools that run your own images registry, Sysdig Secure offers
Security an additional layer of run-time security against the use of untrusted containers. This is
especially important for the kube-system namespace where the allowed list of pods is
Guide. pretty small and immutable.

The following rule will automatically detect (and kill) any container that doesn't come
from any of the allowed images:

- macro: allowed_containers
condition: (container.image startswith gcr.io/google_containers/etcd-amd64 or
container.image startswith gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64 or
container.image startswith gcr.io/google_containers/k8s-dns-kube-dns-amd64 or
container.image startswith gcr.io/google_containers/k8s-dns-sidecar-amd64 or
container.image startswith gcr.io/google_containers/kube-apiserver-amd64 or
container.image startswith gcr.io/google_containers/kube-controller-manager-amd64 or
container.image startswith gcr.io/google_containers/kube-proxy-amd64 or
container.image startswith gcr.io/google_containers/kube-scheduler-amd64)

- rule: Unallowed container running in kube-system namespace

desc: Unallowed container running in kube-system namespace
condition: container and not allowed_containers
output: Unallowed container running in kube-system namespace (%container.info)
priority: ERROR

Sysdig | REPORT 55
Kubernetes Detect Processes Attempting to Access a Secret File After Boot
Kubernetes has a secrets mechanism to securely initialize pods with artifacts like pri-
Security vate keys, passwords, tokens, etc. Generally, a pod will need to access its secrets during
start up. You can detect a long running process that attempts to access a secret file.
Guide. Assuming you mount your secrets under /etc/secrets (replace with your configured
path) this rule will detect and react when a process in a container reads those sensible
files:

- macro: proc_is_new
condition: proc.duration <= 5000000000

- rule: Read secret file after startup

desc: >
an attempt to read any secret file (e.g. files containing user/password/authentication
information) Processes might read these files at startup, but not afterwards.
condition: fd.name startswith /etc/secrets and open_read and not proc_is_new
output: >
Sensitive file opened for reading after startup (user=%user.name
command=%proc.cmdline file=%fd.name)
priority: WARNING

Sysdig | REPORT 56
Kubernetes Detect Pods Attempting to Connect to Their Local Kubelet
In a Kubernetes cluster, the API server talks to the kubelets and these set up the pods in
Security each node. Usually, the pods shouldn't connect to the kubelet or try to scrape its metrics
unless you have a service that explicitly needs that. Let's fire an alarm to see if we can
Guide. detect this behavior.

The API server connects to the kubelet service using port 10250. 10255 and 10248 (now
deprecated) are read-only health check and stats ports. To detect any pod connecting to
the local kubelet, we will use a rule like this:

- macro: kubelet_ports
condition: fd.sport in (10248, 10250, 10255)

- rule: Pod connecting to kubelet

desc: A pod is opening an outbound network connection to the local kubelet
condition: outbound and fd.sip = "127.0.0.1" and kubelet_ports
output: Pod connecting to local kubelet (command=%proc.cmdline %container.info connection=%fd.name)
priority: WARNING

Sysdig | REPORT 57
Kubernetes Kube-system User Security Policies
Another common attack symptom could be the modification of attributes and permis-
Security sions of the default users to gain access to privileged information (forging an URL for
example).
Guide. These kube-system components do not need to modify or create any new user, you can
detect and alert if any container tries to perform these actions.

Conveniently enough, there are already lists and macros for the user management tools.
The following rule makes it quite easy:

- rule: User management operations

desc: User created or modified inside a pod
condition: proc.name in (user_mgmt_binaries) and not proc.pname in (cron_binaries, systemd, run-parts)
output: User management binary run on container (command=%proc.cmdline %container.info)
priority: ERROR

Let's test this policy:

$ kubectl exec -it kube-dns-545bc4bfd4-zm282 --namespace=kube-system sh

Defaulting container name to kubedns.
Use 'kubectl describe pod/kube-dns-545bc4bfd4-zm282' to see all of the containers in this pod.
/ # passwd postgres

In this case the default ruleset of Sysdig Secure already detects any write below /etc, of-
fering extra protection in the event an attacker tries to change the valid sources of users,
for example modifying /etc/nsswitch.conf ;-).

Sysdig | REPORT 58
Kubernetes Kubernetes File System Security: Write-allowed Directories
Most container directories are generally static and read-only. The list of write-allowed
Security directories should be easy to define and any violation immediately detected. This is an
example to detect writes outside the allowed path:
Guide.
- macro: etcd_write_allowed_directories
condition: evt.arg[1] startswith /var/lib/etcd

- rule: Write to non write allowed dir (etcd)

desc: attempt to write to directories that should be immutable
condition: open_write and not etcd_write_allowed_directories
output: "Writing to non write allowed dir (user=%user.name command=%proc.cmdline file=%fd.name)"
priority: ERROR

Kubernetes Network Security: Processes Opening a Listening Port

Another interesting whitelist is the list of processes that can open a listening connec-
tion. For example, in this rule we only let kube-proxy to open listening ports:

- list: kube_proxy_port_processes
items: [kube-proxy]

- rule: Unauthorized process opened a port

desc: A kube_proxy process tried to open a port and is not whitelisted
condition: evt.type=listen and not proc.name in (kube_proxy_port_processes)
output: Non-whitelisted process opened a port (command=%proc.cmdline connection=%fd.name)
priority: WARNING

Sysdig | REPORT 59
Kubernetes Kubernetes Network Security: Processes
Opening an Outbound Connection
Security The list of processes that can initiate a connection from your kube-system pods

Guide.
should be fairly limited as well. For example here, only kube-apiserver can initiate an
outbound connection:

- list: kube_apiserver_outbound_processes
items: [kube-apiserver]

- rule: Unauthorized process opened an outbound connection

desc: A kube-apiserver process tried to open an outbound connection and is not whitelisted
condition: outbound and not proc.name in (kube_apiserver_outbound_processes)
output: Non-whitelisted process opened an outbound connection (command=%proc.cmdline connection=%fd.name)
priority: WARNING

Let's try it:

$ kubectl exec -it kube-apiserver --namespace=kube-system sh / # wget www.google.com

Alert description text:

Non-whitelisted process opened an outbound connection (command=wget www.google.com

connection=10.0.11.228:41636->172.217.13.68:80)

Sysdig | REPORT 60
Kubernetes Kubernetes Network Security: Detecting NodePort Endpoints
A human error or maybe a malicious user can configure a service as type Nodeport, thus
Security bypassing the firewalls and other security measures that you have configured for your
load balancers:
Guide. 30000 to 32767 is the default port range in Kubernetes for NodePort services. This is a
rule to detect that:

- rule: Unexpected NodePort connection

desc: A service has been declared using type NodePort
condition: (outbound or inbound) and fd.sport >= 30000 and fd.sport <= 32767
output: A service is using a NodePort connection (command=%proc.cmdline connection=%fd.name)
priority: WARNING

Sysdig | REPORT 61
Kubernetes Default Sysdig Secure
Rules and Policies
Security In addition to the security rules and policies Launch Sensitive Mount Container

Guide.
we have listed here, the default Sysdig Se- detect the initial process started by a con-
cure policies had been designed to protect tainer that has a mount from a sensitive
containers from several common threats host directory (i.e. /proc). Exceptions are
and attacks. Most of them will be useful to made for known trusted images.
protect your kube-system out of the box, or
with some minor customizations.
Launch Privileged Container
You will find policies including: detect the initial process started in a priv-
ileged container. Exceptions are made for
Modify binary dirs known trusted images.
an attempt to modify any file below a set
of binary directories System procs network activity
any network activity performed by system
Change thread namespace binaries that are not expected to send or
an attempt to change a program/thread receive any network traffic.
namespace (commonly done as a part of
creating a container) by calling setns

Sysdig | REPORT 62
Kubernetes This input file will auto generate the mentioned whitelisting rules for processes, write-al-
lowed directories, process allowed to start outbound connections and process allowed to
Security open listening ports.

Guide. This simple python script automatically generate a ruleset that you can directly copy
and paste as custom rules for Sysdig Secure

$ ./generate-secure-rules.py input_rules.yaml > output_rules.yaml

Summary
Strict run-time security for your kube-system pods is an effective mechanism against
any attack that has already bypassed your existing security measures or that managed
to exploit a new vulnerability.

Get a free Sysdig Secure trial and protect your containers and microservices with run-
time security or learn more about Sysdig Falco (single host, command line only), the
open source relative of Sysdig Secure.

Sysdig | REPORT 63
Kubernetes Auto Generate Kubernetes Security Rules and Policies
Creating all these whitelisting rules by hand can be laborious, Fortunately, you can
Security auto-generate the rules that will be used as base for your policies using the following
YAML format:
Guide.
- podname: etcd
proc: [etcd]
write_dir: [/var/lib/etcd]
outbound_proc: [etcd]
listen_proc: [etcd]

- podname: kube_apiserver
proc: [kube-apiserver]
write_dir: false
outbound_proc: [kube-apiserver]
listen_proc: [kube-apiserver]

- podname: kube_dns
proc: [dnsmasq, dnsmasq-nanny, sidecar, kube-dns]
write_dir: [/var/run/dnsmasq.pid, /dev/null]
outbound_proc: [kube-dns]
listen_proc: [kube-dns, sidecar, dnsmasq]

- podname: kube_controller
proc: [kube-controller-manager]
write_dir: false
outbound_proc: [kube-controller-manager]
listen_proc: [kube-controller-manager]

- podname: kube_scheduler
proc: [kube-scheduler]
write_dir: false
outbound_proc: [kube-scheduler]
listen_proc: [kube-scheduler]

Sysdig | REPORT 64