Certified Kubernetes Administrator

(CKA)
Behrad Eslamifar
[Link]@[Link]
Security
12%
Outlines

● Know how to configure authentication and authorization.

● Understand Kubernetes security primitives.
● Know to configure network policies.
● Create and manage TLS certificates for cluster components
● Work with image securely.
● Define security contexts.
● Secure persistent key value store.
Certification Authority (CA)
Certificate-Based Authentication
TLS Handshake
TLS/SSL Workshop
 Kubernetes Certificates
root@master:~# cd /etc/kubernetes/pki # openssl x509 -in [Link] -noout -text
root@master:/etc/kubernetes/pki# ls ...
[Link] Validity
[Link] Not Before: Aug 12 [Link] 2020 GMT
[Link] Not After : Aug 12 [Link] 2021 GMT
[Link] Subject: CN = kube-apiserver
[Link] ...
... X509v3 Subject Alternative Name:
DNS:ubuntu, DNS:kubernetes,
DNS:[Link], DNS:[Link],
DNS:[Link], IP
Address:[Link], IP Address:[Link]
...

# openssl x509 -in [Link] -noout -text

# openssl verify -verbose -CAfile [Link] [Link]

[Link]: OK
 Renew apiserver Certificate
# cat [Link] # openssl genrsa -out [Link] 2048 (optional)
[ req ]
default_bits = 2048
prompt = no # openssl req -new -key [Link] -out [Link] -
default_md = sha256
req_extensions = req_ext
config [Link]
distinguished_name = dn
# openssl x509 -req -in [Link] -CA [Link] -CAkey
[ dn ]
CN = kube-apiserver
[Link] -CAcreateserial -out [Link] -days 365 -
extensions v3_ext -extfile [Link]
[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1=kubernetes
DNS.2=[Link]
DNS.3=[Link]
DNS.4=[Link]
DNS.5=[Link]
DNS.6=[Link]
DNS.7=ubuntu
IP.1=[Link]
IP.2=[Link]

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth
subjectAltName=@alt_names
 Admin Certificate
# cat /etc/kubernetes/[Link] # echo <client-certificate-data> | base64 -d | openssl
... x509 -noout -text -in - > [Link]
users:
- name: kubernetes-admin # echo <client-key-data> | base64 -d | openssl x509 -
user: noout -text -in - > [Link]
client-certificate-data:
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4a
kNDQWRxZ0F3SUJBZ0lJUWxPMXhxangvUzh3RFFZSktvWk
lodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVml
aWEp1WlhSbGN6QWVGUkg0Tm9JdHZJOE45cz0KLS0tLS1F
TkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data:
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNS
UlFcEFJQkFBS0NBUUVBcjdRMEJMemY1dXpY3BZeWh4WXk
4dQo1eHJxOUFKRU9qcC9KQTNiSWtqUi9nN25rZzUwUHMv
YWFqMnVDNTZiV0ZlemQwOFVvUVJkcVE9PQotLS0tLUVOR
CBSU0EgUFJJVkFURSBLRVktLS0tLQo=
...
 Renew Admin Certificate
# cat [Link] # openssl req -new -key [Link] -out [Link] \
[ req ] -config [Link]
default_bits = 2048
prompt = no # openssl x509 -req -in [Link] -CA \
default_md = sha256 /etc/kubernetes/pki/[Link] \
distinguished_name = dn -CAkey /etc/kubrenetes/pki/[Link] -CAcreateserial \
-out [Link] -days 365 -extensions \
[ dn ] v3_ext -extfile [Link]
O=system:masters
CN=kubernetes-admin

[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=clientAuth

# cat [Link] | base64

Authentication and
Autherization
User Authentication: Who are you?

● Authentication Modules
○ Basic HTTP Auth
○ Static Bearer Tokens
○ X509 Client Certs
○ Service Account Tokens
○ OpenID Connect
○ Custome Webhook
Basic HTTP Auth

● Static Password File

○ API Server option
●
--basic-auth-file=SOMEFILE
password1,user1,uid1,"group1,group2,group3"
password2,user2,uid2,"group1,group2,group3"
Static Bearer Tokens

● Static Token File

○ API Server option
●
--token-auth-file=SOMEFILE
token1,user1,uid1,"group1,group2,group3"
token2,user2,uid2,"group1,group2,group3"
X509 Client Certs

● X509 Client Certs

○ API Server option
●
--client-ca-file=SOMEFILE
○ The common name (CN) as the username
○ The organization fields (O) as the groups

openssl req -new -key [Link] -out [Link] -config [Link]

Service Account Tokens

● Service Account Tokens

○ API Server option
●
--service-account-key-file
○ To use in-cluster process talk to API server
○ To use outside the cluster
○ Authenticated with
●
system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT)
○ Assigned to the groups system
●
serviceaccounts
●
system:serviceaccounts:(NAMESPACE)
Service Account Tokens
Kubectl get serviceaccounts gitlab -o yaml

apiVersion: v1
kind: ServiceAccount
metadata:
# ...
secrets:
- name: gitlab-token-2wsdd

kubectl get secret gitlab-token-2wsdd -o yaml

apiVersion: v1
data:
[Link]: (APISERVER'S CA BASE64 ENCODED)
namespace: ZGVmYXVsdA==
token: (BEARER TOKEN BASE64 ENCODED)
 Authentication
and Autherization:

Authorization with RBAC

Autherization Mode

● Node
○ A special-purpose authorization mode
○ Grants permissions to kubelets
○ API server option: --authorization-mode=Node
● Attribute-Based Access Control
○ Static file with a single “admin” identity
○ A simple file-based autherization policy
○ API server option: --authorization-mode=ABAC

● Role Based Access Control (RBAC)

○ API server option: --authorization-mode=RBAC

● Webhook
○ API server option: --authorization-mode=Webhook
API Server GET Requests
API Server POST Requests
API Server PATCH Requests
API Server DELETE Requests
API Workshop
Access to API
$ kubectl proxy
Starting to serve on [Link]:8001
● Access to API without $ curl localhost:8001/

authentication {
"paths": [
"/api",
"/api/v1",
...

$ curl localhost:8001/api/v1
...
{
"name": "services",
"singularName": "",
"namespaced": true,
"kind": "Service",
"verbs": [
"create",
...
API Structure

● API url to access with “curl”

/<api_basic_path>/<version>/<resources_all>
/<api_basic_path>/<version>/namespaces/<ns>/<rs_type>/<rs_name>
/<api_basic_path>/<version>/namespaces/<ns>/<rs_type>/<rs_name>/<subresource>

/api/v1/namespaces
/api/v1/namespaces/default/secrets/secret-example
/api/v1/namespaces/default/pods/pod-example/log
Create Pod with curl

$ curl -XPOST -H 'Accept: application/json' \

● Create first-pod by -H 'Content-Type: application/json' \
-d { "apiVersion": "v1", "kind": "Pod", \

directly request to "metadata": { "name": "first-pod", "labels": \

{ "role": "front", "app": "nginx" } }, \
"spec": { "containers": [ { "name": "web", \
api server "image": "nginx", "ports": [ { \
"containerPort": 80 } ] } ] } } \
localhost:8001/api/v1/namespaces/default/pods/
 Role Base Access Control:

Role and Cluster Role

User and Groups

● An authentication plugin returns the username and group(s)

● User doesn’t store any information anywhere
○ Actual user
○ Pods
● K8s used it to verfiy authorization
● Groups: Both human users and ServiceAccounts can belong to one
or more groups
● Builtin groups
○ system:unauthenticated
○ system:authenticated
○ system:serviceaccounts
○ system:serviceaccounts:<namespace>
RBAC: Role

● An RBAC Role contains rules that apiVersion: [Link]/v1

represent a set of permissions kind: Role
metadata:
● sets permissions within a particular namespace: default
namespace name: pod-reader
● you have to specify the namespace rules:
- apiGroups: [""] # "" indicates the
core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
RBAC: ClusterRole

● An RBAC ClusterRole contains rules apiVersion: [Link]/v1

that represent a set of permissions kind: ClusterRole
metadata:
● Is a non-namespaced resource # "namespace" omitted since
● Cluster role usage ClusterRoles are not namespaced
○ define permissions on namespaced name: secret-reader
rules:
resources and be granted within
- apiGroups: [""] # "" indicates the
individual namespace(s) core API group
○ define permissions on namespaced resources: ["secrets"]
resources and be granted across all verbs: ["get", "watch", "list"]
namespaces
○ define permissions on cluster-scoped
resourcesok
RBAC: RoleBinding

● Grants the permissions defined in a apiVersion: [Link]/v1

role to a user or set of users kind: RoleBinding
metadata:
● Grants permissions within a specific name: read-pods
namespace namespace: default
● It holds a list of subjects (users, subjects:
- kind: User
groups, or service accounts) name: jane # "name" is case sensitive
● Usage apiGroup: [Link]
roleRef:
○ Reference any Role in the same kind: Role #this must be Role or
namespace ClusterRole
○ reference a ClusterRole and bind name: pod-reader
apiGroup: [Link]
that ClusterRole to the namespace
of the RoleBinding
RBAC: ClusterRoleBinding

● Grant permissions across a whole apiVersion: [Link]/v1

cluster kind: ClusterRoleBinding
metadata:
● After you create a binding, you cannot name: read-secrets-global
change the Role or ClusterRole that it subjects:
- kind: Group
refers to
name: manager
● It holds a list of subjects (users, apiGroup: [Link]
groups, or service accounts) roleRef:
kind: ClusterRole
name: secret-reader
apiGroup: [Link]
 Authentication
and Autherization:

RBAC Workshop
Namespace Admin
apiVersion: [Link]/v1
kind: Role $ kubectl create serviceaccount behrad
metadata: serviceaccount/behrad created
namespace: default
name: namespace-admin $ kubectl apply -f [Link]
rules: [Link]/namespace-admin created
- apiGroups:
- "*"
verbs: $ kubectl apply -f [Link]
- "*" [Link]/namespace-admin-behrad created
resources:
- "*"
$ kubectl config set-credentials behrad --token $TOKEN
apiVersion: [Link]/v1 User "behrad" set.
kind: RoleBinding
metadata: $ kubectl config set-context default-admin --cluster kubernetes --user behrad
name: namespace-admin-behrad Context "default-admin" created.
namespace: default
roleRef: $ kubectl --context default-admin -n kube-system get pods
apiGroup: [Link] Error from server (Forbidden): pods is forbidden: User
kind: Role
name: namespace-admin
"system:serviceaccount:default:behrad" cannot list resource
subjects: "pods" in API group "" in the namespace "kube-system"
- kind: ServiceAccount
name: behrad
namespace: default
Securing Cluster Nodes
Allowing the Pod to
Use the Host’s Linux
Namespaces
hostNetwork
 hostNetwork
apiVersion: v1 $ Kubectl exec -ti pod-with-hostnetwork ip address show
kind: Pod docker0
metadata: Link encap:Ethernet HWaddr [Link]
name: pod-with-hostnetwork inet addr:[Link] Bcast:[Link] Mask:[Link]
spec: ...
lo
hostNetwork: true
...
containers: ens160
- name: main link/ether [Link] brd [Link]
image: busybox inet [Link]/24 brd [Link] scope .. ens160
command: ["/bin/sleep", "999999"]
hostPort
 hostPort
apiVersion: v1 $ Kubectl exec -ti pod-with-hostport ip address show
kind: Pod 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
metadata: UNKNOWN qlen 1000
name: pod-with-hostport link/loopback [Link] brd [Link]
inet [Link]/8 scope host lo
spec:
valid_lft forever preferred_lft forever
containers: 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen
- image: nginx:stable-alpine 1000
name: nginx link/ipip [Link] brd [Link]
ports: 4: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu
- containerPort: 80 1440 qdisc noqueue state UP
hostPort: 80 link/ether [Link] brd [Link]
inet [Link]/32 scope global eth0
protocol: TCP
valid_lft forever preferred_lft forever
 hostPID and hostIPC
apiVersion: v1 $ Kubectl exec -ti pod-with-host-pid-and-ipc ps aux
kind: Pod PID TTY STAT TIME COMMAND
metadata: 1 ? Ss 9:29 /sbin/init
name: pod-with-host-pid-and-ipc 2 ? S 0:01 [kthreadd]
4 ? I< 0:00 [kworker/0:0H]
spec:
6 ? I< 0:00 [mm_percpu_wq]
hostPID: true 7 ? S 3:09 [ksoftirqd/0]
hostIPC: true 8 ? I 18:47 [rcu_sched]
containers: 9 ? I 0:00 [rcu_bh]
- name: main ...
image: busybox
command: ["/bin/sleep", "999999"]
Security Context
Security Context

● Specify the user (the user’s ID)

● Prevent the container from running as root
● Run the container in privileged mode, giving it full access to the
node’s kernel
● Adding or dropping capabilities
● Prevent the process from writing to the container’s filesystem.
● Set SELinux
Security Context

● runAsNonRoot: true (Default: false)

● privileged: true (Default: false)
● capabilities:
○ add
○ drop
● readOnlyRootFilesystem: true (Default: false)
○ only allow them to write to mounted volumes
● fsGroup and supplementalGroups
 Security Context - privileged
apiVersion: v1 $ kubectl exec -ti pod-privileged ls /dev
kind: Pod autofs sda2 tty7
metadata: block sda3 tty8
name: pod-privileged bsg sda4 tty9
spec: btrfs-control sda5 ttyprintk
containers: bus serial ttyS0
- name: main char sg0 ttyS1
image: busybox console shm ttyS10
command: ["/bin/sleep", "999999"] core snapshot ttyS11
securityContext: cpu snd ttyS12eel)
privileged: true
 Security Context - runAsUser
apiVersion: v1 $ kubectl run test-box –image=buxybox
kind: Pod pod/test-box created
metadata: $ kubectl exec -ti test-box -- id
name: pod-as-user uid=0(root) gid=0(root) groups=10(wheel)
spec:
containers: $ kubectl apply -f [Link]
- name: main pod/pod-as-user created
image: busybox $ kubectl exec -ti pod-as-user -- id
command: ["/bin/sleep", "999999"] uid=1001 gid=0(root)
securityContext:
runAsUser: 1001
 Security Context - capabilities
apiVersion: v1 $ kubectl apply -f [Link]
kind: Pod pod/test-box created
metadata: $ kubectl exec -ti pod-settime -- date -s "[Link]"
name: pod-settime [Link]
spec: $ kubectl exec -ti pod-settime – date
containers: Sun Aug 30 [Link] +0430 2020
- name: main
image: busy-box
command: ["/bin/sleep", "999999"]
securityContext:
capabilities:
add:
- SYS_TIME
Docker Default Capabilities

● These capabilities can be drop

○ SETPCAP Modify process capabilities.
○ MKNOD Create special files using mknod(2).
○ AUDIT_WRITE Write records to kernel auditing log.
○ CHOWN Make arbitrary changes to file UIDs and GIDs (see chown(2)).
○ NET_RAW Use RAW and PACKET sockets.
○ DAC_OVERRIDE Bypass file read, write, and execute permission checks.
○ FOWNER Bypass permission checks on operations that normally require the file
system UID of the process to match the UID of the file.
○ FSETID Don’t clear set-user-ID and set-group-ID permission bits when a file is
modified.
○ KILL Bypass permission checks for sending signals.
○ ...
Docker Default Capabilities

● These capabilities can be drop

○ KILL Bypass permission checks for sending signals.
○ SETGID Make arbitrary manipulations of process GIDs and supplementary GID
list.
○ SETUID Make arbitrary manipulations of process UIDs.
○ NET_BIND_SERVICE Bind a socket to internet domain privileged ports (port
numbers less than 1024).
○ SYS_CHROOT Use chroot(2), change root directory.
○ SETFCAP Set file capabilities.
 Security Context – Share Volume
apiVersion: v1 $ kubectl apply -f [Link]
kind: Pod pod/test-box created
metadata: $ kubectl exec -ti pod-with-shared-volume -c first
name: pod-with-shared-volume sh
spec: /$ id
securityContext: uid=1111 gid=0(root) groups=555,666,777
fsGroup: 555
supplementalGroups: [666, 777]
containers:
- name: first
image: busybox
command: ["/bin/sleep", "999999"]
securityContext:
runAsUser: 1111
volumeMounts:
- name: shared-volume
mountPath: /volume
...
volumes:
- name: shared-volume
emptyDir:
Pod Security Policy
PodSecurityPolicy (psp)

● Restricting the use of security-related features in pods

● Is a cluster-level (non-namespaced) resource
● The PodSecurityPolicy is a admission control plugin
● A PodSecurityPolicy resource defines:
○ Whether a pod can use the host’s IPC, PID, or Network namespaces
○ Which host ports a pod can bind to
○ What user IDs a container can run as
○ Whether a pod with privileged containers can be created
○ Which kernel capabilities are allowed, default, always dropped
○ What SELinux labels a container can use
○ Whether a container can use a writable root filesystem or not
○ Which filesystem groups the container can run as
○ Which volume types a pod can use
Enabling Pod Security Policy
apiVersion: [Link]/v1beta2
● Enable when init cluster kind: ClusterConfiguration
kubernetesVersion: v1.18.6
with kubeadm apiServer:
● Enable by edit kube- extraArgs:
advertise-address: [Link]
apiserver manifests enable-admission-plugins:
NodeRestriction,PodSecurityPolicy
...

$ vi /etc/kubernetes/manifests/[Link]
...
- --enable-admission-
plugins=NodeRestriction,PodSecurityPolicy
...
 Default Pod Security Policy
apiVersion: extensions/v1beta1 $ kubectl create clusterrole psp-default \
kind: PodSecurityPolicy --verb=use --resource=podsecuritypolicies \
metadata: --resource-name=default
name: default clusterrole "psp-default" created
spec:
hostIPC: false $ kubectl create clusterrolebinding psp-all-users \
hostPID: false --clusterrole=psp-default \
hostNetwork: false --group=system:authenticated
hostPorts: [Link]/psp-
- min: 10000 all-users created
max: 11000
- min: 13000
max: 14000
privileged: false
readOnlyRootFilesystem: true
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
…
runAsUser, fsGroup, and supplementalGroups
runAsUser: $ kubectl create -f [Link]
rule: MustRunAs Error from server (Forbidden): error when creating
ranges: "[Link]"
- min: 2 : pods "pod-as-user-guest" is forbidden: unable to
max: 2 validate against any pod
fsGroup: security policy: [[Link]: Invalid
rule: MustRunAs value: 100: UID on
ranges: container main does not match required range. Found
- min: 2 100, allowed: [{2 2}]]
max: 10
- min: 20
max: 30
supplementalGroups:
rule: MustRunAs
ranges:
- min: 2
max: 10
- min: 20
Max: 30
 Capabilities
apiVersion: extensions/v1beta1 $ kubectl create -f [Link]
kind: PodSecurityPolicy Error from server (Forbidden): error when creating
spec: "[Link]": pods "ingress-
allowedCapabilities: nginx-controller" is forbidden: unable
- SYS_TIME to validate against any pod security policy:
defaultAddCapabilities: [[Link]: Invalid
- CHOWN value: "NET_ADMIN": capability may not be added]
requiredDropCapabilities:
- SYS_ADMIN
- SYS_MODULE
- NET_ADMIN
 HostPaths
apiVersion: extensions/v1beta1 $
kind: PodSecurityPolicy
spec:
allowedHostPaths:
# This allows "/foo", "/foo/",
"/foo/bar" etc., but
# disallows "/fool", "/etc/foo" etc.
# "/foo/../" is never valid.
- pathPrefix: "/foo"
readOnly: true # only allow read-only
mounts