10/16/22, 9:06 AM 19.

Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

19. Custom Resource

Definitions in Kubernetes
Overview

In this chapter, we will show how you can use

Custom Resource Definitions (CRDs) to extend
Kubernetes and add new functionality to your
Kubernetes cluster. You will also learn how to
define, configure, and implement a complete CRD.
We will also describe various example scenarios
where CRDs can be very helpful. By the end of this
chapter, you will be able to define and configure a
CRD and a Custom Resource (CR). You will also
learn how to deploy a basic custom controller to
implement the required functionality of the CR in
your cluster.

Introduction
In previous chapters, we learned about different
Kubernetes objects, such as Pods, Deployments,
and ConfigMaps. These objects are defined and
managed by the Kubernetes API (that is, for
these objects, the API server manages their
creation and destruction, among other
operations). However, you may want to extend
the functions provided by Kubernetes to provide
a feature that is not shipped with standard
Kubernetes, and that cannot be enabled by the
built-in objects provided by Kubernetes.

To build these functionalities on top of

Kubernetes, we use Custom Resources (CRs).

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 1/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Custom Resource Definitions (CRDs) allow us

to add a capability through which users can add
custom objects to the Kubernetes server and use
those CRs like any other native Kubernetes
object. A CRD helps us to introduce our custom
objects to the Kubernetes system. Once our CRD
is created, it can be used like any other object in
the Kubernetes server. Not only that, but we can
also use the Kubernetes API, Role-Based Access
Control (RBAC) policies, and other Kubernetes
features for the CRs we have introduced.

When you define a CRD, it is stored in the

Kubernetes configuration database (etcd). Think
of CRDs as the definition of the structure of your
custom object. Once a CRD is defined, Kubernetes
creates objects that abide by the definition of the
CRD. We call these objects CRs. If we were to
compare this to the analogy of programming
languages, CRD is the class and the CR is the
instance of the class. In short, a CRD defines the
schema of a custom object and a CR defines the
desired state of an object that you would like to
achieve.

CRs are implemented via a custom controller. We

will take a closer look at custom controllers in
the first topic of this chapter.

What Is a Custom
Controller?
CRDs and CRs help you define the desired state
for your CRs. There is a need for a component
that makes sure that the state of the Kubernetes

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 2/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

system matches the desired state as defined by

the CR. As you have seen in earlier chapters, the
Kubernetes components that do this are called
controllers. Kubernetes comes up with many of
these controllers whose job is to make sure that
the desired state (for example, the number of
replicas of Pods defined in a Deployment) is
equal to the value defined in the Deployment
object. In summary, a controller is a component
that watches the state of resources through the
Kubernetes API server and attempts to match the
current state with the desired state.

The built-in controllers that are included in a

standard setup of Kubernetes are meant to work
with built-in objects such as Deployments. For
our CRDs and their CRs, we need to write our
own custom controllers.

The Relationship between a CRD, a

CR, and a Controller
The CRD provides a way to define a CR, and
custom controllers provide the logic to act on the
CR objects. The following diagram summarizes
the CRD, CR, and controller:

Figure 19.1: How CRD, CR, and controllers are

tied together

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 3/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

As illustrated in the preceding diagram, we have

a CRD, a custom controller, and the CR object
that defines the desired state as per the CRD.
There are three things to note here:

The CRD is the schema that defines how the

object will look. Every resource has a defined
schema that tells the Kubernetes engine what
to expect in a definition. Core objects such as
PodSpec have schemas that are baked into the
Kubernetes project.
Note
You can find the source code for PodSpec at this
link:
https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/typ
The CR object, which is created based on the
schema (the CRD), defines the desired state of
the resource.
The custom controller is the application that
provides the functionality to bring the current
state to the desired state.

Remember, the CRD is a way through which

Kubernetes allows us to define the schema or
definition for our CRs declaratively. Once our
CRD (the schema) is registered with the
Kubernetes server, a CR (the object) is defined as
per our CRD.

Standard Kubernetes API

Resources
Let's list all the resources and APIs that are
available in the Kubernetes cluster. Recall that
everything we have used is defined as an API

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 4/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

resource, and an API is a gateway through which

we communicate with the Kubernetes server to
work with that resource.

Get a list of all the current Kubernetes resources

by using the following command:

kubectl api-resources
You should see the following response:

Figure 19.2: Standard Kubernetes API resources

In the preceding screenshot, you can see that the

resources defined in Kubernetes have an
APIGroup property, which defines what internal
API is responsible for managing this resource.
The Kind column lists the name of the resources.
As we have seen earlier in this topic, for
standard Kubernetes objects such as Pods, the
schema or definition of a Pod object is built into
Kubernetes. When you define a Pod specification
to run a Pod, this could be said to be analogous
to a CR.

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 5/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

For every resource, there is some code that can

take action against the resource. This is defined
as a group of APIs (APIGroup). Note that multiple
API groups can exist; for example, a stable
version and an experimental version. Issue the
following command to see what API versions are
available in your Kubernetes cluster:

kubectl api-versions
You should see the following response:

Figure 19.3: Various API groups and their

versions

In the preceding screenshot, note that the apps

API group has multiple versions available. Each
of these versions may have a different set of
features that is not available in other groups.

Why We Need Custom

Resources?

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 6/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

As stated earlier, CRs provide a way through

which we can extend the Kubernetes platform to
provide functionalities that are specific to
certain use cases. Here are a few use cases where
you will encounter the use of CRs.

Example Use Case 1

Consider a use case in which you want to

automate the provisioning of a business
application or a database onto the Kubernetes
cluster automatically. Abstracting away the
technical details, such as configuring and
deploying the application, allows teams to
manage them without having an in-depth
knowledge of Kubernetes. For example, you can
create a CR to abstract the creation of a database.
Thus, users can create a database Pod by just
defining the name and size of the database in a
CRD, and the controller will provision the rest.

Example Use Case 2

Consider a scenario where you have self-serving

teams. Your Kubernetes platform is used by
multiple teams and you would like the teams to
provision namespaces and other resources by
themselves. In this case, you want teams to
define the total CPU and memory they need for
the workloads, as well as default limits for a Pod.
You can create a CRD and teams can create a CR
with the namespace name and other parameters.
Your custom controllers would create the
resources they need and associate the correct
RBAC policies for each team. You can also add
additional functionality, such as a team being

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 7/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

restricted to three environments. The controller

can also generate audit events and record all the
activities.

Example Use Case 3

Let's say you are an administrator of a

development Kubernetes cluster where
developers come and test their application. The
problem you are facing is that the developers left
the Pods running and have moved on to new
projects. This may create a resource issue for
your cluster.

In this chapter, we will build a CRD and a custom

controller around this scenario. A solution that
we can implement is to delete the Pod after a
certain amount of time has passed following
their creation. Let's call this time
podLiveForThisMinutes. A further
requirement is to have a configurable way of
defining podLiveForThisMinutes for each
namespace, as different teams may have
different priorities and requirements.

We can define a time limit per namespace and

that would provide the flexibility to apply
controls on different namespaces. To implement
the requirements defined in this example use
case, we will define a CRD that allows two fields
– a namespace name and the amount of time to
allow the Pods to run
(podLiveForThisMinutes). In the rest of this
chapter, we will build a CRD and a controller
that will allow us to achieve the functionality
mentioned here.

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 8/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Note

There are other (better) ways to implement the

preceding scenario. In the real world, a
Kubernetes Deployment object would recreate
the Pod if the Pod had been created using the
Deployment resource. We have chosen this
scenario to keep the example simple and easy to
implement.

How Our Custom Resources

Are Defined
To come up with a solution for Example Use Case
3 in the previous section, we have decided that
our CRD will define two fields, as mentioned in
the preceding example. To accomplish this, our
CR object will look as follows.

apiVersion:
"controllers.kube.book.au/v1"
kind: PodLifecycleConfig
metadata:
  name: demo-pod-lifecycle
spec:
  namespaceName: crddemo
  podLiveForThisMinutes: 1
The preceding specification defines our target
object. As you can see, it looks just like normal
Kubernetes objects, but the specifications (the
spec section) are defined as per our
requirements. Let's dig a bit deeper into the
details.

apiVersion

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 9/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

This is the field required by Kubernetes to group

objects. Note that we put the version (v1) as part
of the group key. This grouping technique helps
us keep multiple versions of our object. Consider
whether you want to add a new property
without affecting existing users. You can just
create a new group with v2, and an object
definition with both versions — v1 and v2 — can
exist at the same time. Because they are
separated, it allows different versions of
different groups to evolve at a different rate.

This approach also helps if we want to test new

features. Say we want to add a new field to the
same object. Then, we could just change the API
version and add the new field. Thus, we can keep
the stable version separate from the new,
experimental version.

kind

This field mentions a specific type of object in a

group defined by apiVersion. Think of kind as
the name of the CR object, such as Pod.

Note

Do not confuse this with the name of the object

that you create using this specification, which is
defined in the metadata section.

Through this, we can have multiple objects

under one API group. Imagine you are about to
create an awesome functionality that would
require multiple different types of objects to be
created. You can have multiple objects using the
Kind field under the same API group.

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 10/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

spec

This field defines the information needed to

define the specification of the object. The
specification contains information that defines
the desired state of our resource. All the fields
that describe the characteristics of our resource
go inside the spec section. For our use case, the
spec section contains the two fields that we need
for our CR – podLiveForThisMinutes and
namespaceName.

namespaceName and
podLiveForThisMinutes

These are the custom fields that we want to

define. namespaceName will contain the name of
the target namespace, and
podLiveForThisMinutes will contain the time
(in minutes) that we want the Pod to be active
for.

The Definition of a CRD

In the previous section, we showed the different

components of a CR. However, before we define
our CR, we need to define a schema, which
governs how the CR would be defined. In the
following exercise, you will define the schema or
the CRD for the resource mentioned in the How
Our Custom Resources Are Defined section.

Consider this example CRD, which we will use in

the following exercise. Let's understand the
important bits of the CRD by observing the
following definition:

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 11/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

pod-normaliser-crd.yaml

1  apiVersion:
apiextensions.k8s.io/v1beta1
2  kind: CustomResourceDefinition
3  metadata:
4    name:
podlifecycleconfigs.controllers.ku
be.book.au
5  spec:
6    group:
controllers.kube.book.au
7    version: v1
8    scope: Namespaced
9    names:
10     kind: PodLifecycleConfig
11     plural: podlifecycleconfigs
12     singular:
podlifecycleconfig
13  #1.15 preserveUnknownFields:
false
14   validation:
15     openAPIV3Schema:
16       type: object
17       properties:
18         spec:
19           type: object
20           properties:
21             namespaceName:
22               type: string
23             podLiveForThisMinut
es:
24               type: integer
Now, let's look at various components of this
CRD:

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 12/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

apiVersion and kind: These are the API and

the resource for the CRD itself and are
provided by Kubernetes for the CRD definition.
group and version: Think of an API group as
a set of objects that are logically related to one
another. These two fields define the API group
and the version of our CR, which will then be
translated into the apiVersion field of our CR,
defined earlier in the previous section.
kind: This field defines the kind of our CR,
defined earlier in the How Our Custom
Resources Are Defined section.
metadata/name: The name must match the
spec fields, and the format is a combination of
two fields – that is, <plural>.<group>.
scope: This field defines whether the CR will
be namespace-scoped or cluster-scoped. By
default, the CR is cluster-scoped. We have
defined it as namespace-scoped here.
plurals: These are plural names to be used in
the Kubernetes API server URL.
openAPIV3Schema: This is the schema that is
defined based on the OpenAPI v3 standards. It
refers to the actual fields/schema of our CR. A
schema is something that defines what fields
are available in our CR, the names of the fields,
and the data types for them. It basically
defines the structure of the spec field in our
CR. We have used the namespaceName and
podLiveForMinutes fields in our CR. You can
see this in step 2 of the following exercise.

It is interesting to know that the component of

the API server that serves the CRs is called
apiextensions-apiserver. When kubectl

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 13/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

requests reach the API server, it first checks

whether the resource is a standard Kubernetes
resource, such as a Pod or a Deployment. If the
resource is not a standard resource, then
apiextensions-apiserver is invoked.

Exercise 19.01: Defining a CRD

In this exercise, we will define a CRD, and in the

next exercise, we will create a CR for the defined
CRD. The definition of the CRD is stored in the
Kubernetes etcd server. Remember that the CRD
and CR are just definitions, and until you deploy
a controller that is associated with your CRs,
there is no functionality attached to the CRD/CR.
By defining a CRD, you are registering a new
type of object with the Kubernetes cluster. After
you define the CRD, it will be accessible via the
normal Kubernetes API and you can access it via
Kubectl:

1. Create a new namespace called crddemo:

kubectl create ns crddemo
This should give the following response:
namespace/crddemo created
2. Now, we need to define a CRD. Create a file
named pod-normaliser-crd.yaml using the
following content:
apiVersion:
apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name:
podlifecycleconfigs.controllers.
kube.book.au
spec:

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 14/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

  group:
controllers.kube.book.au
  version: v1
  scope: Namespaced
  names:
    kind: PodLifecycleConfig
    plural: podlifecycleconfigs
    singular: podlifecycleconfig
  #1.15 preserveUnknownFields:
false
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            namespaceName:
              type: string
            podLiveForThisMinute
s:
              type: integer
3. Using the definition from the previous step,
create the CRD using the following command:
kubectl create -f pod-
normaliser-crd.yaml -n crddemo
You should see the following response:

Figure 19.4: Creating our CRD

4. Verify that the CR is registered with
Kubernetes using the following command:
kubectl api-resources | grep
podlifecycleconfig
You should see the following list of resources:

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 15/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Figure 19.5: Verifying whether the CR has been

registered with Kubernetes
5. Verify that the API is available in the
Kubernetes API server by using the following
command:
kubectl api-versions | grep
controller
You should see the following response:
controllers.kube.book.au/v1

In this exercise, we have defined a CRD, and

now, Kubernetes will be able to know what our
CR should look like.

Now, in the following exercise, let's create a

resource object as per the CRD we defined. This
exercise will be an extension of the previous
exercise. However, we have separated them
because CRD objects can exist on their own; you
don't have to have a CR paired with a CRD. It may
be the case that a CRD is provided by some third-
party software vendor, and you are only
required to create the CR. For example, a
database controller provided by a vendor may
already have a CRD and the controller. To use the
functionality, you just need to define the CR.

Let's proceed to make a CR out of our CRD in the

following exercise.

Exercise 19.02: Defining a CR Using a

CRD

In this exercise, we will create a CR as per the

CRD defined in the previous exercise. The CR will
be stored in the etcd datastore as a normal
Kubernetes object, and it is served by the
https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 16/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Kubernetes API server – that is, when you try to

access it via Kubectl, it will be handled by the
Kubernetes API server:

Note

You will only be able to perform this exercise after

successfully completing the previous exercise in
this chapter.

1. First, make sure that there is no CR for the

podlifecycleconfigs type. Use the
following command to check:
kubectl get podlifecycleconfigs
-n crddemo
If there is no CR, you should see the following
response:
No resources found.
If there is a resource defined, you can delete it
using the following command:
kubectl delete
podlifecycleconfig
<RESOURCE_NAME> -n crddemo
2. Now, we have to create a CR. Create a file
named pod-normaliser.yaml using the
following content:
apiVersion:
"controllers.kube.book.au/v1"
kind: PodLifecycleConfig
metadata:
  name: demo-pod-lifecycle
  # namespace: "crddemo"
spec:
  namespaceName: crddemo
  podLiveForThisMinutes: 1

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 17/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

3. Issue the following command to create the

resource from the file created in the previous
step:
kubectl create -f pod-
normaliser.yaml -n crddemo
You should see the following response:

Figure 19.6: Creating our CR

4. Verify that it is registered by Kubernetes by
using the following command:
kubectl get podlifecycleconfigs
-n crddemo
You should see the following response:
NAME AGE
demo-pod-lifecycle 48s

Note that we are using normal kubectl

commands now. This is a pretty awesome way to
extend the Kubernetes platform.

We have defined our own CRD and have created

a CR against it. The next step is to add the
required functionality for our CR.

Writing the Custom Controller

Now that we have a CR in our cluster, we will

proceed to write some code that acts upon it to
achieve the purpose of the scenario we set out in
the Why We Need Custom Resources section.

Note

We will not teach the actual programming for

writing the Go code for our controller since that is
beyond the scope of this book. However, we will

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 18/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

provide you with the programming logic required

for Example Use Case 3.

Let's imagine that our custom controller code is

running as a Pod. What would it need to do to
respond to a CR?

1. First, the controller has to be aware that a new

CR has been defined/removed in the cluster to
get the desired state.
2. Second, the code needs a way to interact with
the Kubernetes API server to request the
current state and then ask for the desired
state. In our case, our controller has to be
aware of all the pods in a namespace and the
time when the Pods have been created. The
code can then ask Kubernetes to delete the
Pods if the allowed time is up for them, as per
the CRD. Please refer to the Example Use Case 3
section to refresh your memory on what our
controller would be doing.

The logic for our code can be visualized using the

following diagram:

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 19/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Figure 19.7: Flowchart describing the logic for a

custom controller

If we were to describe the logic as simple

pseudocode, it would be as follows:

1. Fetch all the new CRs that have been created

for our custom CRD from the Kubernetes API
server.
2. Register callbacks in case CRs are added or
deleted. The callbacks would be triggered each
time a new CR is added or deleted in our
Kubernetes cluster.
3. If the CR is added to the cluster, the callback
will create a sub-routine that continuously
fetches the list of Pods in the namespace
defined by the CR. If the Pod has been running
for more than the time specified, it will be
terminated. Otherwise, it will sleep for a few
seconds.
4. If the CR is deleted, the callback will stop the
sub-routine.

The Components of the Custom Controller

As mentioned earlier, explaining in detail how

custom controllers are built is beyond the scope
of this book, and we have provided a fully
working custom controller to suit the needs of
Example Use Case 3. Our focus is to make sure
that you can build and execute the controller to
understand its behavior and that you are
comfortable with all the components involved.

Custom controllers are components that provide

functionality against a CR. To provide this, a
custom controller would need to understand

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 20/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

what a CR is meant for and its different

parameters, or the structural schema. To make
our controller aware of the schema, we provide
the details about our schema to the controller
through a code file.

Here is an excerpt of the code for the controller

that we have provided:

types.go

12 type PodLifecycleConfig struct

{
13
14     // TypeMeta is the metadata
for the resource, like kind and
apiversion
15     meta_v1.TypeMeta
`json:",inline"`
16
17     // ObjectMeta contains the
metadata for the particular object
like labels
18     meta_v1.ObjectMeta
`json:"metadata,omitempty"`
19
20     Spec PodLifecycleConfigSpec
`json:"spec"`
21 }
22
23 type PodLifecycleConfigSpec
struct{
24     NamespaceName   string
`json:"namespaceName"`
25     PodLiveForMinutes int
`json:"podLiveForThisMinutes"`

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 21/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

26 }
...
32 type PodLifecycleConfigList
struct {
33     meta_v1.TypeMeta
`json:",inline"`
34     meta_v1.ListMeta
`json:"metadata"`
35
36     Items []PodLifecycleConfig
`json:"items"`
37 }
You can find the complete code at this link:
https://packt.live/3jXky9G.

As you can see, we have defined the

PodLifecycleConfig structure as per our
example of the CR provided in the How Our
Custom Resources Are Defined section. It is
repeated here for easier reference:

apiVersion:
"controllers.kube.book.au/v1"
kind: PodLifecycleConfig
metadata:
  name: demo-pod-lifecycle
  # namespace: "crddemo"
spec:
  namespaceName: crddemo
  podLiveForThisMinutes: 1
Note that in types.go, we have defined objects
that can hold the full definition of this example
spec. Also, notice in types.go that
namespaceName is defined as string and
podLiveForThisMinuets is defined as int. This

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 22/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

is because we are using strings and integers for

these fields, as you can see in the CR.

The next important function of the controller is

to listen to events from the Kubernetes system
that are related to the CR. We are using the
Kubernetes Go client library to connect to the
Kubernetes API server. This library makes it
easier to connect to the Kubernetes API server
(for example, for authentication) and have
predefined request and response types to
communicate with the Kubernetes API server.

Note

You can find more details about the Kubernetes Go

client library at this link:
https://github.com/kubernetes/client-go.

However, you are free to use any other library or

any other programming language to
communicate with the API server over HTTPS.

You can see how we have implemented it by

checking the code at this link:
https://packt.live/3ieFtVm. First, we need to
connect to the Kubernetes cluster. This code is
running inside a Pod in the cluster, and it will
need to connect to the Kubernetes API server. We
need to give sufficient rights to our Pod to
connect to the master server, which will be
covered in the activity later in this chapter. We
will use RBAC policies to achieve this. Please
refer to Chapter 13, Runtime and Network
Security in Kubernetes, to get a refresher on how
Kubernetes implements RBAC functionality.

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 23/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Once we are connected, we use the

SharedInformerFactory object to listen to
Kubernetes events for the controller. Think of
the event as a way for us to be notified by
Kubernetes when a new CR is created or deleted.
SharedInformerFactory is a way provided by
the Kubernetes Go client library to listen to
events generated by the Kubernetes API server.
A detailed explanation of
SharedInformerFactory is beyond the scope of
this book.

The following snippet is an excerpt from our Go

code to create SharedInformerFactory:

main.go

40 // create the kubernetes client

configuration
41     config, err :=
clientcmd.BuildConfigFromFlags("",
"")
42     if err != nil {
43         log.Fatal(err)
44     }
45
46     // create the kubernetes
client
47     podlifecyelconfgiclient,
err :=
clientset.NewForConfig(config)
48
49
50     // create the shared
informer factory and use the
client to connect to kubernetes

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 24/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

51     podlifecycleconfigfactory
:=
informers.NewSharedInformerFactory
WithOptions
(podlifecyelconfgiclient,
Second*30,
52     informers.WithNamespace(os.
Getenv(NAMESPACE_TO_WATCH)))
You can find the complete code at this link:
https://packt.live/3lXe3FM.

Once we have connected to the Kubernetes API

server, we need to register to be notified
whether our CR has been created or deleted. The
following code performs this action:

main.go

62 // fetch the informer for the

PodLifecycleConfig
63 podlifecycleconfiginformer :=
podlifecycleconfigfactory.Controll
ers().V1().
PodLifecycleConfigs().Informer()
64
65 // register with the informer
for the events
66 podlifecycleconfiginformer.AddE
ventHandler(
...
69 //define what to do in case if
a new custom resource is created
70         AddFunc: func(obj
interface{}) {
...

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 25/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

83 // start the subroutine to

check and kill the pods for this
namespace
84             go
checkAndRemovePodsPeriodically(sig
nal, podclientset, x)
85         },
86
87 //define what to do in case if
a  custom resource is removed
88         DeleteFunc: func(obj
interface{}) {
You can find the complete code at this link:
https://packt.live/2ZjtQoy.

Note that the preceding code is an extract from

the full code, and the snippet here is modified
slightly for better presentation in this book. This
code is registering callbacks to the Kubernetes
server. Notice that we have registered for
AddFunc and DeleteFunc. These will be called
once the CR has been created or deleted, and we
can write custom logic against that. You can see
that for AddFunc, a Go subroutine is being called.
For every new CR, we have a separate
subroutine to keep on watching for the Pods
created in the namespace. Also, note that
AddFunc will print out A Custom Resource
has been Added to the logs. You may also have
noticed that in DeleteFunc, we have closed the
signal channel, which will flag the Go
subroutine to stop itself.

Activity 19.01: CRD and Custom

Controller in Action

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 26/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

In this activity, we will build and deploy custom

controllers, CRs, and CRDs. Note that the coding
required for building the custom controller is
beyond the scope of this book and a ready-made
code is provided in the code repository to
facilitate the Deployment of a working
controller.

We will create a new CRD that can take two fields

– a podLiveForThisMinutes field, which
defines the time (in minutes) for a Pod to be
allowed to run before it is killed, and the
namespaceName field, which defines which
namespace these rules will be applied to.

We will create a new CR as per the CRD. Also, we

will create a new Kubernetes role that allows
this new CRD to be queried from the Kubernetes
API server. We will then show you how to
associate the newly created role with the
ServiceAccount named default, which is the
default ServiceAccount that a Pod will use when
we run it in the namespace named default.

Generally, we build a custom controller that

provides logic against the CRD we created. We
will just use the code packaged as a container
and deploy it as a Pod. The controller will be
deployed as a normal Pod.

At the end of the activity, to test our controller,

you will create a simple Pod and verify whether
our custom controller can delete the Pod.

Activity Guidelines:

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 27/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

1. Delete the existing crddemo namespace and

create a new one with the same name.
2. Get the code and the Dockerfile for creating
the controller using the following command:
git
clone  https://github.com/PacktW
orkshops/Kubernetes-Workshop.git
cd
Chapter19/Activity19.01/controll
er
3. Create a CRD with the following fields.
The metadata should contain the following:
name:
podlifecycleconfigs.controllers.
kube.book.au
The OpenAPIV3Schema section should contain
the following properties settings:
openAPIV3Schema:
  type: object
  properties:
    spec:
      type: object
      properties:
        namespaceName:
          type: string
        podLiveForThisMinutes:
          type: integer
4. Create a CR that allows Pods to live for 1
minute in the crddemo namespace.
5. Create a Role that allows the following
permissions for the specified API resources:
rules:
- apiGroups:
["controllers.kube.book.au"]

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 28/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

  resources:
["podlifecycleconfigs"]
  verbs: ["get", "list",
"watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch",
"list", "delete"]
6. Using a RoleBinding object, associate this new
Role with the default ServiceAccount in the
crddemo namespace.
7. Build and deploy the controller Pod using the
Dockerfile provided in step 2.
8. Create a Pod that runs for a long time using the
k8s.gcr.io/busybox image in the crddemo
namespace.
Watch the Pod created in the previous step and
observe whether it is being terminated by our
controller. The expected result is that the Pod
should be created, and then it should be
automatically terminated after about a minute,
as in the following screenshot:

Figure 19.8: The expected output of Activity 19.01

Note

The solution to this activity can be found at the

following address: https://packt.live/304PEoD.

Adding Data to Our Custom

Resource

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 29/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

In the previous activity, you created a CRD and

CR. We mentioned earlier that once we define
our CR, we can query them using standard
kubectl commands. For example, if you would
like to see how many CRs of the
PodLifecycleConfig type have been defined,
you can use the following command:

kubectl get PodLifecycleConfig -n

crddemo
You will see the following response

NAME AGE
demo-pod-lifecycle 8h
Note that it only shows the name and age of the
object. However, if you issue a command for a
native Kubernetes object, you will see a lot more
columns. Let's try that for Deployments:

kubectl get deployment -n crddemo

You should see a response similar to this:

NAME READY UP-TO-DATE AVAILABLE

AGE
crd-server 1/1 1 1 166m
Notice the additional columns that Kubernetes
has added, which provide way more information
about the objects.

What if we want to add more columns so that the

output of the preceding command shows more
details for our CRs? You are in luck, as
Kubernetes provides a way to add additional
information columns for the CRs. This is useful
for displaying the critical values of each type of
custom object. This can be done using additional

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 30/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

data defined in the CRD. Let's see how we can do

that in the following exercise.

Exercise 19.03: Adding Custom

Information to the CR List Command
In this exercise, you will learn how to add
custom information to the CR list obtained by
means of the kubectl get command:

Note

You will only be able to perform this exercise after

successfully completing Activity 19.01, CRD and
Custom Controller in Action.

1. Let's define another CRD with additional

columns. Create a file named pod-
normaliser-crd-adv.yaml with the
following content:
apiVersion:
apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name:
podlifecycleconfigsadv.controlle
rs.kube.book.au
spec:
  group:
controllers.kube.book.au
  version: v1
  scope: Namespaced
  names:
    kind: PodLifecycleConfigAdv
    plural:
podlifecycleconfigsadv

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 31/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

    singular:
podlifecycleconfigadv
  #1.15 preserveUnknownFields:
false
  validation:
    openAPIV3Schema:
      type: object
      properties:
        spec:
          type: object
          properties:
            namespaceName:
              type: string
            podLiveForThisMinute
s:
              type: integer    
  additionalPrinterColumns:
  - name: NamespaceName
    type: string
    description: The name of the
namespace this CRD is applied
to.
    JSONPath:
.spec.namespaceName
  - name: PodLiveForMinutes
    type: integer
    description: Allowed number
of minutes for the Pod to
survive
    JSONPath:
.spec.podLiveForThisMinutes
  - name: Age
    type: date
    JSONPath:
.metadata.creationTimestamp

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 32/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Notice how we have a new section named

additionalPrinterColumns. As the name
suggests, this defines additional information
for your resource. The two important fields of
the additionalPrinterColumns sections are
as follows:
– name: This defines the name of the column to
be printed.
– JSONPath: This defines the location of the
field. Through this path, the information is
fetched from the resources and is displayed in
the corresponding column.
2. Now, let's create this new CRD using the
following command:
kubectl create -f pod-
normaliser-crd-adv.yaml -n
crddemo
You will see the following output:

Figure 19.9: Creating our modified CRD

3. Once we have created the CRD, let's create the
object for the CRD. Create a file named pod-
normaliser-adv.yaml with the following
content:
apiVersion:
"controllers.kube.book.au/v1"
kind: PodLifecycleConfigAdv
metadata:
  name: demo-pod-lifecycle-adv
  # namespace: "crddemo"
spec:
  namespaceName: crddemo
  podLiveForThisMinutes: 20
Now, the fields in the spec section should be
visible in the list obtained by the kubectl get

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 33/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

command, similar to native Kubernetes

objects.
4. Let's create the CR defined in the previous step
using the following command:
kubectl create -f pod-
normaliser-adv.yaml -n crddemo
This should give the following response:

Figure 19.10: Creating our CR

5. Now, let's issue the kubectl get command to
see whether additional fields are displayed:
kubectl get
PodLifecycleConfigAdv -n crddemo
You should see the following information
displayed for our object:
NAME NAMESPACENAME
PODLIVEFORMINUTES AGE
demo-pod-lifecycle-adv crddemo
20 27m
You can see that the additional fields are
displayed and we now have more information
about our CRs.

In this exercise, you have seen that we can

associate additional data for our CR while
querying it via the Kubernetes API server. We
can define the field names and the path for the
data for the fields. This resource-specific
information becomes important when you have
many resources of the same type, and it is also
useful for the operations team to better
understand the resources defined.

Summary

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 34/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

In this chapter, you learned about custom

controllers. As per the Kubernetes glossary, a
controller implements a control loop to watch
the state of the cluster through the API server
and makes changes in an attempt to move the
current state toward the desired state.

Controllers can not only watch and manage user-

defined CRs, but they can also act on resources
such as Deployments or services, which are
typically part of the Kubernetes controller
manager. Controllers provide a way to write
your own code to suit your business needs.

CRDs are the central mechanism used in the

Kubernetes system to extend its capability. CRDs
provide a native way to implement custom logic
for the Kubernetes API server that satisfies your
business requirements.

You have learned about how CRDs and

controllers help provide an extension
mechanism for the Kubernetes platform. You
have also seen the process through which you
can configure and deploy custom controllers on
the Kubernetes platform.

As we come to the end of our journey, let's reflect

on what we have achieved. We started with the
basic concepts of Kubernetes, how it is
architected, and how to interact with it. We were
introduced to Kubectl, the command-line tool to
interact with Kubernetes, and then later, we saw
how the Kubernetes API server works and how
to communicate with it using curl commands.

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 35/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

The first two chapters established the

fundamentals of containerization and
Kubernetes. Thereafter, we learned the basics of
kubectl – the Kubernetes command center. In
Chapter 04, How to Communicate with
Kubernetes (API Server), we looked at how
kubectl and other HTTP clients communicate
with the Kubernetes API server. We consolidated
our learning by creating a Deployment at the end
of the chapter.

From Chapter 5, Pods, through to Chapter 10,

ConfigMaps and Secrets, we dug into concepts
that are critical to understanding the platform
and to start designing applications to run on
Kubernetes. Concepts such as Pods,
Deployments, Services, and PersistentVolumes
enable us to use the platform to write fault-
tolerant applications.

In the next series of chapters, stretching from

Chapter 11, Build Your Own HA Cluster, to
Chapter 15, Monitoring and Autoscaling in
Kubernetes, we learned about installing and
running Kubernetes on a cloud platform. This
covered the installation of the Kubernetes
platform in high availability (HA) configuration
and how to manage network security in the
platform. In this part of the book, you also
looked at stateful components and how
applications can use these features of the
platform. Lastly, this section talked about
monitoring your cluster and setting up
autoscaling.

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 36/37
10/16/22, 9:06 AM 19. Custom Resource Definitions in Kubernetes | The Kubernetes Workshop

Finally, in this last part, starting from Chapter 16,

Kubernetes Admission Controllers, we began
learning about advanced concepts such as how
you can apply custom policies using admission
controllers. You have also been introduced to the
Kubernetes scheduler, a component that decides
where your application will be running in the
cluster. You learned how to change the default
behavior of the scheduler. You have also seen
how CRDs provide a way to extend Kubernetes,
which can be useful not only to build custom
enhancements but also as a way for third-party
providers to add functionality to Kubernetes.

This book serves as a good launchpad to get

started with Kubernetes. You are now equipped
to design and build systems on top of Kubernetes
that can bring cloud-native experience to your
organization. Although this is the end of this
book, it is only the beginning of your journey as
a Kubernetes professional.

Support Sign Out

©2022 O'REILLY MEDIA, INC.  TERMS OF SERVICE PRIVACY POLICY

https://learning.oreilly.com/library/view/the-kubernetes-workshop/9781838820756/B14870_19_Final_SZ_ePub.xhtml 37/37