Kustomize supports 2 main client-side patching methods for Kubernetes
manifests, JSON
Patching and Strategic
Merge Patch. In the JSON Patching method, you have a
"meta" syntax that specifies operation/target/value. In the
Strategic Merge Patch method, you can override values by
providing a patch file with the same structure but with new values, and
it will override the original values (it simply merges the 2 files with
the same structure).
Each method has pros and cons but generally speaking, I would
arguably say that Strategic Merge Patch is better for big
changes/patches, and JSON Patching is better for smaller
fine-grained patches. And for my use case, I will use Strategic
Merge Patch, but I just faced a problem with patching Kubernetes
Custom Resources!
ToC
TL;DR
Kustomize's default patch strategy for the lists
(arrays) is replace. That means the patch list will
override the original list, which is not always the desired behavior.
That behavior could be changed only if an OpenAPI schema for a
Kubernetes resource is available to define the patch strategy.
The OpenAPI schema for Kustomize core resources (like Namespace,
Deployment, Pod, etc.) is already part of Kustomize, so changing the
patch strategy works out of the box for these resources. However, if you
have a Kubernetes Custom Resource, you need to provide to Kustomize the
OpenAPI schema of that custom resource. And that's only useful if the
custom resource includes the OpenAPI extensions related to merging
strategy.
This post shows how to add those extensions to have control over the
patch strategy. You can jump directly to the solution section if you already know all these
details.
1. Task
I want to use Kustomize to patch Kubernetes
Custom Resources like Prometheus AlertmanagerConfig),
and I want to use merge as a patch strategy for
lists. That means the original lists in the same path
should be merged, not overridden by the patch list. That works out of
the box for Kustomize core resources but not for custom resources.
First, let's see that in action, then dive into the explanation
afterward.
2. Issue reproduction
Let's have a look at this example using the core resource
Pod, given this Kustomization file:
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- pod.yaml
patches:
- pod-patch01.yaml
- pod-patch02.yaml
And these resources and patches files:
# pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_01
value: source
# pod-patch01.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_01
value: patch 01
# pod-patch02.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_02
value: patch 02
The kustomize build . will return:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
env:
- name: MY_ENV_VAR_02
value: patch 02
- name: MY_ENV_VAR_01
value: patch 01
As you see, the env key MY_ENV_VAR_01 overrode by the
value from pod-patch01.yaml, and the env key
MY_ENV_VAR_02 has just been added from
pod-patch02.yaml. That's great; the lists are merged based
on the name key.
...
However, if you tried to do that with a CustomResource
like AlertmanagerConfig, it would not work! Let's give it a
try! Given this Kustomization file:
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- alertmanagerconfig.yaml
patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml
And these resources and patches files:
# alertmanagerconfig.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'webhook01'
webhookConfigs:
- url: 'http://example.com/'
# alertmanagerconfig-patch01.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'webhook01'
webhookConfigs:
- url: 'http://example01.com/'
# alertmanagerconfig-patch02.yaml
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: 'webhook02'
webhookConfigs:
- url: 'http://example02.com/'
The kustomize build . will return:
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: webhook02
webhookConfigs:
- url: http://example02.com/
As you see, the last patch from the file
alertmanagerconfig-patch02.yaml replaced everything in the
spec.receivers list, and that's the default behavior in
Kustomize. The patch list will replace everything in the original list.
Why? Because that's the safest choice since Kustomize
doesn't know anything about AlertmanagerConfig schema!
Before diving into the fix, let's learn more about the
why.
3. Background
The Strategic Merge Patch is a client-side merge method
that merges 2 or more Kubernetes manifests together based on the
manifest apiVersion, kind, and
metadata.name. To merge 2 YAML files, you need to decide
the "merge strategy" for different data types, i.e.,
what should happen for the "string", "int", "list", "map", and so on?
Should they merge together? Or do the patch values override the original
values?
Also, each data type could be patched differently; for example, how
to patch a list? Kustomize provides different
patch formats like merge, replace, and
delete. In fact, in a previous post (Delete
a manifest from Kustomize base), I mentioned the delete
patch strategy, which works out of the box with core Kubernetes
primitive (namespace, deployment, pod, etc.), but not the
CustomResources.
Why does it work with core resources only? Because
of 2 things.
- The Kubernetes project includes specific keys (extensions) in the
core resources OpenAPI schema to deal with that. Namely the OpenAPI
extensions x-kubernetes-patch-strategy and
x-kubernetes-patch-merge-key (see them in Kubernetes swagger.json).
- The OpenAPI schema for Kubernetes' core resources is embedded in
Kustomize.
If those keys are not included in the OpenAPI schema, and Kustomize
doesn't have access to the OpenAPI schema, the default behavior will be
applied, which in Kustomize, the patch list will fully
replace the original list.
4. Solution
Now there are 2 cases, First, if the custom resource definition
already has the x-kubernetes-patch-*`, and second, if the
custom resource definition doesn't have them at all. The Kustomize
supports "openapi"
field, which specifies where Kustomize gets its OpenAPI schema.
For the first case, it's easy; you just need to point Kustomize to
the OpenAPI schema and that's it!
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
openapi:
# It could be also a URL.
path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
[...]
However, for the second case, in a Platonic world, you should contact
the upstream to add the OpenAPI extension keys
x-kubernetes-patch-strategy and
x-kubernetes-patch-merge-key. But as you know, in reality,
that will take ages, and in the best-case scenario, it will not happen
overnight! So the pragmatic solution is to tell Kustomize how to deal
with that custom resource via OpenAPI schema.
We will simply get the Custom Resource's OpenAPI schema and add the
x-kubernetes-patch-* keys to it with the merge strategy,
which can also be customized using different
patch formats like merge, replace, and
delete.
The following are the step to get the OpenAPI schema of a
custom resource, clean it, add the merge strategy keys, and finally use
it in kustomization.yaml file.
4.1 Get the custom
resource OpenAPI schema
You can get the OpenAPI schema for the resource from the upstream
project, or if you have already installed its
CustomResourceDefinition, then you can get it directly by
calling Kubernetes API. And since K8s API will return every definition
it has (probably thousands of lines), we will use jq to get
the exact custom resource OpenAPI schema.
Here is a snippet that will help to get the OpenAPI definition for a
particular resource:
get_openapi_definition () {
jq \
--arg group "${1}" \
--arg version "${2}" \
--arg kind "${3}" \
'.definitions | with_entries(select(.value."x-kubernetes-group-version-kind"[0] |
.group==$group and
.version==$version and
.kind==$kind
))'
}
And we can get the schema for the exact resource from Kubernetes API
by running the following (remember, you should have installed the CRD
for that resource into your Kubernetes cluster to be able to do
that):
kustomize openapi fetch | get_openapi_definition "monitoring.coreos.com" "v1alpha1" "AlertmanagerConfig" > alertmanagerconfig_openapi_schema_map.json
4.2 Find the desired key path
Here is most of the manual work, but the good news is that you need
to do it once. We need 2 things, the data of the path
spec.receivers and the key
x-kubernetes-group-version-kind. Open the schema file and
remove everything not under the hierarchy of the path we want to
customize.
Here is what it looks like after removing everything unrelated:
# alertmanagerconfig_openapi_schema_map.json
{
"com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
"properties": {
"spec": {
"properties": {
"receivers": {
"type": "array"
}
},
"type": "object"
}
},
"type": "object",
"x-kubernetes-group-version-kind": [
{
"group": "monitoring.coreos.com",
"kind": "AlertmanagerConfig",
"version": "v1alpha1"
}
]
}
}
4.3 Create the custom
OpenAPI schema file
Now we just need to put everything together adding
x-kubernetes-patch-* keys and the definitions
parent. The final result will look like the following:
# monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
{
"definitions": {
"com.coreos.monitoring.v1alpha1.AlertmanagerConfig": {
"properties": {
"spec": {
"properties": {
"receivers": {
"x-kubernetes-patch-merge-key": "name",
"x-kubernetes-patch-strategy": "merge",
"type": "array"
}
},
"type": "object"
}
},
"type": "object",
"x-kubernetes-group-version-kind": [
{
"group": "monitoring.coreos.com",
"kind": "AlertmanagerConfig",
"version": "v1alpha1"
}
]
}
}
}
4.4
Update kustomization.yaml with the OpenAPI schema file
The final step, we need to tell Kustomize about our custom OpenAPI
schema file as follows (it also accepts YAML files in case you like to
convert it):
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
openapi:
path: monitoring.coreos.com_v1alpha1_alertmanagerconfig.json
resources:
- alertmanagerconfig.yaml
patches:
- alertmanagerconfig-patch01.yaml
- alertmanagerconfig-patch02.yaml
Now the kustomize build . will return:
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: example
spec:
receivers:
- name: webhook02
webhookConfigs:
- url: http://example02.com/
- name: webhook01
webhookConfigs:
- url: http://example01.com/
Great, it works as expected! 🎉️ And the custom resource list is
merged based on the name key (you can choose any merge key
based on your use case).
Conclusion
Kustomize
is super powerful and has many capabilities to manage your entire
Kubernetes infrastructure as code! And the most fantastic thing? It's
now part of kubectl, so it's almost the standard way to
deal with advanced Kubernetes manifest structure.
Enjoy :-)
