Promotions Between Kustomizations

2024‑11-20 Kubernetes K8S Kustomize kustomize-promote open-source

  Wander Boessenkool

In a typical containerized application we have a lifecycle that goes a little like this:

1. Developers push code to an application repository.
2. A pipeline builds the latest code and pushes the resulting image to a registry
3. The pipeline updates the images: transformer in a Kustomization Overlay for the dev environment with the new image by using a command like kustomize edit set image foobar=registry.example.com/foobar@sha256:deadbeef
4. A tool like ArgoCD picks up the new image and starts a new deployment.

This works well for the dev environment, but when that image needs to be promoted to a new environment like tst we either need to copy the updated image transformer manually, or try to read the updated version from the dev Kustomization.

And images are not the only thing. A typical Kustomize Overlay will include a versioned Base with something like this:

1apiVersion: kustomize.config.k8s.io/v1beta1
2kind: Kustomization
3resources:
4- git.example.com/foobar.git/deploy/base?ref=v0.2.1

In this example we see that the Base that is being used comes from the Git tag or branch called v0.2.1. If a newer version of the application needs a newer Base a developer can update that reference in the dev Overlay, but promotions will need to be handled as well.

Simply copying the kustomization.yaml file between Overlays will most likely not work, since different Overlays typically use different Replica transformers, different ConfigMap en Secret Generators etc.

Read on to learn how you can automate this.

A promotion Pipeline

When it is time to promote images and Bases to a new environment an application team can trigger a “Promotion Pipeline”. This pipeline can look something like this:

1. Developer triggers a pipeline to promote from dev to tst
2. Pipeline reads updated images and Bases from the dev Overlay and applies them to the tst Overlay.
3. Pipeline prepares a Merge Request on the deployment repository for the new Overlay
4. Another developer approves and merges the Merge Request
5. A tool like ArgoCD picks up on the changes and deploys the modified tst environment.

Step number two is where all the logic goes that fetches the relevant images and Bases, and applies them to the updated environment. Commonly this is done either manually, or by custom tooling. Sometimes even with some extra configuration somewhere that stores the different image and Base version for each environment for easy lookup, breaking the principal that your Kustomizations are the single source of truth.

Introducing kustomize-promote

To solve this issue we have created a tool called kustomize-promote. kustomize-promote will read the Kustomization from its first argument, and apply relevant image transformers and resource references from that Kustomization to the other Kustomization(s) specified.

For example, imagine a project with a (versioned) base, and two overlays in the directories overlays/dev and overlays/tst. These file currently look like this:

 1# overlays/dev
 2apiVersion: kustomize.config.k8s.io/v1beta1
 3kind: Kustomization
 4
 5resources:
 6- git.example.com/foobar.git/deploy/base?ref=v0.2.1
 7
 8replicas:
 9- name: foobar
10  count: 2
11
12images:
13- name: foobar
14  newName: registry.example.com/foobar
15  digest: sha256:1234

 1# overlays/tst
 2apiVersion: kustomize.config.k8s.io/v1beta1
 3kind: Kustomization
 4
 5resources:
 6- git.example.com/foobar.git/deploy/base?ref=v0.2.0
 7- extra/resource/for/tst
 8
 9replicas:
10- name: foobar
11  count: 3
12
13images:
14- name: foobar
15  newName: registry.example.com/foobar
16  digest: sha256:5678

As you can see the “replica” count for the foobar deployment is different between these two, as are the version of the Base being used and the digest of the foobar container image.

Now we can run this command:

1$ kustomize-promote overlays/dev overlays/tst

After the command completes our files now look like this:

 1# overlays/dev
 2apiVersion: kustomize.config.k8s.io/v1beta1
 3kind: Kustomization
 4
 5resources:
 6- git.example.com/foobar.git/deploy/base?ref=v0.2.1
 7
 8replicas:
 9- name: foobar
10  count: 2
11
12images:
13- name: foobar
14  newName: registry.example.com/foobar
15  digest: sha256:1234

 1# overlays/tst
 2apiVersion: kustomize.config.k8s.io/v1beta1
 3kind: Kustomization
 4
 5resources:
 6- git.example.com/foobar.git/deploy/base?ref=v0.2.1
 7- extra/resource/for/tst
 8
 9replicas:
10- name: foobar
11  count: 3
12
13images:
14- name: foobar
15  newName: registry.example.com/foobar
16  digest: sha256:1234

As you can see the replica counts have remained as they were before, but the tst Overlay now uses the same versions of bases and images as the dev Overlay.

But I Only Want to Promote Images/Overlays, not Overlays/Images

By default kustomize-promote promotes both images and resource references. If by any chance you don’t want both you can disable either of them with the --images=false and --references=false commandline options.

Download and Install

If you want to build kustomize-promote from source yourself you can do so by downloading the sources from the project GitLab page. A simple go build should see you started in no time.

If you rather download a pre-built binary those are available from the project Releases page. Binaries are available for Linux, Mac, and Windows, all both for x86_64 and ARM64.

Future Enhancements

Bug Reports, Feature Requests, Merge Requests, Comments, and more are all welcome. Visit the project page on GitLab to submit your Issues and Merge Requests.