Disclaimer

3 Steps to a scalable build pipeline with Kubernetes, Jenkins and Kaniko

kubernetes ci-cd

Intro

By looking at some newer Kuberntes related articles, one might be tempted to think that everyone has long created their Kuberntes CI/CD pipelines, and is now busy building lots of Kubernetes operators or complaining that Kubernetes had become boring.

However, I don’t believe that’s the case for all, so I hope it will be useful to have an example of deploying Jenkins with the minimal configuration required to build and push container images to a registry.

The build is executed in a pod using a Kaniko container, which is one of the ways of building containers inside Kubernetes, while avoiding docker-in-docker approaches which are prone to security issues.

The pod is being created when there is a build requiring an agent with the kaniko label and gets terminated when the build is complete.

There are great managed solutions which take care of all aspects of building container images, such as Azure Pipelines, so if this is a viable option for you/your company, it would make sense to use it and reduce the overhead.

If that’s not the case, or you’re interested in a solution which can also be deployed locally, keep reading.

What’s Kaniko?

Kaniko is a tool for building container images without using the Docker daemon.

One of the alternatives to Kaniko is buildah which also allows building from a Dockerfile, and my first impression is that it has more features than Kaniko (I realized that it supported builds using Dockerfile when I was almost done with this article - so maybe I’ll cover this in the future).

Prerequisites

Does the value file need further customizations?

Absolutely - this is just an example, not a definitive guide.

I only made minimal changes to the default value file of the Jenkins chart , however the values used for persistence, authentication and monitoring would need further tuning - more details in the Jenkins’ chart’s README.

The steps

The following steps will use the values.yaml and config.json.template files from this repository.

1 - Create the namespace

First - create a new namespace, it will make the configuration and subsequent cleanup easier.

kubectl create ns kaniko-demo

2 - Create the registry secret

This is required to push to a registry (Dockerhub in this case), have a look here for more info.

You’ll need to update the user and token placeholders before executing the following commands.

export BASE64_CREDENTIALS=$(echo -n "<user>:<token>"| base64) && \
  curl -sL https://raw.githubusercontent.com/serbangilvitu/k8s-kaniko-jenkins/master/config.json.template \
  | envsubst > config.json && \
  kubectl -n kaniko-demo create secret generic kaniko-secret --from-file=config.json

3 - Install the Jenkins chart

Optionally, you can have a look at the rendered template using helm template, which is a good idea before installing any chart.

The value file is customized to create a pod template with 2 containers: jnlp(Jenkins agent), and kaniko (see the podTemplates section - resource requests/limits can be tweaked).

If you want to make further changes, first download the file and update the --values argument to point to the local values.yaml file.

To install the chart:

helm install demo stable/jenkins \
  --namespace kaniko-demo \
  --version 2.5.0 \
  --values https://raw.githubusercontent.com/serbangilvitu/k8s-kaniko-jenkins/master/values.yaml \
  --set namespaceOverride=kaniko-demo

Jenkins is now deploying, and will soon be accessible with a configuration which supports building container images using Kaniko.

Demo

Accessing Jenkins

Getting the admin password

printf $(kubectl get secret demo-jenkins \
  --namespace kaniko-demo \
  -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

Port Forwarding

It will take a few minutes for Jenkins to initialize; to see when the Jenkins server becomes ready you can use

kubectl get pods -w -n kaniko-demo

The following command will allow you to access jenkins on http://localhost:8080

kubectl port-forward -n kaniko-demo svc/demo-jenkins 8080:8080

Optional - Pod Templates Review

The kaniko pod template should now be available. This template will be used to spin up the pods which will run the container image builds.

Sample pipelines

To spin up a kaniko pod, all that is required is to specify

agent { label 'kaniko'}

in your pipeline.

As there are 2 containers in the pod, when executing steps

container('kaniko')

is also required.

This can be seen in the examples below.

Building without pushing

The following command is used for building without pushing:

executor \
  --no-push \
  --context=git://github.com/serbangilvitu/sample-go-http-app.git#refs/heads/master

To test this out in your environment, create a Pipeline job pointing to the sample pipeline.

If you have some Dockerfile of your own, then you can update the context argument to point to it.

Building and pushing to a registry

I’ve replaced --no-push with --destination and in this example, the image will be tagged with the timestamp, then pushed to docker.io - see the other sample pipeline

Obviously, if you want to use this pipeline, you will have to provide your own destination repository.

executor \
  --destination=docker.io/serbangilvitu/kaniko-test:$(date -u +%Y-%m-%dT%H%M%S) \
  --context=git://github.com/serbangilvitu/sample-go-http-app.git#refs/heads/master

Cleaning Up

Delete the namespace

kubectl delete ns kaniko-demo