3 Steps to a scalable build pipeline with Kubernetes, Jenkins and Kaniko
kubernetes | ci-cd |
Intro
By looking at some newer Kuberntes related articles, one might be tempted to think that everyone has long created their Kuberntes CI/CD pipelines, and is now busy building lots of Kubernetes operators or complaining that Kubernetes had become boring.
However, I don’t believe that’s the case for all, so I hope it will be useful to have an example of deploying Jenkins with the minimal configuration required to build and push container images to a registry.
The build is executed in a pod using a Kaniko container, which is one of the ways of building containers inside Kubernetes, while avoiding docker-in-docker approaches which are prone to security issues.
The pod is being created when there is a build requiring an agent with the kaniko
label and gets terminated when the build is complete.
There are great managed solutions which take care of all aspects of building container images, such as Azure Pipelines, so if this is a viable option for you/your company, it would make sense to use it and reduce the overhead.
If that’s not the case, or you’re interested in a solution which can also be deployed locally, keep reading.
What’s Kaniko?
Kaniko is a tool for building container images without using the Docker daemon.
One of the alternatives to Kaniko is buildah which also allows building from a Dockerfile, and my first impression is that it has more features than Kaniko (I realized that it supported builds using Dockerfile when I was almost done with this article - so maybe I’ll cover this in the future).
Prerequisites
- An existing Kubernetes cluster - These steps can be run locally by using microk8s / k3s / minikube, or a remote Kubernetes cluster (e.g. using AWS EKS, Azure AKS, etc).
- kubectl
- helm
Does the value file need further customizations?
Absolutely - this is just an example, not a definitive guide.
I only made minimal changes to the default value file of the Jenkins chart , however the values used for persistence, authentication and monitoring would need further tuning - more details in the Jenkins’ chart’s README.
The steps
The following steps will use the values.yaml
and config.json.template
files from this repository.
1 - Create the namespace
First - create a new namespace, it will make the configuration and subsequent cleanup easier.
kubectl create ns kaniko-demo
2 - Create the registry secret
This is required to push to a registry (Dockerhub in this case), have a look here for more info.
You’ll need to update the user
and token
placeholders before executing the following commands.
export BASE64_CREDENTIALS=$(echo -n "<user>:<token>"| base64) && \
curl -sL https://raw.githubusercontent.com/serbangilvitu/k8s-kaniko-jenkins/master/config.json.template \
| envsubst > config.json && \
kubectl -n kaniko-demo create secret generic kaniko-secret --from-file=config.json
3 - Install the Jenkins chart
Optionally, you can have a look at the rendered template using helm template
, which is a good idea before installing any chart.
The value file is customized to create a pod template with 2 containers: jnlp(Jenkins agent), and kaniko (see the podTemplates section - resource requests/limits can be tweaked).
If you want to make further changes, first download the file and update the --values
argument to point to the local values.yaml file.
To install the chart:
helm install demo stable/jenkins \
--namespace kaniko-demo \
--version 2.5.0 \
--values https://raw.githubusercontent.com/serbangilvitu/k8s-kaniko-jenkins/master/values.yaml \
--set namespaceOverride=kaniko-demo
Jenkins is now deploying, and will soon be accessible with a configuration which supports building container images using Kaniko.
Demo
Accessing Jenkins
Getting the admin password
printf $(kubectl get secret demo-jenkins \
--namespace kaniko-demo \
-o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
Port Forwarding
It will take a few minutes for Jenkins to initialize; to see when the Jenkins server becomes ready you can use
kubectl get pods -w -n kaniko-demo
The following command will allow you to access jenkins on http://localhost:8080
kubectl port-forward -n kaniko-demo svc/demo-jenkins 8080:8080
Optional - Pod Templates Review
The kaniko pod template should now be available. This template will be used to spin up the pods which will run the container image builds.
Sample pipelines
To spin up a kaniko pod, all that is required is to specify
agent { label 'kaniko'}
in your pipeline.
As there are 2 containers in the pod, when executing steps
container('kaniko')
is also required.
This can be seen in the examples below.
Building without pushing
The following command is used for building without pushing:
executor \
--no-push \
--context=git://github.com/serbangilvitu/sample-go-http-app.git#refs/heads/master
To test this out in your environment, create a Pipeline job pointing to the sample pipeline.
If you have some Dockerfile of your own, then you can update the context argument to point to it.
Building and pushing to a registry
I’ve replaced --no-push
with --destination
and in this example, the image will be tagged with the timestamp, then pushed to docker.io - see the other sample pipeline
Obviously, if you want to use this pipeline, you will have to provide your own destination repository.
executor \
--destination=docker.io/serbangilvitu/kaniko-test:$(date -u +%Y-%m-%dT%H%M%S) \
--context=git://github.com/serbangilvitu/sample-go-http-app.git#refs/heads/master
Cleaning Up
Delete the namespace
kubectl delete ns kaniko-demo