How to Set up Scalable Jenkins on Top of a Kubernetes Cluster (At Home)
Before we start, I'll make some intro about the scenario and reasons for my choices, if you prefer, skip this part, the meat of the article starts at the Docker section.
A few days ago I decided to start a new side project, a tool that will take care of a bunch of things I need; I pay for different ones, and yet they don't cover everything I need. Since I have another excuse, and, it's for myself, I've decided to go fully buzzword-compliant, something hard to achieve at work.
One of the critical points is the building process of course, I could use Travis like I do some times but since the source code is private, and I have no intention to pay for Travis I've decided to spin up a Jenkins running on top of my beloved home devices, later it may change, but for now it'll be like that.
Just as a matter of fact, these are the devices I'm talking about.
Another thing that worth to mention, the version I'll describe here is the first version I got working, the simplest one, accessing the master Jenkins through
NodePort. The final version was a bit longer road and would make this article far longer it has two services, one for the
jnlp of type
ClusterIP and another one, the
http of type
The problem with the final version is that in my case "I'm not sure if all this was really necessary" I had to use Metallb to have my internal bare metal Load Balancer also because of the BGP configuration, I had to flash my Netgear and replace its original firmware by dd-wrt and also, a few other configurations here and there.
If was necessary or not, at least it was quite fun.
I'll assume that if you're here reading about Jenkins and Kubernetes is because you already have some knowledge about any of the container runtime environments supported by Kubernetes, since I know only about Docker, I'll stick with it.
I'll assume as well that you already have your Kubernetes cluster up and running, with all the nodes ready to go, running a
kubectl get nodes -o wide you should get enough information.
To start with, let's take a look at the Dockerfile, the idea here is to create a basic image ready to go with the initial necessary plugins. By ready to go I mean the latest version, interesting to know that the
latest Jenkins image at Docker Hub doesn't contain the
latest Jenkins version, if you tried to setup Jenkins before, you know what it means, many plugins will refuse work with an outdated Jenkins version.
Lines 3 to 5 (see below) takes care of the platform update so the plugins will be happy when Jenkins starts. Luckily
updates.jenkins-ci.org keeps a link to the
latest bundle, so we need to point there and problem solved.
To build this image you need to execute the following commands, please replace
<your user at docker hub>/<the name you prefer>.
The commands above will build and push the
latest version. Alternatively, you can set a specific version as well.
As mentioned before, for the sake of simplicity, I'll place here my initial working version, without load balancing capabilities, for this version you'll access your Jenkins instance at port
30000 of a fixed host, defined by a tag.
If you prefer, you can see an alternative version here. If what you need is more likely the alternative version, please read the first part of the article, before "Assumptions", because you may need some more steps if you're deploying it at home.
A bit of explanation about this file:
Lines: 16-17 - We're telling Jenkins to skip the initial wizard so we won't need to install any other plugin, set any password or get any hash from the logs right now when it starts, we'll go straight to business.
Lines: 26-28 - We're mounting a volume at the master instance to keep our configurations alive across different deployments.
Lines: 31-32 - These are the lines that define where Kubernetes will place your Jenkins master instance.
Since you have
nodeSelector, before you execute this file, you need to pick one of your nodes to host the Jenkins master, to do so, execute the lines below. Please replace
boss by the name of your node.
The first line adds the label; the second one will give you a description about the node, look for
Labels: you should see amongst the labels a line like
Assuming that you've saved the
yaml above in a file called
jenkins.yaml, is time to apply it; to do so, you must be at your master host.
This process sometimes turns to be a bit repetitive and tedious, if you prefer, use a shell script to make it a bit easier, here you'll find an example.
If you've executed this script you'll see an output similar to the one below, if not, copy and paste the first lines after the
echoes, and I'll manage to see what you have so far.
Notice that the service doesn't have any information about its public address, but we've specified a host where Jenkins should be deployed the same one you labelled before, with this in hand and the
nodePort we've specified we have enough information to access Jenkins. In my case http://boss:30000
Before jumping into the Kubernetes plugin configuration, let's take care of something essential, credentials. First, let's create a service account and fetch the secret.
Copy the whole content printed at the console by the third command and go to
Jenkins > Credentials > System > Global credentials > Add Credentials, change the
Kind drop-down options to
Secret text and past into
Secret, set an
ID to make it easy to pick it up later. There are many ways to handle credentials, but I've picked this approach because I think it would be easier to set up different environments later.
Now let's create our configuration at
Jenkins > Manage Jenkins > Configure System > Add a new cloud > Kubernetes
Now, we'll need a few pieces of information to set up our Jenkins cluster:
- Kubernetes URL: At the master node executes
kubectl cluster-info | grep masterand copy the URL you get there, for example,
- Credentials: Select the credentials we've created at the previous step.
- Jenkins tunnel: This information is critical to avoid falling into problems such as
jnlp is not enabled, you'll need to point to your service endpoint, do you remember the name of the service we've provided at the
jenkins.yaml? In this case it was
jenkins-master-svcif you changed, get back there and double-check, we'll need it to build the service FQDN
<service name>.<namespace>.svc.cluster.local:<port>, in this case,
Anything else you can copy from the screenshots below at this time, after that, play around with the configurations until you get what you need.
Ok, now let's create some dummy build projects, a
Freestyle project will do, at
Build > Execute shell add something silly to start with, for example
sleep 30, this job will take a while to finish, and we manage to see multiple instances working in parallel. Create two of three to see it working for real.
Back to the main screen start building all of them and once you see new entries at
Build Executor Status execute the following command to see their real status and the node where they are running at.
By the end, you should see the sunny Jenkins interface.
From this point you have your Jenkins cluster up and running, is up to you know to play with the many configuration options provided by the Kubernetes plugin and to set up your build projects.