How to Set up Scalable Jenkins on Top of a Kubernetes Cluster (At Home)

Before we start, I'll make some intro about the scenario and reasons for my choices, if you prefer, skip this part, the meat of the article starts at the Docker section.

A few days ago I decided to start a new side project, a tool that will take care of a bunch of things I need; I pay for different ones, and yet they don't cover everything I need. Since I have another excuse, and, it's for myself, I've decided to go fully buzzword-compliant, something hard to achieve at work.

One of the critical points is the building process of course, I could use Travis like I do some times but since the source code is private, and I have no intention to pay for Travis I've decided to spin up a Jenkins running on top of my beloved home devices, later it may change, but for now it'll be like that.

Just as a matter of fact, these are the devices I'm talking about.

0-home-devices

Another thing that worth to mention, the version I'll describe here is the first version I got working, the simplest one, accessing the master Jenkins through NodePort. The final version was a bit longer road and would make this article far longer it has two services, one for the jnlp of type ClusterIP and another one, the http of type LoadBalancer.

The problem with the final version is that in my case "I'm not sure if all this was really necessary" I had to use Metallb to have my internal bare metal Load Balancer also because of the BGP configuration, I had to flash my Netgear and replace its original firmware by dd-wrt and also, a few other configurations here and there.

If was necessary or not, at least it was quite fun.

Assumptions

I'll assume that if you're here reading about Jenkins and Kubernetes is because you already have some knowledge about any of the container runtime environments supported by Kubernetes, since I know only about Docker, I'll stick with it.

I'll assume as well that you already have your Kubernetes cluster up and running, with all the nodes ready to go, running a kubectl get nodes -o wide you should get enough information.

Docker

To start with, let's take a look at the Dockerfile, the idea here is to create a basic image ready to go with the initial necessary plugins. By ready to go I mean the latest version, interesting to know that the latest Jenkins image at Docker Hub doesn't contain the latest Jenkins version, if you tried to setup Jenkins before, you know what it means, many plugins will refuse work with an outdated Jenkins version.

Lines 3 to 5 (see below) takes care of the platform update so the plugins will be happy when Jenkins starts. Luckily updates.jenkins-ci.org keeps a link to the latest bundle, so we need to point there and problem solved.

To build this image you need to execute the following commands, please replace allandequeiroz/jenkins by <your user at docker hub>/<the name you prefer>.

The commands above will build and push the latest version. Alternatively, you can set a specific version as well.

Kubernetes

As mentioned before, for the sake of simplicity, I'll place here my initial working version, without load balancing capabilities, for this version you'll access your Jenkins instance at port 30000 of a fixed host, defined by a tag.

If you prefer, you can see an alternative version here. If what you need is more likely the alternative version, please read the first part of the article, before "Assumptions", because you may need some more steps if you're deploying it at home.

A bit of explanation about this file:

Lines: 16-17 - We're telling Jenkins to skip the initial wizard so we won't need to install any other plugin, set any password or get any hash from the logs right now when it starts, we'll go straight to business.
Lines: 26-28 - We're mounting a volume at the master instance to keep our configurations alive across different deployments.
Lines: 31-32 - These are the lines that define where Kubernetes will place your Jenkins master instance.

Since you have nodeSelector, before you execute this file, you need to pick one of your nodes to host the Jenkins master, to do so, execute the lines below. Please replace boss by the name of your node.

The first line adds the label; the second one will give you a description about the node, look for Labels: you should see amongst the labels a line like jenkins=master.

Assuming that you've saved the yaml above in a file called jenkins.yaml, is time to apply it; to do so, you must be at your master host.

This process sometimes turns to be a bit repetitive and tedious, if you prefer, use a shell script to make it a bit easier, here you'll find an example.

If you've executed this script you'll see an output similar to the one below, if not, copy and paste the first lines after the echoes, and I'll manage to see what you have so far.

Notice that the service doesn't have any information about its public address, but we've specified a host where Jenkins should be deployed the same one you labelled before, with this in hand and the nodePort we've specified we have enough information to access Jenkins. In my case http://boss:30000

Jenkins

Before jumping into the Kubernetes plugin configuration, let's take care of something essential, credentials. First, let's create a service account and fetch the secret.

Copy the whole content printed at the console by the third command and go to Jenkins > Credentials > System > Global credentials > Add Credentials, change the Kind drop-down options to Secret text and past into Secret, set an ID to make it easy to pick it up later. There are many ways to handle credentials, but I've picked this approach because I think it would be easier to set up different environments later.

Now let's create our configuration at Jenkins > Manage Jenkins > Configure System > Add a new cloud > Kubernetes

Now, we'll need a few pieces of information to set up our Jenkins cluster:

  • Kubernetes URL: At the master node executes kubectl cluster-info | grep master and copy the URL you get there, for example, https://192.168.1.101:6443.
  • Credentials: Select the credentials we've created at the previous step.
  • Jenkins tunnel: This information is critical to avoid falling into problems such as connection refused or jnlp is not enabled, you'll need to point to your service endpoint, do you remember the name of the service we've provided at the jenkins.yaml? In this case it was jenkins-master-svc if you changed, get back there and double-check, we'll need it to build the service FQDN <service name>.<namespace>.svc.cluster.local:<port>, in this case, jenkins-master-svc.default.svc.cluster.local:50000.

Anything else you can copy from the screenshots below at this time, after that, play around with the configurations until you get what you need.

1-cloud-kubernetes

2-kubernetes-pod-template

Ok, now let's create some dummy build projects, a Freestyle project will do, at Build > Execute shell add something silly to start with, for example sleep 30, this job will take a while to finish, and we manage to see multiple instances working in parallel. Create two of three to see it working for real.

3-build

Back to the main screen start building all of them and once you see new entries at Build Executor Status execute the following command to see their real status and the node where they are running at.

By the end, you should see the sunny Jenkins interface.

4-build-results

From this point you have your Jenkins cluster up and running, is up to you know to play with the many configuration options provided by the Kubernetes plugin and to set up your build projects.