How To Add Multiple Kubernetes Clusters to Jenkins

In this brief guide, we are going to look at how one can be able to connect to more than one Kubernetes Cluster in a single Jenkins server and deploy their applications to any of them. This might be a scenario where you have Test, Staging and Production environments and you would wish specific builds to go to specific environments.

Pre-requisites

In order for this guide to be beneficial, we assume you have a working Jenkins installation, as well as more than one Kubernetes clusters so that you can test as we proceed. In case you have neither, the guides shared below can be a quick reference to set them up before you can proceed:

Step 1: Install Kubernetes Plugin on Jenkins Server

In this step, we are going to look for a plugin that will allow us connect to our Kubernetes Clusters successfully so that we can be able to direct our deployments to either of them as we wish. To proceed, simply login to Jenkins, click on “Manage Jenkins” > “Manage Plugins” > Click on the “Available tab” then search for “Kubernetes“. You will see various plugins listed. Click on the checkboxes for “Kubernetes“, “Kubernetes Credentials“, and “Kubernetes CLI” plugins then click on install without reboot tab below. The screenshots for the above steps are shared below.

Manage Jenkins

Manage Plugins

Available Tab then Search Kubernetes

Let the installation progress as illustrated below then we can “Go back to the top page” once there is a “Success” on each installation item.

We are now ready to go ahead and try out the integration. Stay tuned.

Step 2: Generate Token in you Kubernetes Clusters

In this step, we are going to generate tokens which will be used by Jenkins to connect and deploy containers in the various clusters we have in our environment. This is the part where we will need the Kubernetes clusters to be up and running. If you are ready to proceed, login to your master node and generate a token as follows:

Create an admin account

In case you do not have an admin account already created, you can use the following YAML file to create yours. Note that “kube-admin” can be any name you prefer. This creates a Service Account (any name such as kube-admin here) then assigns it to a cluster role binding of cluster-admin.

$ nano admin.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: kube-admin
    namespace: kube-system

Once that is done, retrieve the token by running the following command. Remember to replace “kube-admin” with your respective user created above.

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep kube-admin | awk '{print $1}')

The command above should return a response similar to the one shared below. Note that the token in the example has been truncated and jumbled up.

Name:         kube-admin-token-cdFGt
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: kube-admin
              kubernetes.io/service-account.uid: a2f8f2e9-549a-4804-b7ff-4336e69c1797

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      cy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UlLWFjY291bnQubmFtZSI6Imt1YmUtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhMmY4ZjJlOS01NDlhLTQ4MDQtYjdiVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlLWFkbWluLXRva2VuLXhwN2s4Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWN

For a more detailed approach, look at How To Create Admin User to Access Kubernetes guide on this site.

Step 3: Add the Token as a credential in Jenkins

Once you have your token generated in the previous step, we now have the liberty to add it in our Jenkins server. We will add the token as a Credential as follows:

Login to Jenkins, click on “Manage Jenkins” > “Manage Credentials” > then click on “global” link as shared below.

Manage Credentials

Click global

This should take you to the “Global Credentials” page. On the left pane, you will see “Add Credential“. Kindly click on that.

It should open a form-like page where we can freely enter our details. Under “Kind“, scroll on the drop-down list and then choose “Secret text“. Under secret, copy the Kubernetes token we generated earlier and paste it there. Put a good descriptive name under “ID” and a description if you like. After you are done, click on “OK“.

Credential successfully added.

Thus far, we are only left with the Jenkinsfile code that will utilise this token to connect to our Kubernetes cluster. This is what we will do next. Please do the same for your other Kubernetes clusters you might be having.

Step 4: Use Pipeline Syntax to generate Kubernetes CLI configuration

In this step, we are going to use “Pipeline Syntax” tool that comes with Jenkins to generate a configuration we will use to connect to our Kubernetes cluster. First, let us create a new Pipeline, which will bring up the “Pipeline Syntax” option. Login to Jenkins, click on “New Item“.

Give the new item a good name then click on “Pipeline” then “OK“.

On the configuration page, click on “Pipeline“. You will see a link named “Pipeline Syntax“. Click on it and it should take you to a new tab.

Click Pipeline Syntax

Therein, under “Steps“, click on the drop-down menu and choose withKubeCredentials item as shared below:

It will immediately open a new data entry form where you will be asked for “Credentials“, “Kubernetes API endpoint“, “Cluster name”, “Context name“, “Namespace“, and “Certificate of certificate authority“. The credentials is the credential that we added in Step 3.

The other information can be found by running the following command in your Kubernetes Master inside or remotely if it has been configured so.

kubectl config view

You should see an output like below:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://192.168.10.10:6443        
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

For the credential, simply click on the drop down arrow and choose the credential you have already added for the individual Kubernetes Clusters as already shared. The rest can be found in the information provided by the command above. Note that “Certificate of certificate authority” and “Context name” are optional in case you do not use them in your environment. Scroll down and click on “Generate Pipeline Script“. You should see an output as shared below:

Same is shared below.

withKubeCredentials(kubectlCredentials: [[caCertificate: '', clusterName: 'kubernetes', contextName: '', credentialsId: 'TestKubernetes', namespace: 'kube-system', serverUrl: 'https://192.168.10.10:6443']]) {
    // some block
}

This can later be added as a step in your Jenkinsfile script as follows:

 steps {
     withKubeCredentials(kubectlCredentials: [[caCertificate: '', 
     clusterName: 'kubernetes', contextName: '', credentialsId: 'TestKubernetes', namespace: 'kube-system', 
     serverUrl: 'https://192.168.10.10:6443']]){               
     kubectl apply -f deployment.yaml          
         }
       }

That will deploy your file from a perhaps cloned git repository in the specified cluster listening at “https://192.168.10.10:6443“. Do the same for all of your Kubernetes clusters and you will have the freedom to deploy to whichever cluster you like with that simple block. Pretty cool right?

Concluding Remarks

We hope the guide was as informative as we intended it to be and in case there is a better way of doing the same, kindly let us know as we learn from each other. Before we close, receive our hearty gratitude for your kind support and your comments. We appreciate and we are thankful as you spend your time on the blog.

Books For Learning Kubernetes Administration:

• Best Kubernetes Study books

Other guides that might interest you include the following:

How To Install Jenkins Server on Kubernetes | OpenShift

How To Install Jenkins on Rocky Linux 8

How To Configure Jenkins FreeIPA LDAP Authentication