You can support us by downloading this article as PDF from the Link below. Download the guide as PDF

After initial setup of a Kubernetes cluster, the most common day 2 operation is scaling your cluster up by adding more nodes that runs your workloads – containers and pods. The way you scale your cluster depends on the tools that were used initially during cluster bootstrapping. This guide will demonstrate how you can add more Worker nodes to Kubernetes cluster using kubeadm command line tool.

See: Deploy Kubernetes Cluster on CentOS 7 / CentOS 8 With Kubeadm, Ansible and Calico CNI

Before you begin

  1. You need to have a working Kubernetes cluster – configured and working control plane node
  2. Container runtime (Docker,cri-o,containerd, e.t.c) and Kubernetes tools(kubeadm and kubelet) installed in your Worker node.
  3. If using Firewall such as firewalld, ports 10250, 30000-32767 and ports required by your Pod network add-on should be opened in the firewall.
  4. SSH access to the machine to be added
  5. Configured kubectl for checking if the node is available in your cluster

These standard requirements are covered in detail in our guide: Deploy Kubernetes Cluster With Ansible and Calico CNI

Here are the standard steps you’ll follow to join or add a new Worker Node to an existing Kubernetes Cluster.

Step 1: Get join Token

A token is required when joining a new worker node to the Kubernetes cluster. When you bootstrap a cluster with kubeadm, a token is generated which expires after 24 hours.

Check if you have a token – Run the command on Control node:

$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
nx1jjq.u42y27ip3bhmj8vj   21h         2020-01-10T20:33:08Z   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

If the token is expired, generate a new one with the command:

sudo kubeadm token create

The grab the token generated using:

$ kubeadm token list

You can also generate token and print the join command:

kubeadm token create --print-join-command

Step 2: Get Discovery Token CA cert Hash

For token-based discovery, the join command validates that the root CA public key by matching its hash with the one provided. Get the discovery token CA cert hash on the control node by running the command below.

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

Step 3: Get API Server Advertise address

Get your API server endpoint with kubectl command:

$ kubectl cluster-info
Kubernetes master is running at https://192.168.122.195:6443
KubeDNS is running at https://192.168.122.195:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.122.195:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

As shown in the output, my API is on https://192.168.122.195:6443.

Step 4: Join a new Kubernetes Worker Node a Cluster

The kubeadm join command is used to bootstrap a Kubernetes worker node or an additional control plane node, and join it to the cluster. The command syntax for joining a worker node to cluster is:

kubeadm join [api-server-endpoint] [flags]

The common flags required are:

  • --token string: Token to use
  • --discovery-token-ca-cert-hash: Has a format: <type>:<value>

So our complete join command will have the format:

kubeadm join \
  <control-plane-host>:<control-plane-port> \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Example:

sudo kubeadm join \
  192.168.122.195:6443 \
  --token nx1jjq.u42y27ip3bhmj8vj \
  --discovery-token-ca-cert-hash sha256:c6de85f6c862c0d58cc3d10fd199064ff25c4021b6e88475822d6163a25b4a6c

Command output:

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Wait for the node to have status “Ready” – Check on control node

$ watch kubectl get nodes

The process may take few minutes as container images are pulled before services are configured and brought up.

$ sudo docker ps
CONTAINER ID        IMAGE                   COMMAND                  CREATED              STATUS              PORTS               NAMES
491662a7affc        k8s.gcr.io/kube-proxy   "/usr/local/bin/kubeā€¦"   About a minute ago   Up About a minute                       k8s_kube-proxy_kube-proxy-zt8f9_kube-system_8f7f407c-d415-4ae0-8066-fbdf3dcb1570_0
743e7239b295        k8s.gcr.io/pause:3.1    "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_calico-node-rr666_kube-system_0c5a592c-08df-4303-a50d-0d93a2aec0ce_0
a69aad86ba4a        k8s.gcr.io/pause:3.1    "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_kube-proxy-zt8f9_kube-system_8f7f407c-d415-4ae0-8066-fbdf3dcb1570_0

Step 5: Removing a Worker Node from the Cluster

To remove a Kubernetes worker node from the cluster, perform the following operations.

  • Migrate pods from the node:
kubectl drain  <node-name> --delete-local-data --ignore-daemonsets
  • Prevent a node from scheduling new pods use – Mark node as unschedulable
kubectl cordon <node-name>
  • Revert changes made to the node by ‘kubeadm join‘ – Run on worker node to be removed
sudo kubeadm reset
  • You can then redo the same process of joining a new node to the cluster once the kubeadm reset command has been executed successfully.

Persistent Volume storage solutions for Kubernetes:

Ceph Persistent Storage for Kubernetes with Cephfs

Persistent Storage for Kubernetes with Ceph RBD

Also check:

How To Configure Kubernetes Dynamic Volume Provisioning With Heketi & GlusterFS

Create Kubernetes Service / User Account and restrict it to one Namespace with RBAC

Install and Use Helm 3 on Kubernetes Cluster

Install and Use Helm 2 on Kubernetes Cluster

You can support us by downloading this article as PDF from the Link below. Download the guide as PDF