You can support us by downloading this article as PDF from the Link below. Download the guide as PDF

You want to setup a three node Kubernetes Cluster on CentOS 7 / CentOS 8 for your Development Projects – 1 Master and Two or more Worker nodes?. This guide will walk you through the steps to setup a Kubernetes cluster on CentOS 8 / CentOS 7 Linux machines with Ansible and Calico CNI with Firewalld running and configured. Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

Similar Kubernetes deployment guides:

My Lab is based on the following environment:

Machine TypeHostnameIP Address
Control Nodek8smaster01.computingforgeeks.com192.168.122.10
Worker Node 1k8snode01.computingforgeeks.com192.168.122.11
Worker Node 2k8snode01.computingforgeeks.com192.168.122.12

We’ll start by making sure the systems are updated, and install all dependencies including container runtime, Kubernetes software packages and configuring firewall for k8s.

Step 1: Setup Standard requirements

I wrote an Ansible role for doing the standard Kubernetes node preparation. The role contain the tasks to:

  • Install basic packages required
  • Setup standard system requirements – Disable Swap, Modify sysctl, Disable SELinux
  • Install and configure a container runtime of your Choice – cri-o, Docker, Containerd
  • Install the Kubernetes packages – kubelet, kubeadm and kubectl
  • Configure Firewalld on Kubernetes Master and Worker nodes

Visit my Github page to go through setup:

https://github.com/jmutai/k8s-pre-bootstrap

This is an output of my recent execution:

TASK [kubernetes-bootstrap : Open flannel ports on the firewall] ***************************************************************************************
skipping: [k8smaster01] => (item=8285) 
skipping: [k8smaster01] => (item=8472) 
skipping: [k8snode01] => (item=8285) 
skipping: [k8snode01] => (item=8472) 
skipping: [k8snode02] => (item=8285) 
skipping: [k8snode02] => (item=8472) 

TASK [kubernetes-bootstrap : Open calico UDP ports on the firewall] ************************************************************************************
ok: [k8snode01] => (item=4789)
ok: [k8smaster01] => (item=4789)
ok: [k8snode02] => (item=4789)

TASK [kubernetes-bootstrap : Open calico TCP ports on the firewall] ************************************************************************************
ok: [k8snode02] => (item=5473)
ok: [k8snode01] => (item=5473)
ok: [k8smaster01] => (item=5473)
ok: [k8snode01] => (item=179)
ok: [k8snode02] => (item=179)
ok: [k8smaster01] => (item=179)
ok: [k8snode02] => (item=5473)
ok: [k8snode01] => (item=5473)
ok: [k8smaster01] => (item=5473)

TASK [kubernetes-bootstrap : Reload firewalld] *********************************************************************************************************
changed: [k8smaster01]
changed: [k8snode01]
changed: [k8snode02]

PLAY RECAP *********************************************************************************************************************************************
k8smaster01                : ok=23   changed=3    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0   
k8snode01                  : ok=23   changed=3    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0   
k8snode02                  : ok=23   changed=3    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0   

Step 2: Initialize single node control-plane

This Deployment is for a single control-plane node with integrated etcd. If you want to do multiple control nodes (3 for HA), checkout Creating Highly Available clusters with kubeadm official guide.

We’ll use the kubeadm to bootstrap a minimum viable Kubernetes cluster that conforms to best practices. The good thing with kubeadm is that it supports other cluster lifecycle functions, such as upgrades, downgrade, and managing bootstrap tokens.

Single Control node bootstrap requirements:

  • Default IP address of the control node machine
  • DNS name / Load Balancer IP if you plan to add more control nodes later
  • SSH access as root user or user with sudo

Login to control node:

$ ssh [email protected]

Check the arguments that can be used for initializing your Kubernetes cluster:

$ kubeadm init --help

The standard arguments that we’ll use are:

  • --pod-network-cidr : Used to specify range of IP addresses for the pod network.
  • --apiserver-advertise-address: The IP address the API Server will advertise it’s listening on.
  • --control-plane-endpoint: Specify a stable IP address or DNS name for the control plane.
  • --upload-certs: Upload control-plane certificates to the kubeadm-certs Secret.
  • If using Calico, the recommended Pod Network is: 192.168.0.0/16
  • For Flannel, it is recommended you set Pod network to 10.244.0.0/16

For me, I’ll run the command:

sudo kubeadm init \
  --apiserver-advertise-address=192.168.122.10 \
  --pod-network-cidr 192.168.0.0/16 \
  --upload-certs

To be able to upgrade this single control-plane kubeadm cluster to high availability you should specify the –-control-plane-endpoint to set the shared endpoint for all control-plane nodes. Such an endpoint can be either a DNS name or an IP address of a load-balancer.

kubeadm init \
  --apiserver-advertise-address=192.168.122.227 \
  --pod-network-cidr 192.168.0.0/16 \
  --control-plane-endpoint <DNS-End-Point>
  --upload-certs

Here is my installation output:

W0109 20:27:51.787966   18069 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0109 20:27:51.788126   18069 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
	[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8smaster01.computingforgeeks.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.122.10]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8smaster01.computingforgeeks.com localhost] and IPs [192.168.122.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8smaster01.computingforgeeks.com localhost] and IPs [192.168.122.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0109 20:32:51.776569   18069 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0109 20:32:51.777334   18069 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 15.507327 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
bce5c1ad320f4c64e42688e25526615d2ffd7efad3e749bc0c632b3a7834752d
[mark-control-plane] Marking the node k8smaster01.computingforgeeks.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8smaster01.computingforgeeks.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: nx1jjq.u42y27ip3bhmj8vj
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.122.10:6443 --token nx1jjq.u42y27ip3bhmj8vj \
    --discovery-token-ca-cert-hash sha256:c6de85f6c862c0d58cc3d10fd199064ff25c4021b6e88475822d6163a25b4a6c

Copy the kubectl configuration file.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Also checkout: Easily Manage Multiple Kubernetes Clusters with kubectl & kubectx

Deploy a pod network to the cluster

I’ll use Calico but you’re free to use any other pod network add-on of your choice.

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml

This creates a number of resources as seen in the following output.

configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

Confirm that all of the pods are running with the following command.

watch kubectl get pods --all-namespaces

Once everything is running as expected, the output will be like below.

NAMESPACE     NAME                                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5c45f5bd9f-c8mwx                    1/1     Running   0          3m45s
kube-system   calico-node-m5qmb                                           1/1     Running   0          3m45s
kube-system   coredns-6955765f44-cz65r                                    1/1     Running   0          9m43s
kube-system   coredns-6955765f44-mtch2                                    1/1     Running   0          9m43s
kube-system   etcd-k8smaster01.computingforgeeks.com                      1/1     Running   0          9m59s
kube-system   kube-apiserver-k8smaster01.computingforgeeks.com            1/1     Running   0          9m59s
kube-system   kube-controller-manager-k8smaster01.computingforgeeks.com   1/1     Running   0          9m59s
kube-system   kube-proxy-bw494                                            1/1     Running   0          9m43s
kube-system   kube-scheduler-k8smaster01.computingforgeeks.com            1/1     Running   0          9m59s

Notice each pod has the STATUS of Running.

Check Calico Documentation for more details.

Step 3: Joining your Worker Nodes to the Cluster

Now that you have the control node ready, you can add new nodes where your workloads (containers and pods, etc) will run. You need to do this on each machine that should be used to run Pods.

  • SSH to the machine
$ ssh [email protected]
  • Run the command that was output by kubeadm init. For example:
sudo kubeadm join 192.168.122.10:6443 --token nx1jjq.u42y27ip3bhmj8vj \
    --discovery-token-ca-cert-hash sha256:c6de85f6c862c0d58cc3d10fd199064ff25c4021b6e88475822d6163a25b4a6c

If the token expired, you can generate a new one with the command:

kubeadm token create

The get the token

kubeadm token list

You can get the value of discovery-token-ca-cert-hash with the command:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

Here is a join command output:


[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Run the same join command on all other Worker nodes, then see the available nodes joined to the cluster with the command:

$ kubectl get nodes
NAME                                STATUS   ROLES    AGE     VERSION
k8smaster01.computingforgeeks.com   Ready    master   26m     v1.17.0
k8snode01.computingforgeeks.com     Ready    <none>   4m35s   v1.17.0
k8snode02.computingforgeeks.com     Ready    <none>   2m4s    v1.17.0

Step 4: Deploy Metrics Server to Kubernetes Cluster

Metrics Server is a cluster-wide aggregator of resource usage data. It collects metrics from the Summary API, exposed by Kubelet on each node. Use our guide below to deploy it:

How To Deploy Metrics Server to Kubernetes Cluster

Step 5: Persistent Storage ideas

If you’re also looking for a Persistent storage solution for your Kubernetes, checkout:

Ceph Persistent Storage for Kubernetes with Cephfs

Persistent Storage for Kubernetes with Ceph RBD

How To Configure Kubernetes Dynamic Volume Provisioning With Heketi & GlusterFS

Check below training courses to learn more:

$15.36
$165.37
in stock
Udemy.com
$14.18
$159.46
in stock
Udemy.com
$15.36
$165.37
in stock
Udemy.com

Conclusion

There you have it, you now have a running Kubernetes Cluster you can Develop Cloud native applications with. We have other guides on Kubernetes such as:

How To Manually Pull Container images used by Kubernetes kubeadm

Install and Use Helm 3 on Kubernetes Cluster

Install and Use Helm 2 on Kubernetes Cluster

Create Kubernetes Service / User Account and restrict it to one Namespace with RBAC

You can support us by downloading this article as PDF from the Link below. Download the guide as PDF