How to setup 3 node Kubernetes Cluster on Ubuntu 18.04 with Weave Net CNI

In this guide, I’ll take you through the steps to install and set up a working 3 node Kubernetes Cluster on Ubuntu 18.04 Bionic Beaver Linux.  Kubernetes is an open-source container-orchestration system used for automating deployment, management, and scaling of containerized applications.

Kubernetes on Ubuntu 18.04 – System Diagram

This setup is based on the following diagram:

Let’s configure system hostnames before we can proceed to next steps:

On Master Node:

Set hostname like below:

$ sudo hostnamectl set-hostname k8s-master

On Worker Node 01:

Set the hostname using hostamectl command line tool.

$ sudo hostnamectl set-hostname k8s-node-01

On Worker Node 02:

Also set hostname for Kubernetes worker node 02.

$ sudo hostnamectl set-hostname k8s-node-02

Once correct hostname has been configured on each host, populate on each node with the values configured.

$ cat /etc/hosts
192.168.2.2 k8s-master 
192.168.2.3 k8s-node-01 
192.168.2.4 k8s-node-02

Setup Kubernetes on Ubuntu 18.04 – Prerequisites (Run on all nodes)

Before doing any Kubernetes specific configurations, let’s ensure all deps are satisfied. Here we will do a system update and create Kubernetes user.

Update system packages to the latest release on all nodes:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install linux-image-extra-virtual
sudo reboot

Add user to manage Kubernetes cluster:

sudo useradd -s /bin/bash -m k8s-admin
sudo passwd k8s-admin
sudo usermod -aG sudo k8s-admin
echo "k8s-admin ALL=(ALL) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/k8s-admin

If you prefer entering sudo password when running sudo commands as k8s-admin user, then you can ignore the last line. You can test if no password prompt for sudo:

$ su - k8s-admin
[email protected]:~$ sudo su -
[email protected]:~#

All looks good, let’s proceed to install Docker engine.

Setup Kubernetes on Ubuntu 18.04 – Install Docker Engine

Kubernetes requires docker to run containers used for hosting applications and other Kubernetes services. We have a comprehensive Docker installation guide:

How to install Docker CE on Ubuntu / Debian / Fedora / Arch / CentOS

If you need a quick installation guide, then use the following commands to install Docker Engine on Ubuntu 18.04. Ensure any old version of Docker engine is uninstalled on your system:

sudo apt-get remove docker docker-engine docker.i

Install dependencies:

$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common

Import Docker repository GPG key:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$ sudo add-apt-repository \
 "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
 $(lsb_release -cs) \
 stable"

Install docker:

sudo apt-get update
sudo apt-get install docker-ce
sudo usermod -aG docker k8s-admin

When docker has been installed, you can continue to configure the Kubernetes master node.

Setup Kubernetes on Ubuntu 18.04 – Install and Configure Kubernetes Master

All commands that will be executed on this section are meant to be run on the master node. Don’t execute any of the commands on Kubernetes worker nodes. Kubernetes Master components provide the cluster’s control plane – API Server, Scheduler, Controller Manager. They make global decisions about the cluster e.g scheduling and detecting and responding to cluster events.

Add Kubernetes repository

As of this writing, there is no official repository for Ubuntu 18.04, we will add a repository for Ubuntu 16.04. I tested it. All packages and dependencies should install fine. I’ll update this article when a repo for Ubuntu 18.04 is available.

# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF

Then import GPG key:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Update apt package index:

sudo apt update

Install Kubernetes Master Components

Install kubectl, kubelet, kubernetes-cni and kubeadm Kubernetes master components:

sudo apt install kubectl kubelet kubeadm kubernetes-cni

Confirm that all package binaries are present on the file system.

$ which kubelet
/usr/bin/kubelet
$ which kubeadm
/usr/bin/kubeadm

If swap is on, turn it off.

sudo swapoff -a

Initialize Kubernetes Cluster:

When all Kubernetes packages have been installed, you’re ready to initialize the cluster using kubeadm command line tool.

Export required variables (Optional)

export API_ADDR=`ifconfig eth0 | grep 'inet'| cut -d':' -f2 | awk '{print $1}'`
export DNS_DOMAIN="k8s.local"
export POD_NET="10.4.0.0/16"
export SRV_NET="10.5.0.0/16"

Then initialize the Kubernetes cluster using variables defined above:

kubeadm init --pod-network-cidr ${POD_NET} --service-cidr ${SRV_NET} \
--service-dns-domain "${DNS_DOMAIN}" --apiserver-advertise-address ${API_ADDR}

If all goes well, you should get a success message with the instructions of what to do next:

---
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.2.2:6443 --token 9y4vc8.h7jdjle1xdovrd0z --discovery-token-ca-cert-hash sha256:cff9d1444a56b24b4a8839ff3330ab7177065c90753ef3e4e614566695db273c

Configure Access for k8s-admin user on the Master server

Switch to k8s-adminand copy Kubernetes configuration file with cluster information.

su - k8s-admin
mkdir -p $HOME/.k8s
sudo cp -i /etc/kubernetes/admin.conf $HOME/.k8s/config
sudo chown $(id -u):$(id -g) $HOME/.k8s/config
export KUBECONFIG=$HOME/.k8s/config
echo "export KUBECONFIG=$HOME/.k8s/config" | tee -a ~/.bashrc

Deploy Weave Net POD Network to the Cluster ( Run as normal user)

Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery. Services provided by application containers on the Weave network can be exposed to the outside world, regardless of where they are running.

Weave Net can be installed onto your CNI-enabled Kubernetes cluster with a single command:

# su - k8s-admin
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created

After a few seconds, a Weave Net pod should be running on each Node and any further pods you create will be automatically attached to the Weave network.

[email protected]:~$ kubectl get pod -n kube-system | grep weav
weave-net-d9v5v                      2/2       Running   0          11h
weave-net-mhp46                      2/2       Running   0          11h
weave-net-vmksr                      2/2       Running   0          11h

Setup Kubernetes Worker Nodes

When Kubernetes cluster has been initialized and the master node is online, start Worker Nodes configuration. A node is a worker machine in Kubernetes, it may be a VM or physical machine.  Each node is managed by the master and has the services necessary to run pods – docker, kubelet, kube-proxy

Step 1: Ensure Docker is installed (covered)

Ensure docker engine is installed on all Worker nodes. Refer to docker installation section

Step 2: Add Kubernetes repository ( covered)

Ensure that repository for Kubenetes packages is added to the system. Refer ^^

Step 3: Install Kubenetes components

Once you’ve added Kubernetes repository, install components using:

sudo apt install kubelet kubeadm kubectl kubernetes-cni

Step 4: Join the Node to the Cluster:

Use the join command given after initializing Kubernetes cluster. E.g

kubeadm join 192.168.2.2:6443 --token 9y4vc8.h7jdjle1xdovrd0z \
 --discovery-token-ca-cert-hash sha256:cff9d1444a56b24b4a8839ff3330ab7177065c90753ef3e4e614566695db273c

---
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node-02" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

When done,  Check nodes status on the master:

[email protected]:~$ kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
k8s-master    Ready     master    35m       v1.11.0
k8s-node-01   Ready     <none>    2m        v1.11.0
k8s-node-02   Ready     <none>    1m        v1.11.0

On the two nodes, Weave Net should be configured.

[email protected]:~# ip ad | grep weave
6: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
    inet 10.44.0.0/12 brd 10.47.255.255 scope global weave
9: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default

[email protected]:~# ip ad | grep weave
6: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
    inet 10.47.0.0/12 brd 10.47.255.255 scope global weave
9: [email protected]: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP group default

Test Kubernetes Deployment

Let us create test pod to confirm that our cluster is running as expected.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: http-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: http-app
    spec:
      containers:
      - name: http-app
        image: katacoda/docker-http-server:latest
        ports:
        - containerPort: 80

Deploy it to the cluster

Create a test namespace:

$ kubectl create namespace test-namespace
namespace/test-namespace created

After namespace is created, create a pod using deployment object defined earlier. -n is used to specify the namespace. We expect three pods to be created since our replicas value is 3.

$ kubectl create -n test-namespace -f http-app-deployment.yml
deployment.extensions/http-app created

Confirm:

$ kubectl -n test-namespace get deployments
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
http-app   3         3         3            3           1m

$ kubectl -n test-namespace get pods
NAME                       READY     STATUS    RESTARTS   AGE
http-app-97f76fcd8-68pxg   1/1       Running   0          1m
http-app-97f76fcd8-f9bdk   1/1       Running   0          1m
http-app-97f76fcd8-vgmq7   1/1       Running   0          1m

You can see we have http-app deployment live.

With the deployment created, we can use kubectl to create a service which exposes the Pods on a particular port. An alternative method is defining a Service object with YAML. Below is our service definition.

$ cat http-app-service.yml 
apiVersion: v1
kind: Service
metadata:
  name: http-app-svc
  labels:
    app: http-app
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30080
  selector:
    app: http-app

Create a service using kubectl command:

$ kubectl -n test-namespace create -f http-app-service.yml 
service/http-app-svc created

This service will be available on Cluster IP and port 30080. To get cluster IP, use:

$ kubectl -n test-namespace get svc
NAME           TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
http-app-svc   NodePort   10.5.45.208   <none>        80:30080/TCP   1m

Kubernetes on Ubuntu 18.04 – Post-Installation

Enable shell autocompletion for kubectl commands.  kubectl includes autocompletion support, which can save a lot of typing!. To enable shell completion in your current session, run:

source <(kubectl completion bash)

To add kubectl autocompletion to your profile, so it is automatically loaded in future shells run:

echo "source <(kubectl completion bash)" >> ~/.bashrc

If you are using zsh edit the ~/.zshrc file and add the following code to enable kubectl autocompletion:

if [ $commands[kubectl] ]; then
source <(kubectl completion zsh)
fi

Or when using Oh-My-Zsh, edit the ~/.zshrc file and update the plugins= line to include the kubectl plugin.

source <(kubectl completion zsh)

Also relevant is Top command for container metrics

Conclusion

We have successfully deployed a 3 node Kubernetes cluster on Ubuntu 18.04 LTS servers. Our next guides will cover Kubernetes HA, Kubernetes Monitoring, How to configure external storage and more cool stuff. Stay tuned!.