You want to deploy a Lightweight Kubernetes Cluster with ease less memory footprint?. all in a binary less than 40mb. Kubernetes has been a game changer in how containerized workloads are deployed and managed at immense scale. The main challenge for Developers revolve around setup process and resource requirements to have a working Kubernetes Cluster.

For Production deployment refer to: Deploy Production Ready Kubernetes Cluster with Ansible & Kubespray

For Development and test impetus, a user should be able to deploy Kubernetes with least resource utilization and low hardware specs. K3s is a tiny Kubernetes distribution created and optimized by Rancher team to run on IoT and ARM powered devices.

Since K3s is optimized to use less resources, some Kubernetes features are stripped off. These include:

  • Legacy, alpha, non-default features
  • Most in-tree plugins (cloud providers and storage plugins) which can be replaced with out of tree addons.
  • etcd3 replaced by sqlite3 as the default storage mechanism

Deploy Lightweight Kubernetes Cluster in 5 minutes with K3s

For my deployment, I have three servers running Debian 10 each with 1GB of Ram and 1vcpu. One of the servers will be used as master and other two as worker nodes.

$ openstack server list
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+
| ID                                   | Name              | Status  | Networks                          | Image     | Flavor    |
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+
| 4df6a6dc-26e8-4ae0-8b6e-2f97daec0ef3 | k3s-master        | ACTIVE  | private=10.10.1.159               | Debian-10 | m1.tiny   |
| 5ca13239-b745-4f62-ab11-0a27949c9b35 | k3s-node02        | ACTIVE  | private=10.10.1.142               | Debian-10 | m1.tiny   |
| a54997f2-4d94-4718-86ab-73609b328761 | k3s-node01        | ACTIVE  | private=10.10.1.126               | Debian-10 | m1.tiny   |
+--------------------------------------+-------------------+---------+-----------------------------------+-----------+-----------+

I’ll add A records for the servers on /etc/hosts file on each server.

sudo tee -a /etc/hosts<<EOF
10.10.1.159 k3s-master
10.10.1.126 k3s-node01
10.10.1.142 k3s-node02
EOF

Install K3s on Master node

There are many ways to run k3s. The quickest method is installation via provided bash script. This script provides a convenient way for installing to systemd or openrc.

curl -sfL https://get.k3s.io | sh -

Installation output.

[INFO]  Finding latest release
[INFO]  Using v0.8.1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v0.8.1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v0.8.1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

The service is automatically started after installation.

$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-09-17 19:20:00 UTC; 2min 24s ago
     Docs: https://k3s.io
  Process: 833 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 836 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 837 (k3s-server)
    Tasks: 98
   Memory: 571.1M
   CGroup: /system.slice/k3s.service
           ├─ 837 /usr/local/bin/k3s server KillMode=process
           ├─ 851 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/conta
           ├─1110 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/f6eeb59978
           ├─1127 /pause
           ├─1207 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/0baf0ca181
           ├─1225 /coredns -conf /etc/coredns/Corefile
           ├─1576 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/dcce4b7e17
           ├─1594 /pause
           ├─1599 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/50816ffba8
           ├─1617 /pause
           ├─1824 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/d0ff393609
           ├─1842 /bin/sh /usr/bin/entry
           ├─1882 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/046779175f
           ├─1899 /bin/sh /usr/bin/entry
           ├─1904 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/93f0fe2361
           └─1921 /traefik --configfile=/config/traefik.toml

Sep 17 19:20:34 deb10 k3s[837]: E0917 19:20:34.714229     837 daemon_controller.go:302] kube-system/svclb-traefik failed with : error storing statu
Sep 17 19:20:34 deb10 k3s[837]: E0917 19:20:34.719452     837 daemon_controller.go:302] kube-system/svclb-traefik failed with : error storing statu
Sep 17 19:20:34 deb10 k3s[837]: I0917 19:20:34.726816     837 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volum
Sep 17 19:20:34 deb10 k3s[837]: I0917 19:20:34.726836     837 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volum
Sep 17 19:20:34 deb10 k3s[837]: I0917 19:20:34.726857     837 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volum
Sep 17 19:20:34 deb10 k3s[837]: I0917 19:20:34.726869     837 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volum
Sep 17 19:20:35 deb10 k3s[837]: I0917 19:20:35.529102     837 reconciler.go:181] operationExecutor.UnmountVolume started for volume "helm-traefik-t
Sep 17 19:20:35 deb10 k3s[837]: I0917 19:20:35.542858     837 operation_generator.go:799] UnmountVolume.TearDown succeeded for volume "kubernetes.i
Sep 17 19:20:35 deb10 k3s[837]: I0917 19:20:35.629277     837 reconciler.go:285] Volume detached for volume "helm-traefik-token-kjwrl" (UniqueName:
Sep 17 19:20:36 deb10 k3s[837]: W0917 19:20:36.355273     837 pod_container_deletor.go:75] Container "2f0c4a787b13c029d65aa865c1b473f5a7497cb6f9b92

A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml:

$ cat /etc/rancher/k3s/k3s.yaml
cat: /etc/rancher/k3s/k3s.yaml: Permission denied
[email protected]:~$ sudo cat /etc/rancher/k3s/k3s.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJWakNCL3FBREFnRUNBZ0VBTUFvR0NDcUdTTTQ5QkFNQ01DTXhJVEFmQmdOVkJBTU1HR3N6Y3kxelpYSjIKWlhJdFkyRkFNVFUyT0RjME56azVOakFlRncweE9UQTVNVGN4T1RFNU5UWmFGdzB5T1RBNU1UUXhPVEU1TlRaYQpNQ014SVRBZkJnTlZCQU1NR0dzemN5MXpaWEoyWlhJdFkyRkFNVFUyT0RjME56azVOakJaTUJNR0J5cUdTTTQ5CkFnRUdDQ3FHU000OUF3RUhBMElBQkM5aTMyUTdkVnhJaTFCVFNEOTRqYzJaZy9ESHFGc051b0Q4eWhSbjZsUlIKQWp5Q0p3UEZYQ3Y4QUdSMmFaK1lSempTYUJvM2M1LzMwQnZwKzY3OFNYeWpJekFoTUE0R0ExVWREd0VCL3dRRQpBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUFvR0NDcUdTTTQ5QkFNQ0EwY0FNRVFDSUJwTXdOejAyZzUwCkExdEloU0Y1MFJqSVprVVVuNk8rODdLV25obWRUYkh5QWlBQnJqcDFxWU1HcWE0RmJ2Ym9rTm1kM3VOelVvQm8KeGxqTGlnWnZCN3ZEVGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://localhost:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    password: 2d99cae31c075743be704bb717ceaae8
    username: admin

Other utilities installed are:

  • kubectl
  • crictl
  • k3s-killall.sh
  • k3s-uninstall.sh

Install K3s on Worker Nodes

To install on worker nodes we should pass K3S_URL along with K3S_TOKEN or K3S_CLUSTER_SECRET environment variables.

The K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on first node.

$ sudo cat /var/lib/rancher/k3s/server/node-token
K1042e2f8e353b9409472c1e0cca8457abe184dc7be3f0805109e92c50c193ceb42::node:c83acbf89a7de7026d6f6928dc270028

So to install Kubernetes on worker nodes I’ll run:

k3s_url="https://k3s-master:6443"
k3s_token="K1042e2f8e353b9409472c1e0cca8457abe184dc7be3f0805109e92c50c193ceb42::node:c83acbf89a7de7026d6f6928dc270028"
curl -sfL https://get.k3s.io | K3S_URL=${k3s_url} K3S_TOKEN=${k3s_token} sh -

Installation output:

[INFO]  Finding latest release
[INFO]  Using v0.8.1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v0.8.1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v0.8.1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent

Login to one of the master node and check cluster status.

$ sudo kubectl config get-clusters 
NAME
default

$ sudo kubectl cluster-info 
Kubernetes master is running at https://localhost:6443
CoreDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

$ sudo kubectl get  nodes
NAME         STATUS   ROLES    AGE     VERSION
k3s-master   Ready    master   14m     v1.14.6-k3s.1
k3s-node01   Ready    worker   3m11s   v1.14.6-k3s.1
k3s-node02   Ready    worker   3m58s   v1.14.6-k3s.1

$ sudo kubectl get namespaces
NAME              STATUS   AGE
default           Active   16m
kube-node-lease   Active   16m
kube-public       Active   16m
kube-system       Active   16m

$ sudo  kubectl get endpoints -n kube-system
NAME       ENDPOINTS                                  AGE
kube-dns   10.42.0.2:53,10.42.0.2:53,10.42.0.2:9153   14m
traefik    10.42.0.5:80,10.42.0.5:443                 14m

$ sudo kubectl get pods -n kube-system
NAME                         READY   STATUS      RESTARTS   AGE
coredns-b7464766c-q9frk      1/1     Running     0          15m
helm-install-traefik-8dhpk   0/1     Completed   0          15m
svclb-traefik-9c2j8          2/2     Running     0          4m49s
svclb-traefik-bf9zd          2/2     Running     0          4m2s
svclb-traefik-v2fpx          2/2     Running     0          14m
traefik-5c79b789c5-k589d     1/1     Running     0          14m

Use crictl command to see running containers.

# Master
$ sudo crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
acfafb50852d3       18471c10e6e4b       16 minutes ago      Running             traefik             0                   bf8534452389f
fee5ac7e88f2e       4a065d8dfa588       16 minutes ago      Running             lb-port-443         0                   e7068ff7ab2f2
bbab5b07e5efb       4a065d8dfa588       16 minutes ago      Running             lb-port-80          0                   e7068ff7ab2f2
65c5d1333ea04       2ee68ed074c6e       16 minutes ago      Running             coredns             0                   435c51f4716fc

# Workers
$ sudo crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
7ad5c83d6466f       4a065d8dfa588       6 minutes ago       Running             lb-port-443         0                   bf8d9fe57c3f3
c1380eabc0b33       4a065d8dfa588       6 minutes ago       Running             lb-port-80          0                   bf8d9fe57c3f3

Check K3s documentation for advanced configurations.

Other deployment options for Kubernetes available on our site are:

Deploy Production Ready Kubernetes Cluster with Ansible & Kubespray

How To run Local Kubernetes clusters in Docker

Deploy Lightweight Kubernetes with MicroK8s and Snap