Home Databases How To Install 3 Node Etcd HA Cluster on Ubuntu 22.04

How To Install 3 Node Etcd HA Cluster on Ubuntu 22.04

December 5, 2023

1563

The concept of containerization is now everywhere. It can be defined as the packaging of packages and all the required dependencies in a lightweight executable called a container. This concept has been highly adopted because it eliminates all the dependencies issues associated with packages. It also provides improved security by isolating workloads, and improved app performance with scalability, portability and flexibility. Containerization has brought several tools such as Docker, Podman, Containerd, Kubernetes, ETCD etc.

etcd is popular among all people who have interacted with Kubernetes. It is a consistent and highly-available key-value store that serves as a backbone of distributed systems for backing all the cluster data. It serves as a primary storage for storing and replicating all the Kubernetes cluster states. This tool was initially built to run clusters on Fedora CoreOS then later adopted on OSX, Linux and BSD systems. Today, many distributed systems such as Google Kubernetes, Cloud Foundry and Fleet, largely depend on etcd.

In a Kubernetes cluster, etcd acts as the brain. It is responsible for holding all the sensitive information required to run the cluster. Whenever etcd runs into an issue, the whole cluster may face issues. To achieve high availability, it is recommended to isolate the ectd cluster. This improves maintenance and manageability.

Below is a diagram illustrating the architecture of the Etcd Cluster

In this guide, we will walk through how to install 3 Node Etcd HA Cluster on Ubuntu 22.04.

Getting Started

To run the ETCD cluster smoothly, you need an odd number of serves, say 3 or 5. You also need to ensure that there is no resource starvation to ensure the cluster is stable.

In this guide, we will use an environment of 4 Ubuntu 22.04 servers configured as shown below:

TASK	IP ADDRESS	HOSTNAME
ETCD1	192.168.200.52	etcd1.computingforgeeks.com
ETCD1	192.168.200.62	etcd2.computingforgeeks.com
ETCD1	192.168.200.63	etcd3computingforgeeks.com
LOAD-BALANCER	192.168.200.56	loadbalancer.computingforgeeks.com

Ensure the systems are updated and all the required packages are installed:

sudo apt update
sudo apt install vim curl -y

Then proceed to set the hostnames on all the servers. For example on;

sudo hostnamectl set-hostname etcd1.computingforgeeks.com #Node1
sudo hostnamectl set-hostname etcd2.computingforgeeks.com #Node2
sudo hostnamectl set-hostname etcd3.computingforgeeks.com #Node3

Then configure /etc/hosts on all the machines as shown:

$ sudo vim /etc/hosts
192.168.200.52 etcd1.computingforgeeks.com  etcd1
192.168.200.62 etcd2.computingforgeeks.com  etcd2
192.168.200.63 etcd3.computingforgeeks.com  etcd3
192.168.200.56 loadbalancer.computingforgeeks.com  loadbalancer

There are many other requirements for Production deployments. One of them is that the minimum etcd version required are 3.4.22+ and 3.5.6+. There are many more requirements, see the resource requirement reference

Once all is set up, you can proceed as shown in the below steps.

1. Install Required Packages

There are some packages required on all the etcd nodes before we proceed to initialize the cluster.

For this method, users are able to pick the latest available versions of etcd. To achieve that, visit the etcd releases page. Once here, export the latest available version:

ETCD_VER=v3.5.10

Export other variables required:

GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

Create the /tmp directory for the achieve:

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

Now pull the binary with the commands:

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

Extract the archive:

tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

Move the binaries to your PATH:

sudo cp /tmp/etcd-download-test/etcd* /usr/local/bin

Now create a service file:

sudo vim /etc/systemd/system/etcd.service

Add these lines to the file:

[Unit]
Description=etcd

[Service]
Type=notify
EnvironmentFile=/etc/default/etcd
ExecStart=/usr/local/bin/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Now verify the installation:

$ etcd --version
etcd Version: 3.5.10
Git SHA: 0223ca52b
Go Version: go1.20.10
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.10
API version: 3.5

Once etcd has been installed on the 3 servers, proceed as shown below.

2. Set up Etcd HA Cluster on Ubuntu

Once the required packages have been installed on the etcd nodes, we can now create the cluster. To achieve that, we need to make a few adjustments to the etcd configuration.

Open the file for editing:

sudo vim /etc/default/etcd

This file contains all the environment variables for the cluster. We need to define all the variables on all the 3 nodes:

ETCD_NAME=<node-name>
##ETCD_DATA_DIR=/var/lib/etcd
ETCD_INITIAL_CLUSTER=<node1-name=http://<node1-ip>:2380,node2-name=http://<node2-ip>:2380,node3-name=http://<node3-ip>:2380>
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://<node-ip>:2380
ETCD_ADVERTISE_CLIENT_URLS=http://<node-ip>:2379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379

Replace all the variables on each node first before starting them. Remember that at least 2 etcd nodes must be started within 50 seconds or else it will throw an error because of the master election timeout.

For example, on my etcd1 node, the variables will be:

ETCD_NAME=etcd1
##ETCD_DATA_DIR=/var/lib/etcd
ETCD_INITIAL_CLUSTER=etcd1=http://192.168.200.52:2380,etcd2=http://192.168.200.62:2380,etcd3=http://192.168.200.63:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.200.52:2380
ETCD_ADVERTISE_CLIENT_URLS=http://192.168.200.52:2379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379

Once the changes have been made on all the 3 nodes, start or restart the etcd service with the commands:

sudo systemctl daemon-reload
sudo systemctl enable --now etcd

##To Restart, Use:
sudo systemctl restart etcd

Verify if the service is running on all the nodes:

$ systemctl status etcd
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-11-19 16:21:21 EAT; 29s ago
   Main PID: 4684 (etcd)
      Tasks: 7 (limit: 4617)
     Memory: 12.7M
        CPU: 455ms
     CGroup: /system.slice/etcd.service
             └─4684 /usr/local/bin/etcd

Nov 19 16:21:43 etcd1.computingforgeeks.com etcd[4684]: {"level":"info","ts":"2023-11-19T16:21:43.013666+0300","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"3c>
Nov 19 16:21:43 etcd1.computingforgeeks.com etcd[4684]: {"level":"info","ts":"2023-11-19T16:21:43.014271+0300","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection >
Nov 19 16:21:43 etcd1.computingforgeeks.com etcd[4684]: {"level":"info","ts":"2023-11-1
....

Now at this point, the cluster is up and running. To verify that, use the below command and replace the IP with that of any etcd servers.

$ etcdctl --endpoints=http://192.168.200.52:2379 member list
12aca8afa5a60d1a, started, etcd1, http://192.168.200.52:2380, http://192.168.200.52:2379, false
3ccdae3b547878c4, started, etcd3, http://192.168.200.63:2380, http://192.168.200.63:2379, false
cdf1b4107740a046, started, etcd2, http://192.168.200.62:2380, http://192.168.200.62:2379, false

From the above output, we have 3 nodes in the etcd cluster.

3. Install and Configure LoadBalancer

First, install the load balancer. In this guide, we will use Haproxy as our load balancer. You can still use Nginx or any other as desired.

To install Haproxy on your Ubuntu system, use:

##On Debian/Ubuntu
sudo apt update && sudo apt install haproxy

Once installed, start and enable the service:

sudo systemctl start haproxy
sudo systemctl enable haproxy

Proceed and create the proxy config:

sudo cp /etc/haproxy/haproxy.cfg{,.old}
sudo vim /etc/haproxy/haproxy.cfg

Bear in mind that the HAProxy config has 4 main parts. These are:

Global section with parameters to all servers running.
Default settings with details that apply to all proxy subsections
Frontend settings define the listening sockets from the client connection
Backend settings defines the server IP address and the load balancer algorithm

In the file, we need to add the below lines defining how it should handle loadbalancing:

frontend main
    bind *:2379
    mode tcp
    default_backend etcd-api

backend etcd-api
    mode tcp
    balance roundrobin
    server node1  etcd1.computingforgeeks.com:2379 check
    server node2  etcd2.computingforgeeks.com:2379 check
    server node3  etcd3.computingforgeeks.com:2379 check

Provide the domain names correctly in the above file. If you have a firewall enabled, allow this port through it on all the nodes:

sudo ufw allow 2379

Now restart the HAProxy service:

sudo systemctl restart haproxy

You can now access the cluster using the loadbalancer IP. For example:

Now you can use the etcd endpoint in your Kubernetes cluster. You need specify it in the Kubernetes config as shown:

$ sudo vim /etc/kubernetes/apiserver
KUBE_ETCD_SERVERS=” — etcd_servers=http://etcd1:2379,http://etcd2:2379, http://etcd3:2379"
##OR
KUBE_ETCD_SERVERS=” — etcd_servers=http://<loadbalancer-IP>:2379

Test if you can write to the cluster:

##Member list in a table
# etcdctl --write-out=table --endpoints=<loadbalancer_IP>:2379 member list
+------------------+---------+--------+------------------------+------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS       |      CLIENT ADDRS      |
+------------------+---------+--------+------------------------+------------------------+
| 8211f1d0f64f3269 | started | etcd1 | http://192.168.200.52:2380  | http://192.168.200.52:2379  |
| 91bc3c398fb3c146 | started | etcd2 | http://192.168.200.62:22380 | http://192.168.200.62:22379 |
| fd422379fda50e48 | started | etcd3 | http://192.168.200.63:32380 | http://192.168.200.63:32379 |
+------------------+---------+--------+------------------------+------------------------+

##Store a Key
$ etcdctl put key hello
OK

##Retrieve the Key
$ etcdctl get key
hello

4. Secure etcd with TLS certificates

It is possible to generate and use certificates for your ETCD cluster. This ensures a secure communication between the nodes. To demonstrate that, I will generate self-signed certs for ETCD.

We can use Mkcert for that. First, install it by pulling the latest binary from the GitHub releases. We will export the version:

VER=v1.4.4

Now download the binaries with the command:

##AMD64 
wget https://github.com/FiloSottile/mkcert/releases/download/$VER/mkcert-$VER-linux-amd64

##ARM64
wget https://github.com/FiloSottile/mkcert/releases/download/$VER/mkcert-$VER-linux-arm64

##ARM
wget https://github.com/FiloSottile/mkcert/releases/download/$VER/mkcert-$VER-linux-arm

Make it executable and move it to your PATH

chmod +x mkcert-$VER-linux-* 
sudo mv mkcert-$VER-linux-* /usr/local/bin/mkcert

Now generate certs with the command:

mkcert -install
mkcert '*.computingforgeeks.com'

You will then have the key and cert generated in your locations. The CA is available in the $(mkcert -CAROOT)/rootCA.pem. You can copy and use the CA on your client to avoid SSL certificate warnings

Copy these certs on all the etcd nodes. You can use SCP for that.

Once the certs are moved to the required directory say /etc/ssl/certs/etcd. You can configure etcd to use them by adding these lines to the config:

Sample on Node 2:

$ sudo vim /etc/default/etcd
ETCD_NAME=etcd2
##ETCD_DATA_DIR=/var/lib/etcd
ETCD_INITIAL_CLUSTER=etcd1=http://192.168.200.52:2380,etcd2=http://192.168.200.62:2380,etcd3=http://192.168.200.63:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.200.62:2380
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.200.62:2379
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379

##[Security]
ETCD_CERT_FILE=/etc/ssl/certs/etcd/_wildcard.computingforgeeks.com.pem
ETCD_KEY_FILE=/etc/ssl/certs/etcd/_wildcard.computingforgeeks.com-key.pem
#ETCD_CLIENT_CERT_AUTH="false"
ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/etcd/rootCA.pem

Once the changes have been made on all the 3 servers, restart the etcd service:

sudo systemctl restart etcd

You can add the CA cert to the list of trusted certs on your system as shown:

sudo cp /etc/ssl/certs/etcd/rootCA.pem /usr/local/share/ca-certificates/rootCA.crt
sudo update-ca-certificates

If you are not able or you don’t want to add the CA to the list of trusted certs, you can access the cluster by specifying it as shown:

$ etcdctl --endpoints=https://192.168.200.56:2379 member list --cacert /path/to/rootCA.crt
12aca8afa5a60d1a, started, etcd1, https://192.168.200.52:2380, https://192.168.200.52:2379, false
3ccdae3b547878c4, started, etcd3, https://192.168.200.63:2380, https://192.168.200.63:2379, false
cdf1b4107740a046, started, etcd2, https://192.168.200.62:2380, https://192.168.200.62:2379, false

That shows that the communication within the nodes is secure.

Verdict

That is the end of this detailed guide on how to install 3 Node Etcd HA Cluster on Ubuntu 22.04. You can now use the etcd in your Kubernetes cluster. I hope this was informative.

See more:

Getting Started

1. Install Required Packages

2. Set up Etcd HA Cluster on Ubuntu

3. Install and Configure LoadBalancer

4. Secure etcd with TLS certificates

Verdict

LEAVE A REPLY Cancel reply

Recent Posts

More Content

Books

EDITOR PICKS

POPULAR POSTS

POPULAR CATEGORY

Books