Monitoring is the next imperative duty after setting up any service used in Production. Having proper monitoring for your Etcd cluster ensures you can respond to issues quickly and know when it’s time to scale. There are a plethora of tools that can be used to monitor an etcd cluster. This guide will discuss the use of Prometheus and Grafana Dashboards, with optional alerting that can be configured.
Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging. Test by running:
$ curl -L http://localhost:2379/metrics
Before you proceed, I expect you to have a ready Etcd cluster on a Linux server. We have a detailed guide on how to setup an Etcd cluster on Linux -CentOS/Ubuntu e.t.c. With a running cluster, you can proceed to setup Prometheus and Grafana to be used in monitoring.
Step 1: Install Grafana
You need Grafana Data visualization & Monitoring tool installed on a Linux system. We have a number of guides you can refer to.
Step 2: Install Prometheus
Prometheus and Grafana can co-exist on same server. For the monitoring to work, the Prometheus server should be able to access Etcd nodes on the service ports.
Check our recent guide which covers the installation of Prometheus on CentOS & Ubuntu Linux systems.
Step 3: Configure Prometheus
We need to set Prometheus’s scraper to target the etcd cluster endpoints:
sudo vim /etc/prometheus/prometheus.yml
Add like below.
- job_name: 'etcd' static_configs: - targets: ['192.168.18.9:2379','192.168.18.10:2379','192.168.18.11:2379'']
Replace the IP addresses specified with your Etcd cluster node’s IP addresses. Restart Prometheus service after adding the configuration lines to the file.
sudo systemctl restart prometheus
The targets should show on Prometheus dashboard Status > Targets section.
Step 4: Add default etcd dashboard
Add data source to Grafana.
Configuration > Data Sources > Add data source > Prometheus
Select Prometheus data source added.
Customize the dashboards to display your desired metrics on grafana.
And you have Etcd cluster on Grafana.