(Last Updated On: October 2, 2018)

This is the continuation of our guides on Smart Infrastructure monitoring with Grafana, InfluxDB, Prometheus, and Telegraf. We have already covered how to Install Grafana and InfluxDB on CentOS 7. As part of server preparation, we’ll look at how to Install Prometheus Server on CentOS 7.

For the record, all servers in my Lab will be running CentOS 7, but other servers to be monitored will be running Ubuntu, Fedora, Debian, Arch and even Windows servers 2012/2016 e.t.c. So for this, I assume you have a server (VM) or LXC container running CentOS 7.x.

Create Prometheus system group

Let’s start by creating the Prometheus system group. You need to use the option  -r or –system with groupadd command.

sudo groupadd --system prometheus

The group with ID < 1000 is a system group.

Create Prometheus system user

Let’s now create Prometheus system user and assign primary group we just created above.

sudo useradd -s /sbin/nologin --system -g prometheus prometheus

This is a system user which doesn’t need /bin/bash shell, that’s why we used -s /sbin/nologin.

Create data directory for Prometheus

Prometheus needs a directory to store its data. We will create one under /var/lib/prometheus. We will specify this data directory when starting the service.

sudo mkdir /var/lib/prometheus

Create configuration directories for Prometheus

Prometheus primary configuration files directory is /etc/prometheus/. It will have some sub-directories.

for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done

Download Prometheus Archive and extract it

We need to download the latest release of Prometheus archive and extract it to get binary files. You can check releases from Prometheus releases Github page.

sudo yum -y install wget
cd /tmp
export RELEASE=2.4.2
wget https://github.com/prometheus/prometheus/releases/download/v${RELEASE}/prometheus-${RELEASE}.linux-amd64.tar.gz

Extract the file.

tar xvf prometheus-${RELEASE}.linux-amd64.tar.gz
cd prometheus-${RELEASE}.linux-amd64/

Copy Prometheus binary files to /usr/local/bin/

Since /usr/local/bin/ is in your $PATH, let’s copy binary files to it.

sudo cp prometheus promtool /usr/local/bin/

Copy consoles and console_libraries to /etc/prometheus directory:

sudo cp -r consoles/ console_libraries/ /etc/prometheus/
ls /etc/prometheus/
console_libraries consoles rules rules.d files_sd

Create a Prometheus configuration file.

Prometheus configuration file will be located under /etc/prometheus/prometheus.yml. Create simple configurations using content:

# Global config
global: 
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.  
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.  
  scrape_timeout: 15s  # scrape_timeout is set to the global default (10s).

# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

You can edit the file to your default liking and save it.

Create a Prometheus systemd Service unit file

To be able to manage Prometheus service with systemd, you need to explicitly define this unit file.

# cat /etc/systemd/system/prometheus.service 

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
Environment="GOMAXPROCS=2"
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Remember to edit the line:

Environment="GOMAXPROCS=2

Replace 2 with the number of vcpus on your server.

Clean Installation

rm -rf prometheus-${RELEASE}.linux-amd64.tar.gz
rm -rf prometheus-${RELEASE}.linux-amd64/

Change directory permissions.

Change the ownership of these directories to Prometheus user and group.

for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/

Reload systemd daemon and start the service.

sudo systemctl daemon-reload
sudo systemctl start prometheus

Check status using systemctl status prometheus command:

Configure firewalld

Open a port on the firewall, for me, I’ll use firewalld-rich rules to allow access from only my trusted subnets and IPs.

sudo firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" \
source address="192.168.10.0/24" port protocol="tcp" port="9090" accept'

sudo firewall-cmd --reload

I’m going to test from my Grafana server if I can access Prometheus service on port 9090.

$ sudo yum -y install nmap-ncat
$ sudo nc -v 62.12.112.233 9090
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 62.12.112.233:9090.

Since I can connect to port 9090, this means our setup is complete. Next, we will cover installing exporters on nodes to be monitored and configuring targets on out Prometheus server so that we can scrap metrics and visualize with Grafana.

Prometheus Monitoring guides

Monitoring Ceph Cluster with Prometheus and Grafana

Monitoring Ceph Cluster with Prometheus and Grafana

How to Monitor Linux Server Performance with Prometheus and Grafana in 5 minutes

How to Monitor BIND DNS server with Prometheus and Grafana

How to Monitor Redis Server with Prometheus and Grafana in 5 minutes

Monitoring MySQL / MariaDB with Prometheus in five minutes