Monitoring stack setup — Part 1: Prometheus & Grafana

Monitoring is critical to every team. Why? It delivers information about the infrastructure’s state, performance and usage pattern. It is the best answer to the growing need of detailed real-time insights to a product’s development.

All the tools in cloud and containers automation are helping movement of code into production as soon as possible. But it’s risky to trust just these tool in the hope of everything going well.

That is where monitoring systems can help!

The most popular solutions for it is Prometheus-Grafana stack.

Prometheus-Grafana Stack

This stack is what most teams use. It comprises of 4 major components.

1. Prometheus. It gets the metrics data from application & stores it.

2. Grafana. Takes data from prometheus to display on multiple visualizations.

3. Alert Manager. Gets alerts from prometheus & sends notifications.

4. Push gateway. Adds support for metrics cannot be scraped.

In this article, we will go through the steps to deploy Prometheus & Grafana.

In the next one, we will cover Alert Manager & Push Gateway.

Prometheus installation

1. Add prometheus user:

sudo useradd --no-create-home prometheus

2. Download and install the Prometheus binary:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.30.3/prometheus-2.30.3.linux-amd64.tar.gz
tar -xvf prometheus-2.30.3.linux-amd64.tar.gz

3. Copy files from prometheus setup:

sudo cp prometheus-2.30.3.linux-amd64/prometheus /usr/local/bin
sudo cp prometheus-2.30.3.linux-amd64/promtool /usr/local/bin
sudo cp -r prometheus-2.30.3.linux-amd64/consoles /etc/prometheus/
sudo cp -r prometheus-2.30.3.linux-amd64/console_libraries /etc/prometheus
sudo cp prometheus-2.30.3.linux-amd64/promtool /usr/local/bin/

4. Adding content to prometheus’s configuration file:

• This is where we configure the time intervals in which prometheus will scrap the metrics.
• Here, we are telling prometheus to scrape the metrics it is generating. So that we can use these metrics later on.

global:
  scrape_interval: 15s
  external_labels:
    monitor: 'prometheus'
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

5. Give user ‘prometheus’ the permission to the file used to run prometheus server.

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /var/lib/prometheus/

6. Add the prometheus startup in the service script.

• So that we can start, stop, restart & check its status easily.

sudo vim /etc/systemd/system/prometheus.service

A service unit describes a service or application on the machine will be managed. It includes instructions for starting or stopping the service and many other things.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

7. Run the following to add the above service unit and start prometheus.

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

The installation is complete. Let’s test prometheus out.

Testing Prometheus

1. Check if the process is running or failing.

sudo systemctl status prometheus

2. Prometheus usually occupies port 9090. Check response.

curl http://localhost:9090/graph

Prometheus access

Let’s open port 9090 to access the UI @
http://<public-ip-address>:9090/graph

Prometheus is complete. Let’s add grafana now.

Grafana installation

1. Add grafana’s required user:

sudo apt-get install -y adduser libfontconfig1

2. Download & install Grafana binary:

wget https://dl.grafana.com/oss/release/grafana_8.2.2_amd64.deb
sudo dpkg -i grafana_8.2.2_amd64.deb

3. Run the following to add the above service unit and start grafana.

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
sudo systemctl enable grafana-server.service

The installation is complete. Let’s test Grafana.

Grafana testing

1. Check grafana process status.

sudo systemctl status grafana-server

Grafana access

Let’s open port 3000 to access the UI @
http://<public-ip-address>:3000/login

The default user & password is ‘admin’ & ‘admin’.

Note: The public ip address has been changed because I had stopped the EC2 for some time. You need not have a different ip address.

Connecting Prometheus & Grafana together

The flow looks like this -

1. Prometheus collects metrics data.
2. Grafana asks Prometheus for the metrics data to display.

So, Grafana needs the address of prometheus.

We just need to specify prometheus address at
Grafana->Configuration -> Add Source.

Specify the address for prometheus as “http://localhost:9090/”.
Save and test!

That’s all.

We have a a Prometheus and Grafana running and accessible on Internet.

In the next article, we will work on Alert Manager & Push Gateway.

Share your feedback and follow for more.

Want to get in touch? Head to my Linkedin!