Monitoring stack setup — Part 2: Alert Manager & Push Gateway

In the last article, we went over the setup of Prometheus & Grafana. In this article, we will cover the setup, testing and access to alertmanager & pushgateway.

Alertmanager:

1. Add a alertmanager user:

sudo useradd --no-create-home alertmanager

2. Download and install the Prometheus binary:

wget https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.23.0.linux-amd64.tar.gz

3. Create folders for use by alertmanager

sudo mkdir -p /var/lib/alertmanager
sudo mkdir -p /etc/alertmanager

3. Copy files for alertmanager setup:

sudo cp alertmanager-0.23.0.linux-amd64/alertmanager /usr/local/bin/
sudo cp alertmanager-0.23.0.linux-amd64/alertmanager.yml  /etc/alertmanager

4. Assign ownership of files to alertmanager user

sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager 
sudo chown -R alertmanager:alertmanager /etc/alertmanager

5. Create a service file

sudo vim /etc/systemd/system/alertmanager.service

Contents -

[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager
--config.file /etc/alertmanager/alertmanager.yml
--storage.path /var/lib/alertmanager/
[Install]
WantedBy=multi-user.target

6. Run the following to add the above service unit and start alertmanager.

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

The installation is complete. Let’s test alertmanager now.

1. Check if the process is running or failing.

sudo systemctl status alertmanager

2. Alertmanager usually occupies port 9093. Check response.

curl http://localhost:9093

Alertmanager access

Let’s open port 9093 to access the UI @
http://<public-ip-address>:9093/

Alertmanager is complete. Let’s add pushgateway now.

1. Add a pushgateway user

sudo useradd --no-create-home pushgateway

2. Download and install the Prometheus binary:

wget https://github.com/prometheus/pushgateway/releases/download/v1.4.2/pushgateway-1.4.2.linux-amd64.tar.gz
tar xvfz pushgateway-1.4.2.linux-amd64.tar.gz

3. Copy files for pushgateway setup:

sudo cp pushgateway-1.4.2.linux-amd64/pushgateway /usr/local/bin/

4. Assign ownership of files to pushgateway user:

sudo chown pushgateway:pushgateway /usr/local/bin/pushgateway

5. Create a service file

sudo vi /etc/systemd/system/pushgateway.service

Contents:

[Unit]
Description=Prometheus Pushgateway
Wants=network-online.target
After=network-online.target
[Service]
User=pushgateway
Group=pushgateway
Type=simple
ExecStart=/usr/local/bin/pushgateway
[Install]
WantedBy=multi-user.target

6. Run the following to add the above service unit and start pushgateway.

sudo systemctl daemon-reload
sudo systemctl enable pushgateway
sudo systemctl start pushgateway
sudo systemctl status pushgateway

The installation is complete. Let’s test pushgateway now.

1. Check if the process is running or failing.

sudo systemctl pushgateway status

2. Pushgateway usually occupies port 9091. Check response.

curl http://localhost:9091

Pushgateway access

Let’s open port 9091 to access the UI @
http://<public-ip-address>:9091/

Push Gateway setup is complete.

Connecting Alertmanager and Prometheus

In order for prometheus to send alerts to the alertmanager, it must be given the address of the alertmanager.

It’s quite simple to do this — add the following snippet to the prometheus configuration file (/etc/prometheus/prometheus.yml)

global:
  scrape_interval: 15s 
  evaluation_interval: 15s 
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093 #alertmanager address 
rule_files:
  - rules1.yml

Here, the rules1.yml is a file containing the defination of multiple alarms.

That’s it! We have discussed the complete setup of a monitoring stack. Do share your feedback!

I share my learnings & viewpoints on cloud & containers regularly. Follow me on Linkedin & Medium for more!