Monitoring your Linux Servers with Prometheus and Grafana in 7 Minutes

September 13, 2018

no comments

Objective


At the end of this post we will get a Grafana dashboard with all the metrics related to our servers.

 

Introduction


Prometheus is a tool for store time series data and manage alerts. It works as a pull based system, the Prometheus server fetches the metrics values from the target servers periodically. To expose the metrics in each server we will use “node_export” which basically expose the server metrics using the port 9100 (by default). And Grafana is a tool for query, visualize and understand your metrics. We will use it to create our dashboard.

 

Understand your Infrastructure


The first thing we need to do is understand our infrastructure. That means know the following about each server:

IP
Server Name
Ports in Use
Type (Dev, Staging, Production, etc)

 

Prepare the Infrastructure


To install “node_export” use the commands below:

cd /opt
sudo wget "https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz"
sudo tar -xvzf "./node_exporter-0.16.0.linux-amd64.tar.gz"
sudo cp "./node_exporter-0.16.0.linux-amd64/node_exporter" "./node_exporter"
sudo rm -rf "node_exporter-0.16.0.linux-amd64" "node_exporter-0.16.0.linux-amd64.tar.gz"

 

To configure “node_export” as a service create a service file “/etc/systemd/system/node_exporter.service”:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=ubuntu
ExecStart=/opt/node_exporter

[Install]
WantedBy=default.target

 

Tip:

- Under [Service] set the user who is going to run the service

- If you want to run node_exporter in a different port, add the flag "--web.listen-address" to the ExecStart, for example:

   ExecStart="/opt/node_exporter/node_exporter" "--web.listen-address=:9110"

 

And run the commands below:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

 

To check that “node_export” is working just browse to:

<server-ip>:9100/metrics

 

Image 1

 

Install and Configure Prometheus


We will deploy Prometheus as Docker container (and save the data in a volume). To do this we will create the configuration files and then we will create our custom image based in the official prometheus image (prom/prometheus).

 

– prometheus.yml: Is the base configuration file of the prometheus server:

# Global Config
global:
  scrape_interval:     15s
  evaluation_interval: 15s
  scrape_timeout: 10s

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:

  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'DEV'
    file_sd_configs:
      - files:
         - dev-targets.yml
         
  - job_name: 'PROD'
    file_sd_configs:
      - files:
         - prod-targets.yml

 

– dev-targets.yml: File used to specify the DEV servers information (used by prometheus.yml)

- targets: ['172.17.0.2:9100']
  labels:
    name: 'Server 1 - DEV'

- targets: ['172.17.0.3:9100']
  labels:
    name: 'Server 2 - DEV'

- targets: ['172.17.0.4:9100']
  labels:
    name: 'Server 3 - DEV'

- targets: ['172.17.0.5:9100']
  labels:
    name: 'Server 4 - DEV'
    
- targets: ['172.17.0.6:9100']
  labels:
    name: 'Server 5 - DEV'

 

– prod-targets.yml: File used to specify the PROD servers information (used by prometheus.yml)

- targets: ['172.17.0.7:9100']
  labels:
    name: 'Server 1 - PROD'

- targets: ['172.17.0.8:9100']
  labels:
    name: 'Server 2 - PROD'

- targets: ['172.17.0.9:9100']
  labels:
    name: 'Server 3 - PROD'

- targets: ['172.17.0.10:9100']
  labels:
    name: 'Server 4 - PROD'
    
- targets: ['172.17.0.11:9100']
  labels:
    name: 'Server 5 - PROD'
    
- targets: ['172.17.0.12:9100']
  labels:
    name: 'Server 6 - PROD'

 

Create a Dockerfile to build the prometheus image:

FROM prom/prometheus
COPY prometheus.yml /etc/prometheus/prometheus.yml
COPY dev-targets.yml /etc/prometheus/dev-targets.yml
COPY prod-targets.yml /etc/prometheus/prod-targets.yml

 

To create the docker image, a docker volume and run the prometheus container execute the below:

docker build -t custom/prometheus .
docker run -d --name prometheus -p 9090:9090 -v prometheus:/prometheus custom/prometheus

 

Check the prometheus configuration by browsing to:

http://localhost:9090/targets

Image 2

 

Install and Configure Grafana


We will deploy Grafana as Docker container (and save the data in a volume). Then we will configure Grafana to retrieve data from Prometheus (data source).

 

To run the grafana container (including several useful plugins) execute the below:

docker run -d --restart always --name grafana -p 3000:3000 -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource,grafana-piechart-panel,novalabs-annotations-panel,vonage-status-panel,fetzerch-sunandmoon-datasource,natel-discrete-panel,natel-influx-admin-panel" -v "grafana:/var/lib/grafana" grafana/grafana 

 

To add prometheus as a data source follow the steps below:

 

1) Browse to http://localhost:3000 and login with the default credentials:

User: admin
Password: admin

Image 3

 

2) Click “Add data source”

Image 4

 

3) Configure the data source with the details below:

Name: Prometheus
Type: Prometheus
Url: http://<prometheus-ip>:9090
Access: Server (Default)

 

To get the prometheus IP just inspect the container and look for the internal IP:

docker inspect prometheus

 

Image 5

 

Create the Grafana Dashboard


There is a lot of dashboards available in https://grafana.com/dashboards that can be imported and customized, we will use the dashboard “Node Exporter Full” (Id 1860)

Image 6

 

To import the dashboard follow the steps below:

1) Click “Import” under the “+” menu

Image 7

 

2) Enter the dashboard Id (1860) and click “Import”

Image 8

 

Conclusions


As we realized configure monitoring to our infrastructure is very easy. The real challenges are retrieve the right metrics without affect performance, build the right dashboards, configure useful alerts and monitor provisioned and dynamic servers.

Image 9

 

Add comment
facebook linkedin twitter email

Leave a Reply