Monitoring your Linux Servers with Prometheus and Grafana in 7 Minutes

September 13, 2018

no comments


At the end of this post we will get a Grafana dashboard with all the metrics related to our servers.



Prometheus is a tool for store time series data and manage alerts. It works as a pull based system, the Prometheus server fetches the metrics values from the target servers periodically. To expose the metrics in each server we will use “node_export” which basically expose the server metrics using the port 9100 (by default). And Grafana is a tool for query, visualize and understand your metrics. We will use it to create our dashboard.


Understand your Infrastructure

The first thing we need to do is understand our infrastructure. That means know the following about each server:

Server Name
Ports in Use
Type (Dev, Staging, Production, etc)


Prepare the Infrastructure

To install “node_export” use the commands below:

cd /opt
sudo wget ""
sudo tar -xvzf "./node_exporter-0.16.0.linux-amd64.tar.gz"
sudo cp "./node_exporter-0.16.0.linux-amd64/node_exporter" "./node_exporter"
sudo rm -rf "node_exporter-0.16.0.linux-amd64" "node_exporter-0.16.0.linux-amd64.tar.gz"


To configure “node_export” as a service create a service file “/etc/systemd/system/node_exporter.service”:

Description=Node Exporter





- Under [Service] set the user who is going to run the service

- If you want to run node_exporter in a different port, add the flag "--web.listen-address" to the ExecStart, for example:

   ExecStart="/opt/node_exporter/node_exporter" "--web.listen-address=:9110"


And run the commands below:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter


To check that “node_export” is working just browse to:



Image 1


Install and Configure Prometheus

We will deploy Prometheus as Docker container (and save the data in a volume). To do this we will create the configuration files and then we will create our custom image based in the official prometheus image (prom/prometheus).


– prometheus.yml: Is the base configuration file of the prometheus server:

# Global Config
  scrape_interval:     15s
  evaluation_interval: 15s
  scrape_timeout: 10s

# Alertmanager configuration
  - static_configs:
    - targets:
      # - alertmanager:9093

  # - "first_rules.yml"
  # - "second_rules.yml"


  - job_name: 'prometheus'
    - targets: ['localhost:9090']

  - job_name: 'DEV'
      - files:
         - dev-targets.yml
  - job_name: 'PROD'
      - files:
         - prod-targets.yml


– dev-targets.yml: File used to specify the DEV servers information (used by prometheus.yml)

- targets: ['']
    name: 'Server 1 - DEV'

- targets: ['']
    name: 'Server 2 - DEV'

- targets: ['']
    name: 'Server 3 - DEV'

- targets: ['']
    name: 'Server 4 - DEV'
- targets: ['']
    name: 'Server 5 - DEV'


– prod-targets.yml: File used to specify the PROD servers information (used by prometheus.yml)

- targets: ['']
    name: 'Server 1 - PROD'

- targets: ['']
    name: 'Server 2 - PROD'

- targets: ['']
    name: 'Server 3 - PROD'

- targets: ['']
    name: 'Server 4 - PROD'
- targets: ['']
    name: 'Server 5 - PROD'
- targets: ['']
    name: 'Server 6 - PROD'


Create a Dockerfile to build the prometheus image:

FROM prom/prometheus
COPY prometheus.yml /etc/prometheus/prometheus.yml
COPY dev-targets.yml /etc/prometheus/dev-targets.yml
COPY prod-targets.yml /etc/prometheus/prod-targets.yml


To create the docker image, a docker volume and run the prometheus container execute the below:

docker build -t custom/prometheus .
docker run -d --name prometheus -p 9090:9090 -v prometheus:/prometheus custom/prometheus


Check the prometheus configuration by browsing to:


Image 2


Install and Configure Grafana

We will deploy Grafana as Docker container (and save the data in a volume). Then we will configure Grafana to retrieve data from Prometheus (data source).


To run the grafana container (including several useful plugins) execute the below:

docker run -d --restart always --name grafana -p 3000:3000 -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource,grafana-piechart-panel,novalabs-annotations-panel,vonage-status-panel,fetzerch-sunandmoon-datasource,natel-discrete-panel,natel-influx-admin-panel" -v "grafana:/var/lib/grafana" grafana/grafana 


To add prometheus as a data source follow the steps below:


1) Browse to http://localhost:3000 and login with the default credentials:

User: admin
Password: admin

Image 3


2) Click “Add data source”

Image 4


3) Configure the data source with the details below:

Name: Prometheus
Type: Prometheus
Url: http://<prometheus-ip>:9090
Access: Server (Default)


To get the prometheus IP just inspect the container and look for the internal IP:

docker inspect prometheus


Image 5


Create the Grafana Dashboard

There is a lot of dashboards available in that can be imported and customized, we will use the dashboard “Node Exporter Full” (Id 1860)

Image 6


To import the dashboard follow the steps below:

1) Click “Import” under the “+” menu

Image 7


2) Enter the dashboard Id (1860) and click “Import”

Image 8



As we realized configure monitoring to our infrastructure is very easy. The real challenges are retrieve the right metrics without affect performance, build the right dashboards, configure useful alerts and monitor provisioned and dynamic servers.

Image 9


Add comment
facebook linkedin twitter email

Leave a Reply