Prometheus monitoring

Prometheus is an open-source systems monitoring and alerting toolkit[1] released in 2012. Prometheus design is focused to collect and process metrics, not as an event logging system for logs.[2]


Install Prometheus

edit

Configuration files

edit
  • Linux: /etc/prometheus/prometheus.yml
  • macOS: /usr/local/etc/prometheus.args

Binaries

edit
  • prometheus
  • promtool

Configuring Prometheus

edit
  • In the /etc/prometheus directory, use nano or your favorite text editor to create a configuration file named prometheus.yml. For now, this file will contain just enough information to run Prometheus for the first time.
sudo nano /etc/prometheus/prometheus.yml


  • In the global settings, define the default interval for scraping metrics. Note that Prometheus will apply these settings to every exporter unless an individual exporter's own settings override the global.


  • This scrape_interval value tells Prometheus to collect metrics from its exporters every 15 seconds, which is long enough for most exporters. Now, add Prometheus itself to the list of exporters to scrape from with the following scrape_configs directive:


  • Prometheus uses the job_name to label exporters in queries and on graphs, so be sure to pick something descriptive here. And, as Prometheus exports important data about itself that you can use for monitoring performance and debugging, we've overridden the global scrape_interval directive from 15 seconds to 5 seconds for more frequent updates. Lastly, Prometheus uses the static_configs and targets directives to determine where exporters are running. Since this particular exporter is running on the same server as Prometheus itself, we can use localhost instead of an IP address along with the default port, 9090.


Your configuration file should now look like this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']


Basic Prometheus Operations

edit
  • Start Prometheus: :sudo systemctl start prometheus
  • Reload systemd:  :sudo systemctl daemon-reload prometheus
  • Verify the service's status: :sudo systemctl status prometheus

Installing Node Exporter

edit

Installing Node Exporter from Docker Hub

edit
  • docker pull prom/node-exporter && docker run prom/node-exporter

Installing Node Exporter from Prometheus binary relases

edit

To expand Prometheus beyond metrics about itself only, we'll install an additional exporter called Node Exporter. Node Exporter provides detailed information about the system, including CPU, disk, and memory usage.

  • Download the current stable version of Node Exporter into your home directory.
cd ~
curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz
  • Use the sha256sum command to generate a checksum of the downloaded file:
    sha256sum node_exporter-0.15.1.linux-amd64.tar.gz
  • Unpack the downloaded archive:
    tar xvf node_exporter-0.15.1.linux-amd64.tar.gz
  • This will create a directory called node_exporter-0.15.1.linux-amd64 containing a binary file named node_exporter, a license, and a notice.
  • Copy the binary to the /usr/local/bin directory and set the user and group ownership to the node_exporter user that you created in Step 1 and remove the leftover files from your home directory as they are no longer needed.
sudo cp node_exporter-0.15.1.linux-amd64/node_exporter /usr/local/bin
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
rm -rf node_exporter-0.15.1.linux-amd64.tar.gz node_exporter-0.15.1.linux-amd64

Now that you've installed Node Exporter, let's test it out by running it before creating a service file for it so that it starts on boot.

Running Node Exporter

edit

The steps for running Node Exporter are similar to those for running Prometheus itself. Start by creating the Systemd service file for Node Exporter and copy the following content into the service file:

sudo vi /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target


  • Reload systemd to use the newly created service: :sudo systemctl daemon-reload
  • Run Node Exporter using the following command: :sudo systemctl start node_exporter
  • Verify that Node Exporter's running correctly: :sudo systemctl status node_exporter


  • Lastly, enable Node Exporter to start on boot.
sudo systemctl enable node_exporter


Configuring Prometheus to Scrape Node Exporter

edit

Because Prometheus only scrapes exporters which are defined in the scrape_configs portion of its configuration file, we'll need to add an entry for Node Exporter, just like we did for Prometheus itself.

At the end of the scrape_configs block, add a new entry called node_exporter.

sudo nano /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']

Finally, restart Prometheus to put the changes into effect and verify status

Activities

edit
  1. Install Prometheus in Linux, Run Prometheus in Linux and connect to graphical interface http://localhost:9090/graph
  2. Read Prometheus changelog: https://github.com/prometheus/prometheus/blob/master/CHANGELOG.md
  3. Read Stackoverflow prometheus questions: https://stackoverflow.com/questions/tagged/prometheus?tab=Frequent

See also

edit
  1. https://prometheus.io/docs/introduction/overview/
  2. https://prometheus.io/docs/introduction/faq/#how-to-feed-logs-into-prometheus?
  3. https://prometheus.io/docs/prometheus/latest/installation/
  4. https://github.com/prometheus/node_exporter