4.1 Installing Prometheus
Prometheus is a powerful open-source monitoring and alerting system designed to collect and store metrics from various sources, including Docker containers. In this lecture, we'll take a detailed look at setting up container monitoring with Prometheus, including installation, metric collection configuration, and Docker integration.
1. Downloading and Installing Prometheus
First, download the latest version of Prometheus from the official website:
wget https://github.com/prometheus/prometheus/releases/download/v2.30.3/prometheus-2.30.3.linux-amd64.tar.gz
Unpack the archive:
tar xvfz prometheus-*.tar.gz
cd prometheus-*
Move the binary files to /usr/local/bin:
sudo mv prometheus /usr/local/bin/
sudo mv promtool /usr/local/bin/
4.2 Setting up the Prometheus configuration file
Prometheus uses the configuration file prometheus.yml to define metric sources and data collection settings. Create or edit the prometheus.yml file:
global:
scrape_interval: 15s # Metric collection interval
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'cadvisor'
static_configs:
- targets: ['localhost:8080']
This configuration file includes two jobs:
prometheus: for collecting metrics from Prometheus itself.cadvisor: for collecting metrics from cAdvisor, which will be used for monitoring Docker containers.
5.3 Starting Prometheus
Start Prometheus using the created configuration file:
prometheus --config.file=prometheus.yml
Prometheus will be available at http://localhost:9090.
5.4 Installing and configuring cAdvisor
cAdvisor (Container Advisor) is a tool for container monitoring that collects and provides metrics for Docker containers. To use it, follow these steps:
Run cAdvisor in a Docker container:
docker run -d \
--name=cadvisor \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--privileged \
gcr.io/cadvisor/cadvisor:latest
cAdvisor will be available at http://localhost:8080.
5.5 Updating Prometheus Configuration for cAdvisor
Make sure the Prometheus configuration includes a job for cAdvisor. An example configuration was shown above.
Restart Prometheus to apply the changes:
prometheus --config.file=prometheus.yml
5.6 Checking metrics collection
Go to the Prometheus web interface at http://localhost:9090 and make sure Prometheus is collecting metrics from cAdvisor. To do this, run the following query in Prometheus:
container_cpu_usage_seconds_total
You should see data collected from Docker containers.
5.7 Adding Extra Metrics
You can add extra metrics in the Prometheus config file to collect more data about your containers. Example of additional metrics:
- job_name: 'node'
static_configs:
- targets: ['localhost:9100']
For this, you’ll need to install node_exporter, which provides metrics about the host system's state.
Download and install node_exporter:
wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
tar xvfz node_exporter-*.tar.gz
sudo mv node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin/
Run node_exporter:
node_exporter &
Now, node_exporter will be available at http://localhost:9100.
5.8 Creating Alert Rules
Prometheus supports creating alert rules to notify administrators about issues. Add the alert rules to the prometheus.yml config file:
rule_files:
- "alert.rules"
alerting:
alertmanagers:
- static_configs:
- targets:
- "localhost:9093"
Create a file alert.rules with an example of rules:
groups:
- name: example
rules:
- alert: HighCPUUsage
expr: container_cpu_usage_seconds_total{job="cadvisor"} > 0.85
for: 1m
labels:
severity: warning
annotations:
summary: "High CPU usage detected"
description: "Container is using more than 85% CPU for more than 1 minute."
GO TO FULL VERSION