Lecture 257: Introduction to Prometheus for metric monitoring

Imagine you're watching over a huge restaurant where every kitchen has its own menu, its own dishes, its own orders. You want to know everything — who ordered what and how much, who fulfills orders faster, where delays happen. Prometheus is like that waiter who runs between all the kitchens, gathers all the info, processes it, and brings it to you on a nice tray.

Prometheus is an open-source monitoring and metrics collection system. It was developed at SoundCloud and became one of the most popular systems for monitoring and alerting microservices. Prometheus helps us collect, store, and analyze metrics, then use them to monitor performance, spot bottlenecks, and prevent issues.

Key features of Prometheus

Prometheus was built with modern monitoring needs in mind. Here are its main "tricks":

1. pull model — Prometheus actively scrapes metrics from apps or services. This gives more control than a passive push model.
2. PromQL query language — for processing and analyzing metrics. With PromQL you can write complex queries and do aggregated computations.
3. Data storage — Prometheus stores time series data itself and organizes it in its own database.
4. Metrics exporter — Prometheus makes it easy to add new metrics for monitoring apps, system services, or hardware.
5. Integration with visualization tools — for example, Grafana for building great dashboards.

Light-hearted joke: "why doesn't Prometheus like the push model? Because the application might think: 'Oh, I didn't make it in time!'"

Prometheus architecture

Prometheus is based on a simple, intuitive architecture:

┌───────────┐   ┌───────────────┐
│   Target  │   │   Exporter    │
└───────────┘   └───────────────┘
       ▲                 ▲
       │ Pull Metrics    │
       │                 │
       ▼                 ▼
┌───────────────────────────┐
│       Prometheus Server   │
│ ┌───────────────────────┐ │
│ │Time Series Database   │ │
│ │(TSDB)                 │ │
│ └───────────────────────┘ │
└───────────────────────────┘
        │
        │ Query Data
        ▼
┌─────────────────────────┐
│     Visualization Tool  │
│   (e.g., Grafana)       │
└─────────────────────────┘

1. Target (Target system) — these are your applications that Prometheus scrapes over HTTP for metrics.
2. Exporter — an application that converts metrics into a format Prometheus understands. For example, node_exporter for collecting server system metrics.
3. Prometheus Server — the main component that scrapes metrics, stores them, and exposes an API for queries.
4. Visualization Tool — tools for visualizing data. Prometheus has a basic web UI, but Grafana is more commonly used.

Why is Prometheus so popular?

Prometheus gained popularity because it's simple and powerful. Here are a few reasons people pick it:

1. Open Source — no fees for usage.
2. Flexibility — you can customize metrics for any need.
3. Community and plugins — there are tons of integrations for Prometheus, including Spring Boot, Docker, Kubernetes, and more. You can even find an exporter to monitor your favorite coffee machine (yes, they exist).
4. Good for microservices — especially when services change frequently or new ones appear.

Example use cases

• You want to know how many requests your REST API handles each minute.
• You need to monitor memory or CPU usage in real time.
• You need to check microservice response times and spot bottlenecks before users start complaining.

Core concepts of Prometheus

Before we jump into hands-on stuff, let's go over the key Prometheus concepts:

1. Metrics — the data that's collected and analyzed. For example, "number of active connections" or "request processing time".
2. Time Series — sequential records of metric values over time.
3. Labels — additional metadata for metrics. For example, a "label" for each microservice.
4. Exporters — programs that prepare data for Prometheus.

Practical exercise: Installing and configuring Prometheus

Setting up the environment

1. Make sure Docker is installed (if not — now's a good time to fix that).
2. Pull the Prometheus image from Docker Hub:

   docker pull prom/prometheus
   

3. Run Prometheus:

   docker run -d -p 9090:9090 --name=prometheus prom/prometheus
   

4. Open your browser and go to http://localhost:9090. Congrats, Prometheus is up!

Integrating Spring Boot with Prometheus

Now let's hook our microservices up to Prometheus! We'll configure Spring Boot Actuator, which already exposes metrics.

1. Add dependencies to pom.xml:

   
   <dependency>
       <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-starter-actuator</artifactId>
   </dependency>
   <dependency>
       <groupId>io.micrometer</groupId>
       <artifactId>micrometer-registry-prometheus</artifactId>
   </dependency>
   

2. Configure application.yml:

   management:
     endpoints:
       web:
         exposure:
           include: "*"
     metrics:
       export:
         prometheus:
           enabled: true
   

3. Start your Spring Boot app. Go to http://localhost:8080/actuator/prometheus. You'll see lots of metrics!

Configuring Prometheus to scrape Spring Boot metrics

1. Create the configuration file prometheus.yml:

   
   global:
   scrape_interval: 15s
   
   scrape_configs:
   - job_name: "spring-boot-app"
     static_configs:
     - targets: ["host.docker.internal:8080"]
   

   Be careful with the host.docker.internal address — it works for Docker on Windows and macOS. On Linux use localhost.
2. Restart Prometheus:

   docker stop prometheus
   docker rm prometheus
   docker run -d -p 9090:9090 \
   -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
   --name=prometheus prom/prometheus
   

3. Go to the Prometheus UI and run the query:

   http_server_requests_seconds_sum
   

   You'll see your application's metrics!

In the next lecture we'll cover how to layer metric monitoring with visualization using Grafana, so you can turn dry numbers into nice dashboards.