Lecture 258: Hands-on: setting up metrics monitoring with Prometheus

In previous lectures we got familiar with the concept of Observability in microservice systems, learned how important the ability to observe the system is for its stable operation and scalability. We figured out that distributed tracing and centralized logging are key tools for analyzing system behavior. We looked at Spring Cloud Sleuth and Zipkin for tracing requests across microservices, and also studied the ELK stack for logging. Now it's time to move on to collecting and monitoring the system's metrics using the popular tool Prometheus.

Think of a server as an athlete. Metrics are their pulse, blood pressure and oxygen level. If you don't track those, how will you know when they're close to "running until they break"? Metrics monitoring lets you:

• See how your application behaves in real time.
• Detect failures early, before users start complaining.
• Find performance bottlenecks.
• Improve the overall stability of the system.

Prometheus is the "doctor" that collects your app's metrics and helps keep it "healthy".

Practical Prometheus setup

Step 1: Installing Prometheus

Prometheus is a service that pulls data via an HTTP interface. Metrics are published by microservices in a format Prometheus understands. For installing Prometheus we'll use a local machine or a Docker container.

Installation via Docker

1. Make sure you have Docker installed.
2. Run Prometheus with the command:

   docker run --name prometheus -d -p 9090:9090 prom/prometheus
   

3. Open your browser and go to http://localhost:9090 to check if Prometheus is running.

Configuring Prometheus

Prometheus uses the prometheus.yml file for configuration. We'll tell Prometheus where to look for our microservices (or other metric sources).

Create a file named prometheus.yml with the following content:

global:
  scrape_interval: 15s  # How often to collect data

scrape_configs:
  - job_name: 'spring-boot-app'
    static_configs:
      - targets: ['localhost:8080']  # Address of your application

If you're running Prometheus via Docker, mount the configuration:

docker run --name prometheus -d -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Step 2: Setting up the Spring Boot app

Adding the Actuator dependency

Spring Boot Actuator lets your app publish metrics that are compatible with Prometheus. Add the dependency to pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Also add the dependency for Prometheus integration:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Configuring Actuator in application.yml

Enable Actuator and configure it to expose metrics:

management:
  endpoints:
    web:
      exposure:
        include: '*'  # Enable all Actuator endpoints
  endpoint:
    metrics:
      enabled: true  # Enable metrics
  metrics:
    export:
      prometheus:
        enabled: true  # Enable export for Prometheus

After this, your application will expose the /actuator/prometheus endpoint, which will output metrics in Prometheus format.

Verifying the setup

Run your Spring Boot app (for example on port 8080), then open http://localhost:8080/actuator/prometheus. You should see a list of metrics like this:

# HELP jvm_memory_used_bytes Used bytes of a given JVM memory area.
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="G1 Eden Space",} 1.23456E7
jvm_memory_used_bytes{area="heap",id="G1 Old Gen",} 6.789E7
...

If you see the metrics output, everything is working correctly!

Step 3: Integrating Prometheus and Spring Boot

Now we need to tell Prometheus where to find our app. Open the prometheus.yml file and add the following configuration:

scrape_configs:
  - job_name: 'spring-boot-app'
    static_configs:
      - targets: ['host.docker.internal:8080']  # Or just 'localhost:8080' outside Docker

After saving the changes, restart the Prometheus container:

docker restart prometheus

In the Prometheus UI (http://localhost:9090) you'll be able to see the spring-boot-app job and start querying metrics.

Step 4: Visualizing metrics (a bit of magic)

While Prometheus is great for collecting data, Grafana is better for analysis and visualization. Grafana setup is the topic of the next lecture, but let's briefly look at how Prometheus metrics appear in Grafana.

1. Install Grafana:

   docker run -d --name=grafana -p 3000:3000 grafana/grafana
   

2. Open Grafana at http://localhost:3000 and add Prometheus as a data source.
3. Create a dashboard and add graphs for metrics like http_server_requests_seconds_count (number of HTTP requests) or jvm_memory_used_bytes.

What to do if something goes wrong?

Sometimes you may run into issues. For example:

• Prometheus doesn't show metrics. Check the prometheus.yml file and make sure the specified address (for example, localhost:8080) is reachable.
• The /actuator/prometheus endpoint doesn't generate metrics. Make sure Actuator and Micrometer dependencies are added to the project, and that the settings in application.yml are correct.
• Prometheus didn't start. Make sure port 9090 is free, or change the port in the Docker configuration.

Real-world applications

This monitoring technique is widely used in production. For example:

• You can track the number of HTTP requests and their processing time to find performance bottlenecks.
• Tracking JVM memory and threads helps prevent critical failures like OutOfMemoryError.
• Prometheus and Grafana can be used to set up alerts (notifications) so you can react instantly to problems.

Now you know how to integrate Prometheus with Spring Boot applications and will start to understand why logging and metrics are the eyes and ears of your system!