Lecture 259: Using Grafana to Visualize Metrics

We've already gotten familiar with the concept of Observability and how it affects maintaining and running microservices. We learned about distributed tracing with Zipkin, set up Sleuth to automatically add trace IDs, and checked how requests cross microservice boundaries. We also studied the ELK stack for centralized logging and started integrating Prometheus for metrics collection. Now it's time for the "dessert" — visualizing all the collected data with Grafana. After all, Observability isn't complete without pretty charts!

Why Grafana?

We've already played around with Grafana a bit, but it's time to revisit the platform and learn how to use it in a microservices architecture.

Imagine you've gathered all the metrics and logs from your microservices. They're neatly sitting in a database, in Prometheus or Elasticsearch. And now you sit down to analyze... via the command line.

As much as you might love the console, parsing thousands of lines of text isn't the most fun. Grafana saves us by giving powerful tools to visualize data. It's like Maps for your data: instead of dull coordinates on paper — colorful maps with a route.

Grafana highlights:

• Connects to lots of data sources: Prometheus, Elasticsearch, MySQL and even Google Sheets.
• Flexible dashboards: you decide what and how to display.
• Powerful metrics: charts, histograms, tables.
• Customizable alerts: get notified about issues before they become critical.
• Supports plugins and custom visualizations.

Installing Grafana

Grafana is available as packages, Docker images and even cloud offerings. But we'll do "real developer stuff" and install via Docker, since it's quick and easy.

Running Grafana from Docker

Make sure Docker is installed on your machine. Then run this command:

docker run -d -p 3000:3000 --name=grafana grafana/grafana

• -d runs the container in the background.
• -p 3000:3000 maps host port 3000 to container port 3000.
• The container is ready! Grafana is now accessible in your browser at http://localhost:3000.

Logging into Grafana

Open your browser and go to http://localhost:3000. Default login and password:

• Login: admin
• Password: admin

On first login Grafana will ask you to change the password. Please change it — you're among professionals now, remember?

Integrating Grafana with Prometheus

We set up Prometheus earlier to scrape metrics from Spring Boot via Actuator. Now our task is to add Prometheus as a data source in Grafana.

Step 1: Add the data source

1. Open Grafana.
2. Go to the menu (the "gear" icon on the left) → Data Sources → Add data source.
3. From the list of sources pick Prometheus.
4. Enter the URL of your Prometheus server (usually http://localhost:9090).
5. Click Save & Test. If everything's set up right you'll see the message "Data source is working".

Step 2: Create your first dashboard

Now it's time to add some visualization magic:

1. Go to the main menu and choose Dashboard → New Dashboard.
2. Click Add new panel.
3. In the panel window choose the metric you want to visualize. For example, start with http_server_requests_seconds_count (the number of requests in your Spring Boot application).
4. Pick the chart type (for example, "Time series" for metrics that change over time).
5. Tweak the display settings and save.

Making pretty charts

Let's dive into how to make the most of Grafana!

Visualization types

Grafana offers many chart types:

• Time series: plots of metric changes over time.
• Table: tables to present data in tabular form.
• Gauge: visualizing numeric indicators, e.g., "CPU load in percent".
• Heatmap: heatmaps to show event density.

Example: monitoring HTTP requests

1. Go to your dashboard.
2. Add a new panel.
3. In the Query field select a Prometheus query:

   sum(rate(http_server_requests_seconds_count[5m])) by (status)
   

   This will show the number of HTTP requests over the last 5 minutes, grouped by status codes (e.g., 200, 404).
4. Choose the visualization type "Bar chart".
5. Save the panel. Voilà! Now you have a chart showing the distribution of HTTP requests.

Alerts in Grafana

Visualization is great, but alerts are a monitoring cornerstone. Grafana can send notifications to Slack, Email, Telegram and other alerting systems.

Example: alert on a spike in requests

Say you want to alert if the number of requests exceeds 100 over the last 5 minutes.

1. Open the panel you created for monitoring HTTP requests.
2. Click "Edit" on the panel → the Alert tab → Create Alert.
3. Configure the condition. For example:
   • Expression: sum(rate(http_server_requests_seconds_count[5m])) > 100
   • Evaluation interval: every 1 minute.
4. Specify the notification channel. For example, Slack:
   • Go to Contact Points in Grafana settings.
   • Add a new contact, pick Slack and provide the Webhook URL.

Dashboards for microservices

Let's think about what else we should monitor in microservices. Here are some ideas to get you started:

• HTTP errors: percentage of requests with 4xx and 5xx codes (count(status_code >= 400)/count(all_requests)).
• CPU and memory load: use JVM metrics provided by Spring Boot Actuator.
• Latency data: average request duration (avg(http_server_requests_seconds_sum / http_server_requests_seconds_count)).

Integrating other data sources

Grafana isn't just Prometheus. You can hook up additional sources to build a single unified dashboard:

• Elasticsearch: for showing logs.
• MySQL/Postgres: for querying and visualizing DB data directly.
• Google Analytics: if you need user traffic data.

Using all these options you can build a powerful observability tool for your microservices.

Frequently asked questions

1. Can I run Grafana in production? Yes, Grafana is widely used in production. You should secure access (for example, set a strong login/password) and use SSL (can be set up via a reverse proxy like Nginx).

2. Grafana is slow with lots of queries — what to do? Throttle query frequency. Use aggregations in Prometheus to reduce the amount of data Grafana has to fetch.

3. When should I use alerts? Only set alerts for critical metrics. Too many alerts will make them useless.

So that's Grafana — the tool that turns boring numbers into beautiful visualizations. Now your monitoring is not only useful but also easy on the eyes!