Lecture 255: Logging events in microservices with ELK

Have you ever tried to find a bug by digging through a pile of text log files? Imagine there are a couple dozen of those files, all scattered across different servers. To make situations like that easier, the ELK stack was invented. ELK is a trio of tools: Elasticsearch, Logstash and Kibana, which help collect, store, analyze, and visualize logs.

In a microservices architecture each service generates its own logs. If an error happens, you'll want to quickly and easily find where it occurred, inspect the chain of events, and understand why things went wrong. Centralized logging lets you:

• Collect logs from all microservices in one place.
• Analyze logs conveniently using queries (instead of "manual eyeballing").
• Quickly find interactions between services and figure out the root cause of issues.
• Visualize data for better understanding.

What is the ELK stack?

Let's break down each of the three letters in ELK:

1. Elasticsearch — a data store and search engine. It finds the logs you need across massive amounts of data in fractions of a second. It also supports queries based on JSON.
2. Logstash — a data processor. It collects logs from various sources, filters or transforms them, and sends them to Elasticsearch. You can think of it as the "courier" for logs.
3. Kibana — a visualization tool. It pulls data from Elasticsearch and draws charts, graphs, and tables so you can understand what's going on without pain.

When these three tools work together, they turn boring text logs into a powerful analysis toolkit.

Installation and configuration of ELK

Elasticsearch

Let's start by installing Elasticsearch, which will act as our log storage.

1. Download and install Elasticsearch.
2. After installation open the config file elasticsearch.yml, which you can find in the config folder.
3. Make sure Elasticsearch is reachable locally:

   network.host: 127.0.0.1
   http.port: 9200
   

4. Start Elasticsearch:

   ./bin/elasticsearch
   

   After startup the service will be available at http://localhost:9200.

Logstash

Now let's set up the second component — Logstash. It's the bridge that will accept logs from our Spring Boot application and send them to Elasticsearch.

1. Download and install Logstash.
2. Create a configuration file logstash.conf:

   
   input {
       beats {
           port => 5044
       }
   }
   
   output {
       elasticsearch {
           hosts => ["localhost:9200"]
       }
   }
   

   • In the input section we specify that we want to accept logs via port 5044.
   • In the output we configure Logstash to send data to Elasticsearch.
3. Start Logstash:

   ./bin/logstash -f logstash.conf
   

Kibana

The final step is installing and running Kibana so we can visualize the data.

1. Download and install Kibana.
2. Configure Kibana by editing the file kibana.yml:

   server.port: 5601
   server.host: "localhost"
   elasticsearch.hosts: ["http://localhost:9200"]
   

3. Start Kibana:

   ./bin/kibana
   

   Now the Kibana interface is available at http://localhost:5601.

So, the main components are installed. Now we can integrate our Spring Boot application with ELK.

Integrating microservices with ELK

Configuring logging in Spring Boot

It's time to make your app send logs directly to Logstash.

1. Open pom.xml and add the following dependencies:

   <dependency>
       <groupId>net.logstash.logback</groupId>
       <artifactId>logstash-logback-encoder</artifactId>
       <version>7.4</version>
   </dependency>
   

2. Configure logback-spring.xml to send logs to Logstash:

   
   <configuration>
       <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
           <destination>127.0.0.1:5044</destination>
           <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
       </appender>
   
       <root level="INFO">
           <appender-ref ref="LOGSTASH" />
       </root>
   </configuration>
   

3. Restart the application. Now logs will be automatically sent to Logstash, and from there — to Elasticsearch.

Checking logs

1. Open the Kibana interface (http://localhost:5601).
2. Go to the Discover section.
3. Set up the index (for example, logstash-*) and you'll see logs from your application.

Practical ELK setup for microservices

Let's imagine we have two microservices:

1. User Service — responsible for managing user data.
2. Order Service — responsible for processing orders.

Steps:

1. Add a Logstash configuration to each service, like shown earlier.
2. Configure both services to send logs to Logstash.
3. Use Kibana to analyze logs:
   • Find requests that go through both services.
   • If requests are correlated (for example, a Trace ID from Sleuth), follow their path.

What to keep in mind?

Logging is a powerful tool, but it comes with some challenges:

• Resource usage. Generating and processing a large volume of logs can slow the system down.
• Security. Make sure sensitive data (like passwords) doesn't end up in the logs.
• Storage. Elasticsearch has limits on data volume, so old logs should be archived or deleted.

Now that your microservices are equipped with centralized logging via ELK, you can sleep a lot easier. After all, diagnosing problems will become a matter of seconds, and visualizing logs in Kibana will definitely impress your team! (Or at least your tech lead.)