Lecture 208: Logging and Event Monitoring in Microservices

In a microservice architecture, logging and monitoring are important for:

• Detecting errors and failures.
• Tracking the "life" of an event, from its occurrence to its processing.
• Debugging and performance analysis.
• Understanding the overall health of the system and its components.

Event logging: the basics

Event logging in microservices has its own specifics. The main idea — record every event so you always know what's happening and in what order. But logging shouldn't be just "writing to a file". It should be structured and well thought out. Let's break down the main components.

When logging events it's important to record:

1. Event ID: a unique identifier for each event.
2. Event type: e.g., "Order creation", "Payment", "Delivery status update".
3. Event source: which producer (service) generated the event.
4. Timestamp: when the event was created or processed.
5. Event status: whether there was a failure during processing (for example, "successful execution" or "error").
6. Correlation ID: used to link multiple events related to the same process (for example, all stages of an order lifecycle).

Example JSON event described above:

{
  "eventId": "12345",
  "eventType": "OrderCreated",
  "source": "OrderService",
  "timestamp": "2023-10-12T10:15:30.000Z",
  "status": "SUCCESS",
  "correlationId": "abc-123-correlation-id"
}

Logging tools

In the Java world the most popular tool for logging is SLF4J along with its implementations like Logback or Log4j2. These libraries let you manage logs via configuration and make integration with other services easier.

Example Logback configuration (logback.xml):

<configuration>
    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>logs/microservice.log</file>
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="FILE" />
    </root>
</configuration>

Example of using SLF4J in Java:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class EventLogger {
    private static final Logger logger = LoggerFactory.getLogger(EventLogger.class);

    public void logEvent(String eventId, String eventType, String status) {
        logger.info("Event logged: id={}, type={}, status={}", eventId, eventType, status);
    }
}

Event monitoring: the basics

If logging is a record of the past, monitoring is a way to look at the present. Event monitoring lets you see what's happening in the system in real time and alert you to issues.

When we talk about event monitoring, analytics matter:

1. Number of events: how many events were processed over a given period.
2. Errors: e.g., the percentage of events that ended with an error.
3. Latency: time from event occurrence to its processing.
4. Queue sizes: how many events are still waiting to be processed.

What does it look like?

Imagine a dashboard where you can see how many orders were created in the last hour, how many of them completed successfully, and how many failed. For example:

Monitoring tools

For monitoring events and building dashboards you can use:

• Prometheus for collecting metrics.
• Grafana for visualization.
• ELK stack (Elasticsearch, Logstash, Kibana) for log analysis.

Example Prometheus setup in Spring Boot:

Add the dependency to pom.xml:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Create a custom metric for the number of processed events:

import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;

@Component
public class EventMetrics {

    private final MeterRegistry meterRegistry;

    public EventMetrics(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    public void incrementProcessedEvents() {
        meterRegistry.counter("event.processed.count").increment();
    }
}

After that the event_processed_count metric will be available for monitoring!

Setting up ELK for log analysis

If you want to analyze event logs and build dashboards, the ELK stack is a great choice.

1. Logstash collects logs from your application.
2. Elasticsearch indexes the log data.
3. Kibana visualizes the data and lets you filter information.

Example Logstash configuration to read logs from a file:

input {
  file {
    path => "/path/to/log/file.log"
    start_position => "beginning"
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "microservice-logs"
  }
}

As a result, all your application's logs will be available in the Kibana UI, where you can build dashboards, search for errors, or analyze performance.

Monitoring points and analysis

In real systems it's important not only to record facts but also to analyze them. For example, if you see an increase in error events, the monitoring system (for example, Prometheus + Grafana or ELK) can send you an alert.

Setting up alerts in Prometheus:

groups:
  - name: event-monitoring
    rules:
      - alert: HighErrorRate
        expr: rate(event_errors_total[5m]) > 0.05
        for: 1m
        labels:
          severity: warning
        annotations:
          description: "High error rate detected in the last 5 minutes!"

Now, if the acceptable error level is exceeded, you'll get a notification.

What can go wrong?

Logging and monitoring of events are perfect only in theory. In practice you may face:

• Too many logs, which makes analysis hard. It's important to log only what you really need.
• Monitoring bottlenecks, for example if the system can't handle the load of metric data.
• Difficulties with data correlation, especially if you're using multiple message brokers or microservices.

To avoid that, stick to the principle "less is more": collect only necessary data and carefully design the structure of events and metrics.

Now we know how to make microservices "transparent", and how to squeeze the maximum useful info out of simple logs and charts. On to the next topic!