Lecture 209: Major mistakes when using EDA and how to avoid them

Let's talk about the pitfalls you might hit when designing an event-driven architecture. As they say, "the smart learn from others' mistakes", so let's break down the most common errors and how to avoid them.

Mistake #1: Excessive coupling between services

If your event system looks like a web where every service depends on dozens of others, something went wrong. The main idea of EDA is to decouple services and make them independent. But if every service depends on others in an "I'm waiting for them to do something" pattern, that's not loose coupling anymore — it's a tangled mess.

Why does this happen?

• Poor event design where one event triggers a cascade of other events.
• Using a single topic for multiple data types, forcing services to inspect data that isn't theirs.
• "Thin" events that don't contain enough context, forcing subscribers to call other services for extra info.

How to avoid it?

• Design events with context in mind. Each event should be self-contained and include all necessary information.
• Minimize the number of subscribers. A service should subscribe only to events it actually needs.
• Use different topics for different event types. This reduces the chance of cross-contamination.

Real-life example: if the "Orders" microservice publishes an event OrderCreated, the "Inventory" microservice shouldn't depend on a response from the "Payment" service. Let them operate independently of each other.

Mistake #2: Weak error handling and lack of guaranteed delivery

Imagine: you sent an event but nobody processed it. Or worse, the service that was supposed to handle the event crashed. Your data is just lost. This is the classic problem tied to missing proper error handling.

Why does this happen?

• Using "at most once" delivery in critical scenarios.
• Incorrect message broker configuration (for example, a short retention period in Kafka).
• Ignoring message retries.

How to avoid it?

• Use the "at least once" approach. Let messages be delivered at least once, even if there's a risk of duplicate processing.
• Implement idempotency. If the same event is processed multiple times, it should not harm the system. For example, process a transaction only if its status is not yet "completed".
• Monitor the message lifecycle. Set up alerts for cases where an event hasn't reached a subscriber within a reasonable time.

Example of idempotent processing code:

@Service
public class PaymentEventProcessor {

    @Autowired
    private PaymentRepository paymentRepository;

    public void processPaymentEvent(PaymentEvent event) {
        // Check if the event was processed before
        if (paymentRepository.existsByTransactionId(event.getTransactionId())) {
            // Just log and exit
            log.info("Event with ID {} has already been processed", event.getTransactionId());
            return;
        }

        // Process the event
        Payment payment = new Payment(event.getTransactionId(), event.getAmount());
        paymentRepository.save(payment);
    }
}

Mistake #3: Overly granular events

Instead of a single OrderCreated event that contains all order info, you generate a bunch of tiny events: OrderItemAdded, OrderItemRemoved, OrderAddressChanged, and so on. That can lead to "event chaos", where subscribers drown in processing a myriad of tiny events.

Why does this happen?

• Overdoing it in an attempt to make the system flexible.
• Lack of experience in event architecture design.

How to avoid it?

• Events should be coarse-grained. Let each event describe a complete action.
• Refactor events. If your events are too small, combine them into larger ones that include context.

Antipattern:

class OrderEvents {
    class OrderItemAdded { /* ... */ }
    class OrderItemRemoved { /* ... */ }
    class OrderAddressChanged { /* ... */ }
}

Better approach:

class OrderCreatedEvent {
    private String orderId;
    private List<OrderItem> items;
    private Address shippingAddress;
    // All the information is here!
}

Mistake #4: Neglecting monitoring and logging

If you don't know what's happening with your events, you lose control of the system. An event can get stuck, fail to reach a subscriber, or cause an error — and you might not even notice.

Why does this happen?

• No centralized logging.
• Monitoring level doesn't match the system's complexity.

How to avoid it?

• Monitoring tools: ELK stack (Elasticsearch, Logstash, Kibana), Prometheus, Grafana.
• Distributed tracing: Use Spring Sleuth and Zipkin to trace the path of events through the system.
• Alerts: Set up alerts to learn about problems in real time.

Example of setting up monitoring with Grafana:

1. Set up metrics in Prometheus.
2. Create a dashboard in Grafana for metrics like message latency, events per hour, etc.

Mistake #5: No strategies for handling increased load

The system works fine at 100 events per minute. What happens if it becomes 10,000? Or a million? If you don't account for growth, your architecture may collapse under the load.

Why does this happen?

• Wrong choice of tools (for example, using RabbitMQ instead of Kafka for high-load systems).
• Lack of horizontal scaling.

How to avoid it?

• Use message brokers suited to your loads. Kafka handles millions of messages well, but it's not ideal for very low-latency request patterns.
• Add partitions. Kafka allows horizontal scaling of throughput using partitions.
• Do load testing. Use tools like Apache JMeter to validate your architecture.

Mistake #6: Improper management of event lifecycle

Your events shouldn't hang around forever. If you forget to clean up old messages or don't set a retention policy, you can run into broker storage exhaustion.

Why does this happen?

• No event lifecycle policy.
• Incorrect message broker settings.

How to avoid it?

• Clean up old data. Use the retention.ms setting in Kafka to remove stale messages.
• Automate processes. Set up scripts to manage lifecycle and cleanup.

We've covered six main mistakes that can arise when designing event-driven systems. Despite the topic's complexity, if you follow the recommendations above, you can build systems that are not only reliable but also easy to scale.