Lecture 205: Tools for Event-Driven Architecture: Kafka, RabbitMQ, ActiveMQ

Imagine you're organizing a conference and need to get an important message out to everyone. Sure, you could go up to each person individually (like REST APIs do in synchronous architectures), but that'll take forever. Instead, it's smarter to use a loudspeaker (!) (a message broker) to broadcast your message once, and the listeners (subscribers) will pick it up on their own.

In the EDA world, these loudspeakers are message brokers. Their job is to deliver an event from the sender (producer) to the receivers (consumers). You also need to ensure:

• Reliable message delivery.
• Scalability and high throughput.
• Asynchronous message processing.

Three popular technologies that help with this are Apache Kafka, RabbitMQ, and ActiveMQ.

Overview of Apache Kafka, RabbitMQ, and ActiveMQ

Apache Kafka

You already know Kafka is the rocket engine of the message broker world. Quick reminder: it's a distributed system originally built at LinkedIn and now maintained by the Apache Software Foundation. Kafka was designed to handle huge amounts of data in real time with very high throughput.

• Main Kafka advantages:
  • High performance: millions of messages per second.
  • Durable message delivery using logs.
  • Supports horizontal scaling.
  • Good integration with the Big Data ecosystem (Spark, Flink, etc.).
• When to use Kafka:
  • Stream processing (e.g., logging or event tracking).
  • Real-time analytics.
  • Building microservices where high throughput is required.
• Drawbacks:
  • Relatively complex to configure and manage.
  • Requires solid knowledge of distributed systems architecture.

RabbitMQ

RabbitMQ is the classic rabbit of the message broker world. It's one of the oldest brokers and uses the AMQP (Advanced Message Queuing Protocol). It offers broader functionality for application integration.

• Main RabbitMQ advantages:
  • Support for multiple protocols (AMQP, MQTT, HTTP, STOMP, etc.).
  • Different routing patterns (queues, pub/sub, fanout, etc.).
  • Flexible configuration and easy to integrate.
• When to use RabbitMQ:
  • When you need flexible message routing.
  • For integrating heterogeneous systems using different protocols.
  • When the system has small-to-medium load.
• Drawbacks:
  • Not as high throughput compared to Kafka.
  • Scaling has limits.

ActiveMQ

ActiveMQ is the veteran of the message broker market, developed by the Apache Foundation. Unlike Kafka, which focuses on data streams, ActiveMQ is more suited for classic messaging scenarios in business apps.

• Main ActiveMQ advantages:
  • Easy to use and configure.
  • Supports transactions and guaranteed delivery.
  • Works well for small systems.
• When to use ActiveMQ:
  • In monoliths or small microservice setups.
  • When you need transaction support and persistent queues.
• Drawbacks:
  • Performance lags far behind Kafka.
  • Fewer features compared to RabbitMQ.

Comparison: Kafka, RabbitMQ, and ActiveMQ

How to pick a tool for EDA?

To avoid picking the wrong tool, ask yourself two key questions:

1. What are your performance requirements?
2. How complex do your message routes need to be?

For stream processing or real-time log analysis your go-to is Kafka.

If you need a flexible tool with lots of interaction patterns, lean toward RabbitMQ.

And if you have a simple system with small queues, ActiveMQ is a solid choice.

Usage examples

Example 1: Apache Kafka

An e-commerce site tracks user actions: adding items to cart, viewing product pages. These events are written to a Kafka topic, and analytics systems consume the streams to build recommendations.

// Example of creating a Kafka producer
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Producer7<String, String> producer = new KafkaProducer7<>(props);
producer.send(new ProducerRecord7<>("shopping-cart", "user123", "add_to_cart: product456"));
producer.close();

Example 2: RabbitMQ

In a CRM system customer orders land in a RabbitMQ queue. The order processing service (consumer) takes messages from the queue for further handling.

ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
try (Connection connection = factory.newConnection();
     Channel channel = connection.createChannel()) {
    channel.queueDeclare("orders_queue", false, false, false, null);
    String message = "New Order: 12345";
    channel.basicPublish("", "orders_queue", null, message.getBytes());
    System.out.println(" [x] Sent '" + message + "'");
}

Example 3: ActiveMQ

In a banking system transactions are exchanged using ActiveMQ. Each message contains transaction info which is then persisted to a database.

ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
try (Connection connection = connectionFactory.createConnection()) {
    connection.start();
    Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
    Queue queue = session.createQueue("transaction_queue");
    MessageProducer producer = session.createProducer(queue);
    TextMessage message = session.createTextMessage("Transaction: 100$ to Account 12345");
    producer.send(message);
    System.out.println("Sent: " + message.getText());
}

Pitfalls when working with message brokers

Working with any broker you can run into typical problems. For example, with Kafka you need to pay attention to correct partition configuration to avoid overloading individual brokers. RabbitMQ may require deep analysis of routing rules for complex scenarios. And ActiveMQ can become a bottleneck under heavy load.

Bottom line: which one to choose?

Your choice of message broker depends on the specific problem and the characteristics of your system. Kafka is your pick for big, high-throughput stream processing systems. RabbitMQ fits better for flexible routing and integration, and ActiveMQ covers the needs of small systems with minimal requirements.

Don't forget it's best to test a few options before committing to one. And yes, remember to regularly clean up your message queues — otherwise those "parties of life" will end sooner than you think.