Messaging and streaming
RabbitMQ
RabbitMQ is a mature message broker with flexible routing through exchanges, acknowledgements and AMQP support.
What it is
RabbitMQ appeared in 2007 at Rabbit Technologies, then moved to VMware and is now developed by Broadcom. It is a broker written in Erlang that implements AMQP 0-9-1 and also supports MQTT, STOMP and AMQP 1.0.
A producer sends a message to an exchange, which routes it to queues by rules (direct, topic, fanout, headers). Consumers take messages from queues and acknowledge processing (ack), otherwise the message returns.
RabbitMQ fits background jobs, RPC and service integration. Newer quorum queues and streams add replication and log-like behavior.
Use it when
- You need background job queues with acknowledgements and retries.
- You need flexible routing by keys or patterns.
- You integrate systems through AMQP or MQTT.
- You need priorities, message TTL and dead-letter queues.
Avoid it when
- You need a long-retention event log with replay at millions of messages per second: Kafka fits better.
- You need strict global ordering of all messages.
Key concepts
- Exchange
- The entry point for messages that routes them to queues by type and key.
- Queue / binding
- A queue stores messages, a binding links it to an exchange by a key.
- Acknowledgement
- The consumer confirms processing, otherwise the broker delivers the message again.
- Dead-letter queue
- A queue for messages that were rejected, expired or exhausted their retries.
- Quorum queue
- Replicated Raft-based queues that survive a node failure.
Example
Cheat sheet
| Command or code | What it does |
|---|---|
docker run -d -p 5672:5672 -p 15672:15672 rabbitmq:management | Run with the management UI on port 15672 |
rabbitmqctl list_queues | List queues and message counts |
rabbitmqctl list_exchanges | List exchanges |
rabbitmqctl list_bindings | Bindings between exchanges and queues |
rabbitmqctl list_consumers | Connected consumers |
rabbitmqctl add_user app secret | Create a user |
rabbitmqctl set_permissions -p / app ".*" ".*" ".*" | Grant permissions on a vhost |
rabbitmqctl purge_queue jobs | Purge a queue |
rabbitmq-plugins enable rabbitmq_management | Enable the management UI plugin |
rabbitmqctl status | Node status |
How to learn it
- Understand the AMQP model: producer, exchange, binding, queue, consumer, and why a producer writes to an exchange, not a queue.
- Run RabbitMQ with the management UI in Docker and look at the queues in the browser.
- Build the simplest example: a producer puts a message in a queue, a consumer reads it.
- Learn exchange types: direct, fanout, topic, headers and pick the right one for each case.
- Master acknowledgements,
prefetch, durable queues and persistent messages so you do not lose data. - Add delayed retries and a dead letter exchange for messages that fail processing.
- Build work queue and pub/sub patterns in your app through an official client (amqplib, Pika).
- Read about clusters, quorum queues and streams before running RabbitMQ in production.
Interview questions
1. What exchange types does RabbitMQ have?
Direct routes by exact key, topic by a dotted pattern with wildcards, fanout broadcasts to all bound queues, headers routes by message headers.
2. What is an ack and why is it needed?
It is the consumer's confirmation of processing. If a worker dies without an ack, the broker puts the message back, so the task is not lost. The cost: possible duplicates.
3. How does RabbitMQ differ from Kafka?
RabbitMQ is a smart broker with routing where a message is removed after ack. Kafka is a log where the consumer manages its position and data stays per retention.
Interview questions from the bank
- An audit found 12 thousand messages from the last quarter in the refunds service DLQ that nobody knew about, and the customers never got their money. How should the process around a DLQ work so it does not become a graveyard?MiddleSystem design
- A RabbitMQ consumer keeps restarting, CPU is 100%, the queue is not moving, though most messages are fine. In the code, on any error, channel.nack(msg, false, true) is called. What is this and how do you treat it? And how does a similar situation look in Kafka?MiddleSystem design
- Order events
created,paid,shippedgo through SQS standard. With one consumer everything worked; after scaling to four, some orders getpaidprocessed beforecreatedand the handler crashes. Why, and how do you get ordering within one order without global ordering?MiddleSystem design - The
order.createdevent in RabbitMQ must reach both billing and the email service. A developer publishes to the default exchange with routing keybilling, and nobody gets the emails. And when the billing queue was deleted for maintenance, messages silently disappeared. Explain both problems and fix the topology.MiddleSystem design - A RabbitMQ consumer with automatic ack. Messages are lost during a deploy. They switched to manual ack, and now one consumer takes all 200 thousand messages and crashes with OOM while the others sit idle. One more incident: messages disappeared when the broker restarted. Explain all three problems.MiddleSystem design
- You need retries with delays of 1, 5 and 30 minutes in RabbitMQ, and a parking lot after that. The broker has no simple "retry in 5 minutes" option in classic queues. How do you build this? What are the pitfalls?MiddleSystem design
- You need a broker for two flows: thumbnail generation jobs, and order events read by five services, where a new analytics service wants to reread the last 30 days. For each flow, RabbitMQ or SQS, or Kafka, and why?MiddleSystem design
Comparisons with this technology
Related technologies
- Apache KafkaKafka is a distributed event log for high-throughput data streaming with durable storage.
- NATSNATS is a lightweight, very fast messaging system written in Go with pub/sub, request-reply and the durable JetStream layer.
- Amazon SQSAmazon SQS is a fully managed message queue from AWS that scales automatically and needs no servers.
- Redis StreamsRedis Streams is a Redis data type shaped like an event log with consumer groups, usable as a light message queue.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).