Messaging and streaming
RabbitMQ
RabbitMQ is a mature message broker with flexible routing through exchanges, acknowledgements and AMQP support.
What it is
RabbitMQ appeared in 2007 at Rabbit Technologies, then moved to VMware and is now developed by Broadcom. It is a broker written in Erlang that implements AMQP 0-9-1 and also supports MQTT, STOMP and AMQP 1.0.
A producer sends a message to an exchange, which routes it to queues by rules (direct, topic, fanout, headers). Consumers take messages from queues and acknowledge processing (ack), otherwise the message returns.
RabbitMQ fits background jobs, RPC and service integration. Newer quorum queues and streams add replication and log-like behavior.
Use it when
- You need background job queues with acknowledgements and retries.
- You need flexible routing by keys or patterns.
- You integrate systems through AMQP or MQTT.
- You need priorities, message TTL and dead-letter queues.
Avoid it when
- You need a long-retention event log with replay at millions of messages per second: Kafka fits better.
- You need strict global ordering of all messages.
Key concepts
- Exchange
- The entry point for messages that routes them to queues by type and key.
- Queue / binding
- A queue stores messages, a binding links it to an exchange by a key.
- Acknowledgement
- The consumer confirms processing, otherwise the broker delivers the message again.
- Dead-letter queue
- A queue for messages that were rejected, expired or exhausted their retries.
- Quorum queue
- Replicated Raft-based queues that survive a node failure.
Example
Cheat sheet
| Command or code | What it does |
|---|---|
docker run -d -p 5672:5672 -p 15672:15672 rabbitmq:management | Run with the management UI on port 15672 |
rabbitmqctl list_queues | List queues and message counts |
rabbitmqctl list_exchanges | List exchanges |
rabbitmqctl list_bindings | Bindings between exchanges and queues |
rabbitmqctl list_consumers | Connected consumers |
rabbitmqctl add_user app secret | Create a user |
rabbitmqctl set_permissions -p / app ".*" ".*" ".*" | Grant permissions on a vhost |
rabbitmqctl purge_queue jobs | Purge a queue |
rabbitmq-plugins enable rabbitmq_management | Enable the management UI plugin |
rabbitmqctl status | Node status |
How to learn it
- Understand the AMQP model: producer, exchange, binding, queue, consumer, and why a producer writes to an exchange, not a queue.
- Run RabbitMQ with the management UI in Docker and look at the queues in the browser.
- Build the simplest example: a producer puts a message in a queue, a consumer reads it.
- Learn exchange types: direct, fanout, topic, headers and pick the right one for each case.
- Master acknowledgements,
prefetch, durable queues and persistent messages so you do not lose data. - Add delayed retries and a dead letter exchange for messages that fail processing.
- Build work queue and pub/sub patterns in your app through an official client (amqplib, Pika).
- Read about clusters, quorum queues and streams before running RabbitMQ in production.
Interview questions
1. What exchange types does RabbitMQ have?
Direct routes by exact key, topic by a dotted pattern with wildcards, fanout broadcasts to all bound queues, headers routes by message headers.
2. What is an ack and why is it needed?
It is the consumer's confirmation of processing. If a worker dies without an ack, the broker puts the message back, so the task is not lost. The cost: possible duplicates.
3. How does RabbitMQ differ from Kafka?
RabbitMQ is a smart broker with routing where a message is removed after ack. Kafka is a log where the consumer manages its position and data stays per retention.
Interview questions from the bank
- Why do you need a dead letter queue? When does a message end up there in SQS, RabbitMQ, BullMQ and pg-boss? What should the process around a DLQ include so that it does not become a graveyard?MiddleSystem design
- What order do Kafka, RabbitMQ, SQS FIFO and BullMQ guarantee? How does increasing the number of consumers affect order? How do you get order only within a single order (purchase), not globally?MiddleSystem design
- Explain the RabbitMQ model: producer, exchange, binding, queue. How do direct, fanout, topic and headers differ? What happens to a message if there is no suitable queue? Which type do you pick for "deliver order.created to billing and to mail"?MiddleSystem design
- What is the fundamental difference between a queue (RabbitMQ, SQS, BullMQ) and a log (Kafka)? What happens to a message after processing? How does this affect scaling, retry of a single message, and replay?MiddleSystem design
- A RabbitMQ consumer keeps restarting, CPU is 100%, the queue is not moving, though most messages are fine. In the code, on any error, channel.nack(msg, false, true) is called. What is this and how do you treat it? And how does a similar situation look in Kafka?SeniorSystem design
- A RabbitMQ consumer with automatic ack. Messages are lost during a deploy. They switched to manual ack, and now one consumer takes all 200 thousand messages and crashes with OOM while the others sit idle. One more incident: messages disappeared when the broker restarted. Explain all three problems.SeniorSystem design
- You need retries with delays of 1, 5 and 30 minutes in RabbitMQ, and a parking lot after that. The broker has no simple "retry in 5 minutes" option in classic queues. How do you build this? What are the pitfalls?SeniorSystem design
Comparisons with this technology
Related technologies
- Apache KafkaKafka is a distributed event log for high-throughput data streaming with durable storage.
- NATSNATS is a lightweight, very fast messaging system written in Go with pub/sub, request-reply and the durable JetStream layer.
- Amazon SQSAmazon SQS is a fully managed message queue from AWS that scales automatically and needs no servers.
- Redis StreamsRedis Streams is a Redis data type shaped like an event log with consumer groups, usable as a light message queue.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).