Messaging and streaming
Apache Kafka
Kafka is a distributed event log for high-throughput data streaming with durable storage.
What it is
Kafka was built at LinkedIn, open sourced in 2011 and later became an Apache project. It is not a classic queue but a distributed log: messages are appended to a topic and kept for a set time or size, even after they are read.
A topic is split into partitions. Order is guaranteed within a partition, and consumers in one consumer group share partitions and scale in parallel. Each consumer tracks its own position (offset) and can re-read history.
Kafka is used for event streams, log and metric collection, CDC from databases, microservice integration and event sourcing. New clusters run in KRaft mode without ZooKeeper.
Use it when
- You need high throughput: hundreds of thousands of messages per second.
- Events must be stored and re-read by several independent consumers.
- You build stream processing, CDC or event sourcing.
- You need strict ordering of events per key.
Avoid it when
- You need a simple task queue for a few workers: RabbitMQ or SQS are simpler.
- Volume is small and the team cannot operate a cluster.
- You need message routing by complex rules or priorities.
Key concepts
- Topic / partition
- A topic is a category of events, a partition is an ordered slice of it for parallelism.
- Offset
- The sequence number of a message in a partition, which tells a consumer where it stopped.
- Consumer group
- A group of consumers shares a topic's partitions: each partition goes to one group member.
- Replication factor
- How many copies of each partition are stored on different brokers.
- Retention / log compaction
- Storage policy: by time, by size, or only the latest value per key.
Example
Cheat sheet
| Command or code | What it does |
|---|---|
kafka-topics.sh --create --topic orders --partitions 3 --bootstrap-server localhost:9092 | Create a topic with 3 partitions |
kafka-topics.sh --list --bootstrap-server localhost:9092 | List topics |
kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092 | Topic details: partitions and replicas |
kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092 | Send messages from the console |
kafka-console-consumer.sh --topic orders --from-beginning --bootstrap-server localhost:9092 | Read a topic from the beginning |
kafka-console-consumer.sh --topic orders --group billing --bootstrap-server localhost:9092 | Read as a consumer group member |
kafka-consumer-groups.sh --describe --group billing --bootstrap-server localhost:9092 | Group lag per partition |
kafka-consumer-groups.sh --group billing --topic orders --reset-offsets --to-earliest --execute --bootstrap-server localhost:9092 | Reset the group offset to the beginning |
kafka-topics.sh --delete --topic orders --bootstrap-server localhost:9092 | Delete a topic |
How to learn it
- Understand the model: topic, partition, offset, producer, consumer and consumer group, and how it differs from a queue.
- Run Kafka locally (KRaft mode, no ZooKeeper) in Docker and use the CLI scripts.
- Create a topic, send messages from the console producer and read them with a consumer.
- Write a producer and consumer in your language (kafkajs, confluent, the Java client) and watch offsets work.
- Learn message keys, ordering within a partition and how partitions are spread across a consumer group.
- Study delivery guarantees: acks, idempotent producer, at least once, exactly once, offset commits.
- Add error handling, retries, a dead letter topic and consumer lag monitoring.
- Read about Kafka Connect, Schema Registry and Streams for integrations and stream processing.
Interview questions
1. How does Kafka differ from a regular queue?
In a queue a message disappears once consumed. In Kafka it stays in the log per the retention policy, and several consumers read it independently with their own offsets.
2. How does Kafka guarantee ordering?
Only within a single partition. To keep events of one entity ordered, write them with the same key so they land in the same partition.
3. What are delivery semantics?
At-most-once can lose messages, at-least-once can duplicate them, exactly-once is reached with an idempotent producer and transactions. In practice you make consumers idempotent.
4. How many consumers can read a partition?
Within one consumer group, just one. So the partition count caps the group's parallelism. Another group reads the same data separately.
Interview questions from the bank
- A RabbitMQ consumer keeps restarting, CPU is 100%, the queue is not moving, though most messages are fine. In the code, on any error, channel.nack(msg, false, true) is called. What is this and how do you treat it? And how does a similar situation look in Kafka?MiddleSystem design
- Order events
created,paid,shippedgo through SQS standard. With one consumer everything worked; after scaling to four, some orders getpaidprocessed beforecreatedand the handler crashes. Why, and how do you get ordering within one order without global ordering?MiddleSystem design - A topic has 6 partitions and lag is growing. The team scaled the consumer deployment from 6 to 12 pods, but lag did not drop and half the pods do nothing. Why, what will actually help, and what happens when another service starts reading the same topic?MiddleSystem design
- You need a broker for two flows: thumbnail generation jobs, and order events read by five services, where a new analytics service wants to reread the last 30 days. For each flow, RabbitMQ or SQS, or Kafka, and why?MiddleSystem design
- At peak 150,000 events per second arrive and ClickHouse can only accept 100,000, consumer lag in Kafka grows by millions, and meanwhile one ingest instance crashes with OOM because the producer buffers in memory. Describe backpressure from the storage to the client: what each layer does and what you deliberately drop.SeniorSystem design
- The orders service keeps a copy of a product's name and price, received from a ProductUpdated event from catalog via Kafka. After an incident some cards show an old price even though catalog has the new one. The consumer logs are clean. Name three causes of how this happens and how to build the copy so that it converges on its own.SeniorSystem design
- A vendor promises "exactly-once delivery". What do you ask? What do Kafka transactions really give? Why does that not solve the double-charge problem in Stripe?SeniorSystem design
- A topic with replication.factor=3, min.insync.replicas=1, and a producer with acks=1. After a broker failure, messages disappeared that the producer got a success for. How exactly did this happen? What configuration would you set and what does it cost when one and two brokers fail?SeniorSystem design
Comparisons with this technology
Related technologies
- RabbitMQRabbitMQ is a mature message broker with flexible routing through exchanges, acknowledgements and AMQP support.
- Apache PulsarPulsar is a distributed messaging and streaming platform that separates brokers from storage and has built-in multi-tenancy.
- NATSNATS is a lightweight, very fast messaging system written in Go with pub/sub, request-reply and the durable JetStream layer.
- Redis StreamsRedis Streams is a Redis data type shaped like an event log with consumer groups, usable as a light message queue.
- Amazon SQSAmazon SQS is a fully managed message queue from AWS that scales automatically and needs no servers.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).