Messaging and streaming

Redis Streams

Redis Streams is a Redis data type shaped like an event log with consumer groups, usable as a light message queue.

Official siteRepository

What it is

Streams appeared in Redis 5.0 in 2018. It is an append-only log: each entry has a unique time-based id and a set of field-value pairs. XADD appends entries and XREAD reads them.

With consumer groups (XREADGROUP) several workers share a stream, and XACK acknowledgements allow redelivery if a worker dies. It resembles Kafka, but at a small scale.

Streams are handy when Redis is already in your stack and you do not want a separate broker. The limits: data sits in memory, so streams are trimmed with MAXLEN, and durability depends on Redis replication settings.

Use it when

  • Redis is already there and you need a simple queue or event stream.
  • Volume is moderate and data fits in memory.
  • You need consumer groups and redelivery without a new broker.

Avoid it when

  • You need long retention of large volumes of events: Kafka or Pulsar fit better.
  • Losing messages on failure is unacceptable and Redis is not tuned for strong durability.
  • You need complex routing and a connector ecosystem.

Key concepts

Entry id
An id of the form milliseconds-sequence that grows and defines entry order.
Consumer group
A group of workers shares a stream: each entry goes to one consumer of the group.
Pending entries list
The list of delivered but unacknowledged entries, used for reprocessing.
Trimming
Trimming a stream by length or id so memory does not run out.

Example

Appending an event, creating a group, reading as a worker and acknowledging.

Interview questions

  1. 1. How do Redis Streams differ from Redis Pub/Sub?

    Pub/Sub does not store messages: with no subscriber, they are lost. Streams store entries, have consumer groups and acknowledgements, so they suit reliable queues.

  2. 2. What if a worker does not acknowledge an entry?

    The entry stays in the pending list. Another worker can take it with XAUTOCLAIM after a timeout and process it again.

  3. 3. When do Redis Streams replace Kafka?

    For small streams, when data fits in memory and Redis is already there. For large volumes, long retention and stream processing ecosystem, Kafka remains.

Related technologies

Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).

Explore the platform