Messaging and streaming

Apache Pulsar

Pulsar is a distributed messaging and streaming platform that separates brokers from storage and has built-in multi-tenancy.

Official siteRepository

What it is

Pulsar was built at Yahoo, open sourced in 2016 and became an Apache top-level project in 2018. Its key idea: brokers do not store data, they only serve clients, while data lives in Apache BookKeeper.

Because of that, brokers scale independently from storage, and adding a node needs no partition rebalancing as in Kafka. Pulsar supports both queue semantics (Shared subscriptions) and streaming (Failover, Key_Shared).

It also has built-in multi-tenancy (tenants and namespaces), geo-replication between data centers and tiered storage to offload old data to object storage.

Use it when

  • You need both queues and streaming in one system.
  • Many teams share one cluster and need multi-tenancy.
  • You need geo-replication across regions.
  • Data accumulates for long and old data must move to cheap storage.

Avoid it when

  • You want the biggest ecosystem and the easiest hiring: Kafka is more widely supported.
  • The team is small and three components (Pulsar, BookKeeper, ZooKeeper or etcd) are too hard to operate.

Key concepts

Broker / BookKeeper
The broker is stateless and serves clients, BookKeeper durably stores the message log.
Subscription types
Exclusive, Failover, Shared and Key_Shared define how messages are split among consumers.
Tenant / namespace
Two-level isolation of teams and topics with their own quotas and policies.
Tiered storage
Old segments are offloaded automatically to object storage (for example S3).

Example

Creating a topic, producing a message and consuming with subscription my-sub.

Interview questions

  1. 1. What is the architectural difference between Pulsar and Kafka?

    In Kafka a broker stores the data of its partitions. In Pulsar brokers are stateless and data lives in BookKeeper, so compute and storage scale separately and rebalancing is faster.

  2. 2. Which subscription types does Pulsar support?

    Exclusive (one consumer), Failover (a standby consumer), Shared (spread across all, unordered) and Key_Shared (spread by key, keeping order per key).

  3. 3. What is tiered storage?

    Automatic offloading of old log segments to cheap object storage while the data stays readable.

Related technologies

Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).

Explore the platform