Messaging and streaming

Google Pub/Sub

Google Pub/Sub is a global managed messaging service with a publish/subscribe model that scales automatically.

Official site

What it is

Pub/Sub is a Google Cloud managed service that became generally available in 2015. You create a topic and then one or more subscriptions, and each subscription gets its own copy of every message.

Subscriptions can be pull (the consumer fetches), push (Pub/Sub calls your HTTPS endpoint) or export straight to BigQuery or Cloud Storage. Messages are kept until acknowledged, up to 7 days by default, longer if configured.

The service needs no cluster management and scales on its own. For strict ordering you can enable ordering keys, and for failing messages there are dead-letter topics.

Use it when

  • You work on Google Cloud and need a managed event bus.
  • You must fan one event out to several independent consumers.
  • You want to stream data into BigQuery with no own infrastructure.
  • You need a global service with minimal administration.

Avoid it when

  • You want to avoid lock-in to Google Cloud.
  • You need strong replay and compaction features like Kafka has.

Key concepts

Topic / subscription
A topic receives messages, each subscription keeps its own queue of copies for its consumer.
Pull / push
Pull: the consumer fetches messages. Push: the service calls an HTTPS endpoint itself.
Acknowledgement deadline
The time to acknowledge: without an ack the message is redelivered.
Ordering key
Messages with the same key are delivered in publish order.

Example

A topic with a subscription, publishing and pulling a message in gcloud.

Interview questions

  1. 1. What is the difference between a topic and a subscription?

    A topic is the publish point. A subscription is a separate queue of copies for a specific consumer or group. Several subscriptions on one topic give fan-out: each gets every message.

  2. 2. What delivery guarantee does Pub/Sub give?

    At-least-once by default, so duplicates are possible and consumers must be idempotent. There is also an exactly-once mode for pull subscriptions within a region.

  3. 3. When push, when pull?

    Push suits serverless (Cloud Run, Functions) because the service calls the endpoint itself. Pull is better for long-lived workers that control their own consumption pace.

Related technologies

Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).

Explore the platform