Message queue

An author uploads a 2 GB video to ShortLesson. Transcoding into several qualities takes minutes, and you cannot keep an HTTP request open that long. The API puts a TranscodeVideo {id: 42} message on a queue and answers «processing» in 50 ms, and a worker picks up the job when it has free capacity.

A queue also smooths spikes. After a newsletter to 100 thousand people the API may accept 5 thousand signups a minute, and workers calmly send welcome emails over 10 minutes without exceeding the mail provider's limits.

A classic queue (RabbitMQ, Amazon SQS) hands each message to one consumer and deletes it after acknowledgment (ack). It fits tasks: «do this once».

A log (Kafka, Redis Streams, Pulsar) keeps messages for a set time, and each consumer group reads from its own position (offset). Several independent systems can read the same events and even replay history. It fits event streams.

If a worker crashes mid task, a message without an ack returns to the queue (in SQS after the visibility timeout). So the same job can run twice, and the handler must be idempotent: check whether the video is already processed before doing the work.

A message that fails again and again (a broken file) moves to a dead letter queue after N attempts (say, 5). A human looks at it there, and the main queue is not blocked. Use exponential backoff between attempts: 1 s, 2 s, 4 s, 8 s.

Watch queue depth and the age of the oldest message. A queue that has been growing for an hour means too few workers or stuck ones.