Messaging and streaming
Amazon SQS
Amazon SQS is a fully managed message queue from AWS that scales automatically and needs no servers.
What it is
SQS is one of the oldest AWS services, launched in 2004. You create a queue, producers send messages through the API, and consumers poll, process and delete them.
There are two queue types. Standard gives nearly unlimited throughput and at-least-once delivery without guaranteed order. FIFO guarantees order within a message group and deduplication, but with lower throughput.
After receipt a message becomes invisible for the visibility timeout. If the consumer does not delete it in time, it appears again. Messages that fail after a set number of attempts can go to a dead-letter queue.
Use it when
- You work on AWS and need a queue without administration.
- You handle bursty load: the queue works as a buffer.
- You connect Lambda, EC2 or ECS through async tasks.
Avoid it when
- You need to replay event history: SQS deletes messages after processing.
- You want to avoid lock-in to one cloud provider.
- You need complex routing or pub/sub to many consumers: use SNS or EventBridge for that.
Key concepts
- Visibility timeout
- The time a received message is hidden from other consumers until it is deleted.
- Standard vs FIFO
- Standard is fast without ordering, FIFO keeps order and deduplicates but is slower.
- Dead-letter queue
- A queue for messages that could not be processed within maxReceiveCount attempts.
- Long polling
- A request waits up to 20 seconds for a message, reducing empty requests.
Example
Interview questions
1. What is the visibility timeout?
It is the period during which a received message is invisible to other consumers. If the worker does not delete it in that time, the message returns to the queue. The timeout must exceed processing time.
2. How does Standard differ from FIFO?
Standard has nearly unlimited throughput but no guaranteed order and possible duplicates. FIFO guarantees order within a group and deduplicates, but throughput is limited.
3. How do you handle failing messages?
Configure a dead-letter queue with maxReceiveCount. After several failed attempts the message moves there and can be inspected separately.
Interview questions from the bank
- Why do you need a dead letter queue? When does a message end up there in SQS, RabbitMQ, BullMQ and pg-boss? What should the process around a DLQ include so that it does not become a graveyard?MiddleSystem design
- What order do Kafka, RabbitMQ, SQS FIFO and BullMQ guarantee? How does increasing the number of consumers affect order? How do you get order only within a single order (purchase), not globally?MiddleSystem design
- What is a visibility timeout in SQS? What happens if processing takes longer than it? How do you choose the value, especially if the consumer is a Lambda?MiddleSystem design
- What is the fundamental difference between a queue (RabbitMQ, SQS, BullMQ) and a log (Kafka)? What happens to a message after processing? How does this affect scaling, retry of a single message, and replay?MiddleSystem design
Comparisons with this technology
Related technologies
- RabbitMQRabbitMQ is a mature message broker with flexible routing through exchanges, acknowledgements and AMQP support.
- Apache KafkaKafka is a distributed event log for high-throughput data streaming with durable storage.
- AWS LambdaAn AWS service that runs functions in response to events and bills only for execution time.
- AWSThe largest cloud platform, with hundreds of services from virtual machines and storage to databases, queues and ML.
- Google Pub/SubGoogle Pub/Sub is a global managed messaging service with a publish/subscribe model that scales automatically.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).