Messaging and streaming
Amazon SQS
Amazon SQS is a fully managed message queue from AWS that scales automatically and needs no servers.
What it is
SQS is one of the oldest AWS services, launched in 2004. You create a queue, producers send messages through the API, and consumers poll, process and delete them.
There are two queue types. Standard gives nearly unlimited throughput and at-least-once delivery without guaranteed order. FIFO guarantees order within a message group and deduplication, but with lower throughput.
After receipt a message becomes invisible for the visibility timeout. If the consumer does not delete it in time, it appears again. Messages that fail after a set number of attempts can go to a dead-letter queue.
Use it when
- You work on AWS and need a queue without administration.
- You handle bursty load: the queue works as a buffer.
- You connect Lambda, EC2 or ECS through async tasks.
Avoid it when
- You need to replay event history: SQS deletes messages after processing.
- You want to avoid lock-in to one cloud provider.
- You need complex routing or pub/sub to many consumers: use SNS or EventBridge for that.
Key concepts
- Visibility timeout
- The time a received message is hidden from other consumers until it is deleted.
- Standard vs FIFO
- Standard is fast without ordering, FIFO keeps order and deduplicates but is slower.
- Dead-letter queue
- A queue for messages that could not be processed within maxReceiveCount attempts.
- Long polling
- A request waits up to 20 seconds for a message, reducing empty requests.
Example
Interview questions
1. What is the visibility timeout?
It is the period during which a received message is invisible to other consumers. If the worker does not delete it in that time, the message returns to the queue. The timeout must exceed processing time.
2. How does Standard differ from FIFO?
Standard has nearly unlimited throughput but no guaranteed order and possible duplicates. FIFO guarantees order within a group and deduplicates, but throughput is limited.
3. How do you handle failing messages?
Configure a dead-letter queue with maxReceiveCount. After several failed attempts the message moves there and can be inspected separately.
Interview questions from the bank
- An audit found 12 thousand messages from the last quarter in the refunds service DLQ that nobody knew about, and the customers never got their money. How should the process around a DLQ work so it does not become a graveyard?MiddleSystem design
- Order events
created,paid,shippedgo through SQS standard. With one consumer everything worked; after scaling to four, some orders getpaidprocessed beforecreatedand the handler crashes. Why, and how do you get ordering within one order without global ordering?MiddleSystem design - An SQS consumer generates PDF reports and emails them. Some customers get the same report two or three times. Generation takes up to 90 seconds, and the visibility timeout is left at the default. Explain what is happening and fix it.MiddleSystem design
- You need a broker for two flows: thumbnail generation jobs, and order events read by five services, where a new analytics service wants to reread the last 30 days. For each flow, RabbitMQ or SQS, or Kafka, and why?MiddleSystem design
Comparisons with this technology
Related technologies
- RabbitMQRabbitMQ is a mature message broker with flexible routing through exchanges, acknowledgements and AMQP support.
- Apache KafkaKafka is a distributed event log for high-throughput data streaming with durable storage.
- AWS LambdaAn AWS service that runs functions in response to events and bills only for execution time.
- AWSThe largest cloud platform, with hundreds of services from virtual machines and storage to databases, queues and ML.
- Google Pub/SubGoogle Pub/Sub is a global managed messaging service with a publish/subscribe model that scales automatically.
Logos belong to their owners and are used for identification only. Icons: Simple Icons (CC0), Devicon (MIT).