Question 1

You are given 45 minutes and the task "design a notification service". How do you split the time and where do you start? Which mistakes most often sink candidates in system design?

Accepted Answer

The first 5 minutes go to requirements. Functional: which channels (push, email, in-app), whether history is needed, whether there is scheduling. Non-functional: how many users, acceptable delivery delay, whether a notification can be lost, whether order matters. I also say explicitly what is out of scope: for example, we do not design email templates. Then 5 minutes for estimation: DAU, notifications per user per day, average and peak QPS, storage per year. The numbers decide the architecture: 50 notifications per second fit in one Postgres with BullMQ, while 50 thousand already need a partitioned broker. Then 10 minutes for the API and a high-level design: who generates the event, the queue, a worker per channel, the status store, the APNs and FCM providers.

Question 2

Estimate on a napkin: 10 million DAU, each makes 50 requests per day, 10% of requests are writes of 1 KB, the average response is 5 KB. Compute average and peak QPS, data volume per year and outgoing traffic. How many Node processes and servers do you need?

Accepted Answer

Requests: 10 million × 50 = 500 million per day, so on average about 5,800 rps, and with a threefold peak about 17 thousand. Writes at 10%: 580 on average and 1,700 at peak. Volume: 50 million records of 1 KB is 50 GB per day, about 18 TB per year, and with indexes and two replicas over 50 TB. One database without partitioning and archiving of old data will not last a year, and backups of that size must be planned separately. Outgoing traffic: 17 thousand rps × 5 KB is about 85 MB/s, so hundreds of Mbit/s at peak. For process count I take the measured rps of one Node process (say 1,000 rps) at 60% utilization: about 30 processes, which is 2 or 3 servers of 16 cores with N+1 headroom.

Question 3

An endpoint assembles a user profile: 40 sequential GETs to Redis and 15 sequential queries to Postgres. Locally the p50 is 8 ms. In production Redis and Postgres moved to a neighboring data center, and the p50 became 400 ms, even though CPU is low. Explain it using the numbers every engineer should know, and fix it.

Accepted Answer

The classic table of orders of magnitude: a memory access is about 100 ns, a round trip within one data center is about 0.5 ms, between data centers in one region a few milliseconds, between continents about 150 ms. Reading 1 MB from memory takes hundreds of microseconds, from disk tens of milliseconds. Here there are 55 sequential calls. Locally the RTT is tens of microseconds, hence 8 ms. Between data centers the RTT is, say, 7 ms: 55 × 7 ≈ 385 ms, plus the work itself. CPU is low because the process spends almost all its time waiting on the network. Vertical scaling will do nothing here.

Question 4

A WebSocket chat behind Nginx. Complaints: connections drop exactly every minute when the chat is quiet. And after every deploy one of three Node processes holds 80% of all connections. Explain both effects and fix the config.

Accepted Answer

The first effect. proxy_read_timeout defaults to 60 seconds: if nothing arrives from the backend in that time, Nginx closes the connection. A quiet chat looks exactly like that. I fix it from both sides: the server sends a ping every 25 or 30 seconds (in ws via ws.ping, Socket.IO does this itself through pingInterval), and I set proxy_read_timeout and proxy_send_timeout to, for example, 1h. The ping is also needed on its own: it finds dead connections when a client vanished without a FIN. The config also lacks proxy_http_version 1.1; without it Nginx talks to the backend over HTTP/1.0 and the upgrade does not work. During a rolling deploy the processes restart one at a time. The clients of the first process reconnect to the two live ones, then the clients of the second to the live ones, and so on.

Question 5

The dashboard shows an average response time of 45 ms, yet users complain about slowness. Why does the average lie here? What do p50, p95 and p99 mean? Why is a page that makes 50 calls to backends slow more often than the p99 of each backend suggests?

Accepted Answer

Latency is not normally distributed. Most requests are fast, but there is a long tail: a GC pause, a cold cache, waiting for a connection from the pool, a database lock. An average of 45 ms can mean that 98% of requests take 20 ms and 2% take 1.3 seconds. A user who clicks 50 times per session will almost certainly see those 1.3 seconds. pN means that N% of requests are faster than this value. p50 is the median, the typical experience. p95 and p99 describe the tail: p99 is every hundredth request. For an active user every hundredth request happens several times per session, so SLOs are written on p95 or p99, not on the average. If a page makes 50 independent calls and waits for all of them, it is slow when at least one of them hit the tail.

Question 6

An API handles 3000 rps with an average latency of 40 ms, and each request makes two Postgres queries of 3 ms. How many requests are in flight at once? What total connection pool do you need? What happens when database queries slow down to 100 ms each?

Accepted Answer

Little's law: the average number of requests in the system equals the arrival rate times the average time spent, L = λ × W. Here 3000 × 0.04 = 120 requests in flight at once. The law holds for any stable system, regardless of distribution. The same law for the database: each request holds a connection for 2 × 3 = 6 ms, so on average 3000 × 0.006 = 18 connections are busy. I size the pool with headroom for unevenness, for example 40 in total. With 12 processes that is 3 or 4 per process, through PgBouncer in transaction mode. More connections will not make it faster: in Postgres each one is a process, and beyond a few dozen active ones on a 16-core server they compete with each other.

Question 7

A sale: traffic is three times the API capacity. A per client rate limiter is in place and no client exceeds its quota, yet latency is 20 seconds for everyone and there are fewer successful responses per second than on a normal day. Why, and what do you add?

Accepted Answer

Rate limiting protects against one client exceeding its quota, but here the overload is aggregate and the limiter cannot see it. The service accepts everything, queues grow, each request waits longer than the client timeout, so the server spends resources on responses nobody is waiting for, and clients retry on top: goodput falls. You need load shedding: cap concurrent requests or queue length and reject the excess immediately with a cheap 503 and Retry-After, preferably by priority (payments matter more than recommendations) and taking the request deadline into account. 429 means 'you, this client, exceeded your quota', 503 means 'the service is overloaded', and clients should react to both with backoff, not an instant retry.

Question 8

Case: a URL shortener. 100 million new links per month, reads are 100 times more than writes, links are stored for 5 years. Which requirements do you clarify first? Compute write QPS and redirect QPS, average and peak, storage volume, and how many base62 characters the code needs.

Accepted Answer

Functional: create a short link from a long one, redirect, optionally a custom alias and an expiry, click statistics, deletion by the owner. Non-functional: redirect is fast (p99 under 20 ms in the data center) and almost always available, codes must not be predictable if links are private. Redirect matters more than analytics: if statistics lag by a minute, nobody notices, but if the redirect goes down, all emails and QR codes already sent break. Writes: 100,000,000 / 2,592,000 s ≈ 40 per second, peak ×5 ≈ 200. Reads are 100 times more: ≈ 4000 rps on average, 15,000 or 20,000 at peak. For Node that is several processes, not a cluster, but 20,000 point SELECTs per second without a cache will already be felt by the database. Over 5 years 100 million × 60 = 6 billion records.

Question 9

Case: a social network home feed. 20 million DAU, each opens the feed 10 times per day and publishes 1 post per 5 days, with 200 follows on average. Clarify the requirements and compute read and write QPS, how many inserts into feeds fan-out on write produces, and how much memory the feeds will take in Redis.

Accepted Answer

Requirements: a chronological or ranked feed, acceptable delay before a post appears (a few seconds is fine), deletion, infinite scroll, and reads matter more than instant appearance. Reads: 200 million per day is about 2300 rps, 8000 or 9000 at peak. Posts: 4 million per day, about 46 per second. Fan-out on write spreads each post into 200 feeds: 800 million inserts per day, about 9300 per second on average and 30,000 at peak, and an account with 10 million followers produces 10 million inserts with a single post. A feed holds only post_id and score, trimmed to 500 items, which is about 30 KB per user. For all 20 million that is 600 GB, so feeds are kept only for active users, about 250 GB across several Redis shards. The main problem here is not reads but the multiplication of writes and the tail of large accounts.

Question 10

After creating a listing, the user is redirected to the My listings page, and the new listing is not there: the Elasticsearch projection lags by 1 to 3 seconds. Users press again and create duplicates. What are the fix options, and which would you choose?

Accepted Answer

The problem is that the UI pretends to be a synchronous system on top of an asynchronous one. There are several options, usually combined. The command returns the created object, and the client adds it to the list locally. It is the cheapest and sufficient in most cases. The command returns a position: an outbox record id or a version. The projector stores the last applied position. The list request passes min_position, and the API waits up to 2 seconds for the projection to reach it, or returns the response with a stale flag.

Question 11

The tech lead wants to move the whole new orders service to event sourcing because there will be an audit and you can rewind time. What do you say? What real problems will the team get in the first six months?

Accepted Answer

First I ask which problem we are solving. Audit and change history can be had much cheaper: an audit_log table, a change trigger, or domain events in an outbox next to ordinary state. Rewinding time is needed rarely, and when it is, the change history of a single order is enough. What the team gets in six months. Every list and search needs a projection, because you cannot run SELECT * FROM orders WHERE status = 'paid' on a log. That means projectors, their lag, read-your-writes in the UI, lag monitoring. The event schema changes with every feature, so upcasters and versioning rules are needed from the first month.

Question 12

A Node api service calls recommendations over gRPC through a regular Kubernetes Service. Recommendations was scaled from 2 to 8 pods, but the load stayed on the two old ones, and the new ones are almost idle. Why? How do you fix it?

Accepted Answer

kube-proxy balances connections, not requests: the pod is chosen once when the TCP connection is established. gRPC runs over HTTP/2 and multiplexes all calls into one long connection. The api client connected when there were 2 pods, and since then all its calls go there. The new 6 pods stay empty until the connection breaks. With HTTP/1.1 and keep-alive the same happens, but there are more connections, so it is much less noticeable. Option one: client-side balancing. A headless Service with clusterIP: None, DNS returns the IPs of all pods. The client keeps a connection to each pod and distributes calls.

Question 13

Services already have e2e tests on staging that run all services together for 40 minutes and are often flaky. What does consumer-driven contract testing give and how does Pact work? What does it not verify?

Accepted Answer

E2E tests on a shared staging verify everything at once, so they are slow, flaky and localize errors poorly: the checkout test fails, and anyone of 8 services could have broken it. Contract testing verifies each consumer and provider pair separately. The consumer, for example orders, describes the interaction in its test: the state user 42 exists, the request GET /users/42, the expected response with id and email fields, where values are set with a like matcher, meaning the type is checked, not the concrete value. Pact starts a mock server, the real HTTP client of orders talks to it, and if the client makes exactly such requests, a pact file in JSON is generated. It is published to a Pact Broker with a version, usually the git sha. The provider users, in its own CI, starts itself, takes from the broker the contracts where it is the provider, prepares data through provider states (for example the handler for user 42 exists inserts a row into the test database), and replays the requests, comparing responses with expectations.

Question 14

During a migration the team did this: when a product changes, the monolith writes to its own database and immediately makes an HTTP POST to the new catalog service. A month later 0.3% of products in the new service have different prices. Why does this happen when there are almost no errors? How do you fix both the discrepancy and the approach itself?

Accepted Answer

This is a dual write: two writes to two systems without a shared transaction, so any failure between them causes a discrepancy. The monolith committed the price, then the POST failed on a timeout: the catalog has the old price. Or the POST went through and the monolith transaction rolled back: the catalog has a price that never existed. There are races too: two admins changed the price almost at once, in the monolith database B committed last, but the requests arrived in the order B, A. A timeout also does not mean a failure, and a retry without a version can overwrite newer data with older. The approach itself must be fixed: sync from a log, that is CDC on the WAL or an outbox in the same transaction, with a product_id key and a conditional write by version. Existing discrepancies are healed by a reconciliation job that compares both databases by product_id and rewrites the diverging ones from the source of truth.

Question 15

A consumer sends a welcome email and then commits the offset. After a pod crash some users got two emails. Another team commits the offset before processing and loses events now and then. Which delivery guarantee does each option give, and what do you choose for emails and what for crediting bonuses?

Accepted Answer

Committing after processing gives at least once: if the process dies between sending the email and committing, the message comes again after the restart, hence duplicates. Committing before processing gives at most once: the offset has moved but the work is not done, so the event is lost. The guarantee is set not by the broker but by the order of 'did the work' and 'acknowledged', plus whether the side effect is idempotent. For emails at least once with deduplication by event id is enough, since a duplicate is annoying but not critical. For bonuses you need at least once plus an idempotent write: a unique key on the event id in the same transaction as the credit, so a retry changes nothing and the result is effectively exactly once. At most once fits where a loss is cheaper than a duplicate: metrics, telemetry, an 'online' status.

Question 16

Redis restarted for 20 seconds. 1000 workers retry the connection every second, and when Redis comes back it falls over again under the wave of connections, several times in a row. What do you change in the retry logic? Write down the delay formula.

Accepted Answer

A fixed interval synchronises the clients: they all lost the connection at once and retry at once, so the server gets waves of 1000 connections exactly when it is weakest. Exponential backoff spreads attempts over time, and jitter desynchronises clients from each other; without it the waves remain, just rarer. The full jitter formula: sleep = random(0, min(cap, base * 2 ** attempt)), where cap limits the longest pause, say 30 seconds, so a client does not wait an hour after a long outage. On top of that, limit the number of attempts or set an overall deadline, and retry only transient errors.

Question 17

An integration with a payment partner retries any error 5 times with a 1 second pause: 400 validation, 401, 429, a timeout and 503. The call is POST /charges without an idempotency key and with a 30 second timeout. When the partner slowed down, double charges appeared and our service went down. What is wrong and how do you rewrite it?

Accepted Answer

Only transient errors should be retried: timeouts, reset connections, 502, 503, 504, and 429 honouring Retry-After; 400, 401, 403, 404 and 422 are permanent, and a retry only adds load. A timeout on a non idempotent POST is ambiguous: the request may have gone through, so a retry without an idempotency key causes a double charge, and you need an Idempotency-Key that is the same for every attempt. A fixed pause without jitter synchronises retries, and 5 attempts of 30 seconds each hold a request and a connection for up to 2.5 minutes, so the pool and memory run out. You need a short timeout, an overall deadline, backoff with jitter and a circuit breaker or retry budget, so a degraded partner does not get multiplied traffic.

Question 18

An audit found 12 thousand messages from the last quarter in the refunds service DLQ that nobody knew about, and the customers never got their money. How should the process around a DLQ work so it does not become a graveyard?

Accepted Answer

A DLQ exists so that a message that failed after N attempts (maxReceiveCount in an SQS redrive policy, a dead letter exchange in RabbitMQ, the failed list after attempts in BullMQ) does not block the queue. But a DLQ without a process just hides failures. You need: an alert on a non zero DLQ depth and on the age of the oldest message, not only on errors in the logs; an owner who triages it within an SLA; the reason, stack, attempt count, time and code version stored with the message; permanent errors (validation, a missing record) separated from transient ones, because permanent ones should not go round in circles. And a tested, rate limited redrive with an idempotent consumer, so reprocessing is safe.

Question 19

A RabbitMQ consumer keeps restarting, CPU is 100%, the queue is not moving, though most messages are fine. In the code, on any error, channel.nack(msg, false, true) is called. What is this and how do you treat it? And how does a similar situation look in Kafka?

Accepted Answer

This is a poison message: one message, for example with broken JSON, fails every time. requeue=true puts it back at the head of the queue, so the consumer instantly receives it again. The RabbitMQ documentation warns about such a requeue/redelivery loop: it eats CPU and network. If the whole process crashes, the connection drops and the message is returned too. The handler must not crash: schema validation before processing, try/catch around it. For permanent errors nack(msg, false, false), so it goes to the DLX. A quorum queue has a delivery-limit: since RabbitMQ 4.0 it defaults to 20, after which the message is dropped or goes to the DLX, so the DLX should be configured. Classic queues do not have this protection.

Question 20

Order events created, paid, shipped go through SQS standard. With one consumer everything worked; after scaling to four, some orders get paid processed before created and the handler crashes. Why, and how do you get ordering within one order without global ordering?

Accepted Answer

SQS standard does not guarantee ordering at all, and several consumers process neighbouring messages in parallel, so with one consumer the order was a coincidence. Global ordering is not needed and is expensive; you need ordering per key: SQS FIFO with MessageGroupId = order id (strict order inside a group, groups processed in parallel), Kafka with the message key = order id (one partition, one consumer in the group), in RabbitMQ a consistent hash exchange or a queue per shard. Even then the handler should tolerate reordering: store a version or sequence number, drop stale events and defer those whose previous state is not there yet, because retries and the DLQ break ordering anyway.

Question 21

An SQS consumer generates PDF reports and emails them. Some customers get the same report two or three times. Generation takes up to 90 seconds, and the visibility timeout is left at the default. Explain what is happening and fix it.

Accepted Answer

A received message is not deleted, it only becomes invisible for the visibility timeout, 30 seconds by default. If processing takes longer and DeleteMessage is not called in time, the message reappears in the queue, another consumer takes it, and the report is generated again. The fix: a visibility timeout with headroom over the real processing time (for Lambda AWS recommends at least six times the function timeout), and for long tasks extend it with ChangeMessageVisibility as a heartbeat. And still make processing idempotent, for example by recording the id of the sent report, because SQS standard is at least once and duplicates can happen even without this bug.

Question 22

A topic has 6 partitions and lag is growing. The team scaled the consumer deployment from 6 to 12 pods, but lag did not drop and half the pods do nothing. Why, what will actually help, and what happens when another service starts reading the same topic?

Accepted Answer

Within one consumer group each partition is assigned to exactly one consumer, so parallelism is capped by the partition count: six partitions mean at most six active consumers, and the rest sit idle waiting for a rebalance. What can help: more partitions (but that changes the key to partition mapping and the ordering for new messages), faster processing per message (batches, fewer synchronous calls), or parallel processing inside the consumer by key, keeping per key order and committing offsets only for processed messages. Another service reads the topic with its own consumer group and its own offsets and gets every message regardless of the first group, because Kafka does not delete what was read.

Question 23

After a consumer restart some events were not processed even though lag showed 0. Another time events were processed twice. The settings are default, and in the code a separate pool processes messages asynchronously. Explain both cases. How do you fix it?

Accepted Answer

enable.auto.commit defaults to true: the offset is committed periodically in the background. It does not know whether your code has finished processing. Loss: pool.submit returns immediately, the handler is considered done, and the offset is committed. The process crashed with jobs in the pool, and after the restart reading continues past them. Lag showed 0 because it is about the committed offset, not finished work. Duplicates: processing completed but the commit had not happened yet. The process crashed, and after the restart the consumer read the same records again.

Question 24

A consumer processes a batch of records for 10 minutes. Rebalances happen constantly, the same batch is processed again and again, and lag grows. The heartbeat is fine meanwhile. What happened and how do you fix it?

Accepted Answer

There are two separate mechanisms. The heartbeat goes in a background thread and proves the process is alive (session.timeout.ms, 45 s by default). max.poll.interval.ms, 300000 ms by default, is the maximum pause between poll() calls. It proves that the consumer is making progress. Processing of 10 minutes exceeds 5 minutes, so the consumer is considered stuck and leaves the group. The partitions go to another one, which starts from the last committed offset, that is, from the same batch. The first one finishes and tries to commit, but the partitions are no longer its own. This is an endless loop, and lag grows.

Question 25

The order.created event in RabbitMQ must reach both billing and the email service. A developer publishes to the default exchange with routing key billing, and nobody gets the emails. And when the billing queue was deleted for maintenance, messages silently disappeared. Explain both problems and fix the topology.

Accepted Answer

The default exchange is a direct exchange where the routing key equals the queue name, so the message lands in exactly the billing queue: the producer effectively knows its consumer, and a second queue cannot be added this way. The right way: the producer publishes to a named exchange, for example topic orders with key order.created, and each service has its own queue bound to the key it needs, so a new consumer is added without touching the producer (fanout if no filtering is needed). The second problem: if no queue matches the route, a message without the mandatory flag is simply dropped. Protection: durable queues that are not deleted for maintenance, an alternate exchange for unroutable messages or mandatory with return handling, and publisher confirms.

Question 26

A RabbitMQ consumer with automatic ack. Messages are lost during a deploy. They switched to manual ack, and now one consumer takes all 200 thousand messages and crashes with OOM while the others sit idle. One more incident: messages disappeared when the broker restarted. Explain all three problems.

Accepted Answer

The broker considers a message processed right after sending it. Everything the consumer holds in memory is lost on a crash. The solution: a manual ack after processing, which is at-least-once. An unacked message is returned to the queue when the connection drops. The value 0 means no limit, so the broker pushes everything it has into the first consumer's socket. It buffers everything in memory. The solution: basic.qos, prefetch 10 to 100 depending on task duration.

Question 27

You need retries with delays of 1, 5 and 30 minutes in RabbitMQ, and a parking lot after that. The broker has no simple "retry in 5 minutes" option in classic queues. How do you build this? What are the pitfalls?

Accepted Answer

The scheme is built on TTL and DLX. The main queue orders has a dead-letter-exchange to a retry exchange. There are three queues without consumers: orders.retry.1m, 5m, 30m, each with its own x-message-ttl and a dead-letter-exchange back to the main exchange. When processing fails, the consumer does a nack without requeue or publishes to the needed retry queue. The message sits for the TTL and then returns to the main queue. The attempt counter: the x-death header, filled in by the broker on dead-lettering, or a custom x-attempt header. After the third attempt the message goes to orders.parking, with no TTL, an alert and manual triage.

Question 28

Kafka broker disks are 90 % full. You cut retention.ms on a 2 TB topic from 7 days to one hour, but disk usage barely moves for hours. Why, and what do you do?

Accepted Answer

Retention deletes data in whole segments, not individual messages, and only closed segments: the active segment being written is not deleted until it rolls by segment.bytes (1 GB by default) or segment.ms (7 days by default). A segment is deleted when its newest record is older than the retention, and the check runs periodically (log.retention.check.interval.ms, 5 minutes). On partitions with slow writes the active segment can hold almost a week of data. What to do: temporarily lower segment.ms or segment.bytes so segments roll, wait for the check, and in future also cap the topic by size (retention.bytes) and have an alert on disk usage.

Question 29

You need a broker for two flows: thumbnail generation jobs, and order events read by five services, where a new analytics service wants to reread the last 30 days. For each flow, RabbitMQ or SQS, or Kafka, and why?

Accepted Answer

Thumbnails are a classic job queue: each job is done by one worker, you need per message acknowledgements, retrying a single job, a DLQ and scaling by just adding workers, so RabbitMQ, SQS or BullMQ. After an ack the message is gone from the queue and cannot be read again. Order events fit Kafka better: it is a log, messages stay after being read for the retention period, each service reads with its own consumer group and offset, and a new service can start from an offset a month back and replay. The cost of a log: parallelism is capped by partitions, retrying one message without blocking the partition is something you build yourself, and Kafka is harder to operate.

Question 30

After 20 minutes of consumer downtime there are 500 thousand jobs in the queue. New ones arrive at 800 per second, one worker handles 50 per second, and there are 20 workers now. How long will it take to drain the queue? How many workers do you need to drain it in 10 minutes? What else should you check before adding workers?

Accepted Answer

Arithmetic. 20 workers at 50 each is 1000 per second. New ones arrive at 800, so the queue shrinks by only 200 per second. 500000 / 200 = 2500 s, about 42 minutes. During all that time new jobs also wait, because they stand at the end of the queue. To drain in 10 minutes (600 s), the net drain must be 500000 / 600, about 833 per second. Together with the inflow of 800 that is 1633 per second, which is 33 workers. Little's law: L = λ W. The number in the system equals the arrival rate times the average time spent.

System design interview questions and answers

Middle questions

1. You are given 45 minutes and the task "design a notification service". How do you split the time and where do you start? Which mistakes most often sink candidates in system design?

2. Estimate on a napkin: 10 million DAU, each makes 50 requests per day, 10% of requests are writes of 1 KB, the average response is 5 KB. Compute average and peak QPS, data volume per year and outgoing traffic. How many Node processes and servers do you need?

4. A WebSocket chat behind Nginx. Complaints: connections drop exactly every minute when the chat is quiet. And after every deploy one of three Node processes holds 80% of all connections. Explain both effects and fix the config.

5. The dashboard shows an average response time of 45 ms, yet users complain about slowness. Why does the average lie here? What do p50, p95 and p99 mean? Why is a page that makes 50 calls to backends slow more often than the p99 of each backend suggests?

6. An API handles 3000 rps with an average latency of 40 ms, and each request makes two Postgres queries of 3 ms. How many requests are in flight at once? What total connection pool do you need? What happens when database queries slow down to 100 ms each?

7. A sale: traffic is three times the API capacity. A per client rate limiter is in place and no client exceeds its quota, yet latency is 20 seconds for everyone and there are fewer successful responses per second than on a normal day. Why, and what do you add?

8. Case: a URL shortener. 100 million new links per month, reads are 100 times more than writes, links are stored for 5 years. Which requirements do you clarify first? Compute write QPS and redirect QPS, average and peak, storage volume, and how many base62 characters the code needs.

10. After creating a listing, the user is redirected to the My listings page, and the new listing is not there: the Elasticsearch projection lags by 1 to 3 seconds. Users press again and create duplicates. What are the fix options, and which would you choose?

11. The tech lead wants to move the whole new orders service to event sourcing because there will be an audit and you can rewind time. What do you say? What real problems will the team get in the first six months?

12. A Node api service calls recommendations over gRPC through a regular Kubernetes Service. Recommendations was scaled from 2 to 8 pods, but the load stayed on the two old ones, and the new ones are almost idle. Why? How do you fix it?

13. Services already have e2e tests on staging that run all services together for 40 minutes and are often flaky. What does consumer-driven contract testing give and how does Pact work? What does it not verify?

16. Redis restarted for 20 seconds. 1000 workers retry the connection every second, and when Redis comes back it falls over again under the wave of connections, several times in a row. What do you change in the retry logic? Write down the delay formula.

18. An audit found 12 thousand messages from the last quarter in the refunds service DLQ that nobody knew about, and the customers never got their money. How should the process around a DLQ work so it does not become a graveyard?

19. A RabbitMQ consumer keeps restarting, CPU is 100%, the queue is not moving, though most messages are fine. In the code, on any error, channel.nack(msg, false, true) is called. What is this and how do you treat it? And how does a similar situation look in Kafka?

20. Order events created, paid, shipped go through SQS standard. With one consumer everything worked; after scaling to four, some orders get paid processed before created and the handler crashes. Why, and how do you get ordering within one order without global ordering?

21. An SQS consumer generates PDF reports and emails them. Some customers get the same report two or three times. Generation takes up to 90 seconds, and the visibility timeout is left at the default. Explain what is happening and fix it.

22. A topic has 6 partitions and lag is growing. The team scaled the consumer deployment from 6 to 12 pods, but lag did not drop and half the pods do nothing. Why, what will actually help, and what happens when another service starts reading the same topic?

23. After a consumer restart some events were not processed even though lag showed 0. Another time events were processed twice. The settings are default, and in the code a separate pool processes messages asynchronously. Explain both cases. How do you fix it?

24. A consumer processes a batch of records for 10 minutes. Rebalances happen constantly, the same batch is processed again and again, and lag grows. The heartbeat is fine meanwhile. What happened and how do you fix it?

27. You need retries with delays of 1, 5 and 30 minutes in RabbitMQ, and a parking lot after that. The broker has no simple "retry in 5 minutes" option in classic queues. How do you build this? What are the pitfalls?

28. Kafka broker disks are 90 % full. You cut retention.ms on a 2 TB topic from 7 days to one hour, but disk usage barely moves for hours. Why, and what do you do?

29. You need a broker for two flows: thumbnail generation jobs, and order events read by five services, where a new analytics service wants to reread the last 30 days. For each flow, RabbitMQ or SQS, or Kafka, and why?

31. A Redis Streams consumer crashed mid processing and restarted. It reads with XREADGROUP GROUP g c1 STREAMS orders >, and the 300 messages it was processing at the moment of the crash stay unprocessed. Why, and how do you pick them up?

Senior questions

1. Right now the peak is 2000 rps on three servers with 16 cores each, CPU 45%. Marketing promises a peak four times larger in two months. The vendor ships new servers in three weeks, and there is no cloud. How do you plan capacity and what do you do before day X?

3. Implement a token bucket on Redis for 12 Node processes on three servers. Why not GET and SET from code? Whose clock do you use? What about Redis Cluster, key TTLs, and what does the API do when Redis is unavailable?

4. A B2B API with 300 clients. One large client starts an import and takes 70% of the capacity, and the rest get timeouts even though each is within its own quota. How do you design limits so the system is fair? What do you do with a per-IP limit?

7. Design the API and data model of a URL shortener. How will you generate the code: a hash of the URL, a random string or a counter? What do you do about collisions and how do 10 API instances get unique ids without hitting a shared store on every write?

11. Design a double-entry ledger model and a transfer API between users. Which tables, how do you guarantee that the sum of entries equals zero, how do you store the balance, and how does an Idempotency-Key protect against a double transfer when the client repeated the request after a timeout?

15. The business needs a daily report: revenue by region broken down by product category and customer segment. The data lives in three services: orders, catalog, customers, each with its own database. How do you build the report? Why should you not just give BI read-only replicas and join there?

19. The OrderPlaced v1 event had an amount field in hryvnias as a floating point number. Now currency and kopecks are needed: amount as an object with value and currency. The log has 40 million old events. How do you change the event schema? Why can you not just UPDATE the old events?

20. Tracing 100% of requests at 3000 rps costs more than the whole cluster. How do you configure sampling? How does head-based differ from tail-based? How do you avoid losing the traces of exactly the requests that failed?

21. A vendor promises "exactly-once delivery". What do you ask? What do Kafka transactions really give? Why does that not solve the double-charge problem in Stripe?

22. Design an idempotent POST /payments. The client retries on a timeout. Describe the table schema, handling of two parallel requests with one key, a request with the same key but a different body, and the key retention period.

23. After an outage, 50 thousand messages accumulated in the DLQ over 6 hours. The bug was fixed and redrive was pressed. 5 minutes later the database went down, some messages are back in the DLQ, and some data was overwritten with old values. What went wrong and how should it have been done?

24. Design a relay for an outbox on Postgres. Polling or CDC? How do you run several relay replicas without double publishing? How do you keep order per aggregate? What do you do about table growth?

25. A topic with replication.factor=3, min.insync.replicas=1, and a producer with acks=1. After a broker failure, messages disappeared that the producer got a success for. How exactly did this happen? What configuration would you set and what does it cost when one and two brokers fail?

26. A mobile client makes 3 attempts, the API gateway 3 attempts, the orders service 3 attempts to the payments service, and a background BullMQ job has attempts: 5. What happens when the payments service starts responding slowly? How do you distribute retries across layers correctly?

Train these questions

Answer out loud and get graded

Other topics

Explore the platform

20. Order events `created`, `paid`, `shipped` go through SQS standard. With one consumer everything worked; after scaling to four, some orders get `paid` processed before `created` and the handler crashes. Why, and how do you get ordering within one order without global ordering?

28. Kafka broker disks are 90 % full. You cut `retention.ms` on a 2 TB topic from 7 days to one hour, but disk usage barely moves for hours. Why, and what do you do?

31. A Redis Streams consumer crashed mid processing and restarted. It reads with `XREADGROUP GROUP g c1 STREAMS orders >`, and the 300 messages it was processing at the moment of the crash stay unprocessed. Why, and how do you pick them up?