System design · Question 46 of 50

Why use a message queue? What do at-least-once delivery and idempotent consumers mean?

What they're checking

Asynchronous architecture.

The answer

A queue (RabbitMQ, SQS, Kafka) lets a service hand off work and move on: the order API puts "send the confirmation email" on a queue and answers the user at once, and workers process it at their own pace. It absorbs traffic spikes, retries failures without losing work, and decouples services, so one being slow or down does not break the other.

Most queues deliver at least once: a worker can crash after doing the work but before acknowledging it, so the message comes again, and consumers must be idempotent (check a processed-message id). Messages that keep failing go to a dead-letter queue for a person to look at. Kafka is different: a log that each consumer reads from its own position, good for event streams and replay.

The picture

API ─▶ [ queue: m1 m2 m3 ] ─▶ worker 1
 ▲ answers the user now      ─▶ worker 2   ack after success; no ack → delivered again
                                 └─ fails 5 times ─▶ dead-letter queue
The follow-up they'll ask

How do you keep messages in order?

Show the answer

Order holds within a partition or message group, so put every message for one thing (an order id) on the same partition.

Your progress stays on this device.