System design · Question 46 of 50
Why use a message queue? What do at-least-once delivery and idempotent consumers mean?
Asynchronous architecture.
The answer
A queue (RabbitMQ, SQS, Kafka) lets a service hand off work and move on: the order API puts "send the confirmation email" on a queue and answers the user at once, and workers process it at their own pace. It absorbs traffic spikes, retries failures without losing work, and decouples services, so one being slow or down does not break the other.
Most queues deliver at least once: a worker can crash after doing the work but before acknowledging it, so the message comes again, and consumers must be idempotent (check a processed-message id). Messages that keep failing go to a dead-letter queue for a person to look at. Kafka is different: a log that each consumer reads from its own position, good for event streams and replay.
The picture
API ─▶ [ queue: m1 m2 m3 ] ─▶ worker 1
▲ answers the user now ─▶ worker 2 ack after success; no ack → delivered again
└─ fails 5 times ─▶ dead-letter queueHow do you keep messages in order?
Show the answer
Order holds within a partition or message group, so put every message for one thing (an order id) on the same partition.