A Refund That Keeps Failing

Suppose the payment service is running, but the refund for order 93 fails. For example, the payment service cannot reach the payment processor. The refund is not made, so the payment service does not acknowledge the message. After the 30-second time limit, the broker gives the message out again.

Two kinds of failure

Some failures go away. If the payment processor is unreachable for a minute, the refund fails, the message comes back, and a later attempt succeeds.

Other failures do not go away. Suppose a new release of the payment service has a bug that fails every refund for an order with more than one item. Every attempt at such a refund fails the same way. No matter how many times the broker gives out the message, the refund is not made.

A limit on deliveries

Retrying a message like this only uses up the payment service’s time. One way to deal with a message that keeps failing is to limit the attempts. The broker counts how many times it has given out each message, and when the limit is reached, it stops giving out the message.

The limit has to be high enough that a failure that goes away, such as an outage of the payment processor that lasts a few minutes, does not use up all the attempts.

The dead-letter queue

When the last delivery also ends without an acknowledgment, the broker stops giving out the message. If the broker drops the message, the refund is never made. A common way to prevent this is to set up the broker to move the message to a separate queue that no service takes messages from. This queue is called a dead-letter queue.

A message in the dead-letter queue waits until a person deals with it. So a dead-letter queue usually comes with an alert: when a message reaches the queue, a person is notified, for example by email. If the refund failed because of the bug, a developer finds the bug, and a new release of the payment service fixes it. Then the developer moves the messages back to the refund queue, and the payment service handles them.

Moving the messages back is safe, because the payment service is an idempotent consumer.

Both brokers we have used support dead-letter queues. In Amazon SQS, the limit is called maxReceiveCount. SQS can also move the messages from the dead-letter queue back to the queue they came from. In RabbitMQ, a quorum queue has a delivery limit, and since RabbitMQ 4.0 its default is 20. A message over the limit goes to a dead-letter queue only if we set one up. Otherwise, RabbitMQ drops it.