When a Consumer Crashes

Recall what the broker does when a consumer asks for the next message. It gives the message to the consumer and removes it from the queue. Now suppose the payment service takes the refund message for order 93 and crashes before it makes the refund. The broker has already removed the message, so the refund is lost.

Acknowledgment

To prevent this, the broker keeps the message until the consumer tells it that the message is handled. Telling the broker that a message is handled is called an acknowledgment. We used the same word in replication, where a replica tells the primary that it has saved the changes. We left the acknowledgment out of all our earlier examples, but a consumer must always send it.

The payment service takes the refund message, makes the refund, and then acknowledges the message. Until the acknowledgment arrives, the broker keeps the message. If the payment service crashes before it acknowledges, the broker gives the message out again. When the payment service restarts, it takes the refund message for order 93 again and makes the refund.

Several instances

With acknowledgment, a message that an instance takes is not gone from the queue. The broker keeps the message and marks it as taken. While the message is marked, normally no other instance takes it.

If the instance acknowledges the message, the broker removes it. If the instance crashes before it acknowledges, the broker makes the message available again, and the next instance that is ready takes it. That may be another instance of the payment service, or the same instance after it restarts.

How the broker learns that an instance is gone

The two brokers we have used learn this in different ways:

• RabbitMQ. Each consumer keeps a connection open to the broker. If the instance crashes, the connection drops, and RabbitMQ gives out again every message the instance had not acknowledged. If the instance’s server is lost without closing the connection, RabbitMQ notices because the connection stops sending its regular signals, called heartbeats.
• Amazon SQS. A consumer has no open connection. It sends SQS a request for messages. So SQS uses a time limit instead: if the instance does not acknowledge a message within the limit, SQS makes the message available again. The limit also covers an instance that is stuck.

The time limit

The time limit works like the deadline on a remote procedure call, the time at which the caller stops waiting. The time limit is when the broker stops waiting for an acknowledgment. As with a deadline, the instance does not stop working when the broker stops waiting.

So the limit can be wrong in two ways. If it is too short, the broker gives the message to another instance while the first one is still working on it. If it is too long, a message held by a stuck instance waits longer than it needs to before another instance gets it.

Recall that a payment processor is a company, such as Stripe or PayPal, that takes card payments for other businesses. The payment service makes a refund by calling it. I assume that call takes about a second. For the refund queue, I set the limit to 30 seconds. A working instance finishes long before 30 seconds. If an instance is stuck, its refund waits up to 30 seconds before another instance takes it. For a refund, a 30-second delay is not a problem.

SQS calls this limit the visibility timeout, because a message that is taken is hidden from the other consumers until the limit passes. Its default is 30 seconds. RabbitMQ has a limit too, for a consumer that never acknowledges, but it is meant for much longer waits. Its default is 30 minutes, and it cannot be set below one minute. In the rest of this reading, I use the time limit the way SQS does.