Refunding Once

With at-least-once delivery, a consumer can handle the same message twice. There are two ways this happens with the refund for order 93:

  • The payment service makes the refund and then crashes before it acknowledges the message.
  • The payment service is slow and is still working when the 30-second time limit passes. The broker gives the message to another instance, and both make the refund.

Either way, the shopper gets the money back twice. So the payment service has to recognize a refund message it has already handled.

Recognizing a repeat

The payment service reads the payment for order 93 from the shared database and refunds it. Then it saves a refund record with the shop ID and the order ID. An order is refunded at most once, so a unique index on the shop ID and the order ID allows only one refund record for each pair. Recall that a unique index on placeOrder’s idempotency key stopped two requests with the same key from saving two orders. This index works the same way.

So the payment service handles a refund message like this:

  1. Look for a refund record for this shop and order. If there is one, acknowledge the message and stop. Otherwise, continue to step 2.
  2. Make the refund.
  3. Save the refund record.
  4. Acknowledge the message.

If the payment service crashes after step 3, the broker gives the message out again. Step 1 finds the record and acknowledges the message without a second refund.

The payment processor

Suppose the payment service crashes after step 2 but before step 3. The payment processor has made the refund, but there is no refund record. The broker gives the message out again, step 1 finds no record, and step 2 sends the refund request to the payment processor a second time.

Now the payment service has sent a request and does not know what happened. Payment processors such as Stripe and PayPal accept an idempotency key with a refund request. The payment service sends the key refund-mugshop-93 with the refund for order 93 in Mugshop. The key comes from both the shop and the order, so another shop’s order 93 gets a different key. Every attempt at this refund sends the same key. When the payment processor sees refund-mugshop-93 a second time, it returns the result of the first request instead of making the refund again.

The key also helps when an instance is slow. Suppose two instances hold the same message after the time limit passes, and both pass step 1 before either saves a record. Both send refund-mugshop-93. If the second request arrives while the first is still running, the payment processor responds with an error. Stripe, for example, does this. The second instance does not acknowledge, and when the broker gives the message out again, step 1 finds the record. If the second request arrives after the first is done, the payment processor returns the first result. Then both instances try to save the refund record, and the unique index lets only one succeed. We treat the failed save as success, because the record is there. So the second instance acknowledges the message too. Either way, the payment processor makes the refund once.

With the refund record and the key, the payment service is an idempotent consumer: a consumer that can handle the same message more than once with the same effect as handling it once.