Saving and Publishing Together

So far, we have made sure that a refund message, once the broker has it, leads to exactly one refund. The message can still be lost before it reaches the broker.

A crash between saving and publishing

The cancellation has two steps. The commerce service sets order 93 to CANCELLED and puts the mug back in one transaction. Then it publishes to order-cancelled. Suppose the commerce service crashes after the transaction commits and before it publishes. Order 93 is cancelled, and the message never reaches the broker. The shopper sees the order as cancelled, but gets no refund, and the parcel ships.

The transactional outbox

A common way to fix this is to save the message in the database instead of publishing it right away. The commerce service already sets order 93 to CANCELLED and puts the mug back in one transaction. In the same transaction, it adds the message as a document in an outbox collection:

{
  "shopId": "mugshop",
  "topic": "order-cancelled",
  "message": { "shopId": "mugshop", "orderId": "93" },
  "sentAt": null
}

Now the cancellation and the message are saved together.

A separate program, called the message relay, publishes the messages in the outbox. It repeats three steps:

  1. Read the outbox documents whose sentAt is null.
  2. Publish each message to its topic, and wait for the broker to confirm it.
  3. Set the document’s sentAt to the current time.

The commerce service no longer publishes the message itself. It responds to the storefront as soon as the transaction commits. If it crashes after the transaction, the outbox document is there, and the relay publishes it.

This design is called the transactional outbox.

Publishing a message twice

Suppose the relay publishes the message and crashes before step 3. The document still has sentAt set to null, so the relay publishes the message again when it restarts. This means the transactional outbox delivers each message at least once.

At-least-once delivery is enough here, because each subscriber of order-cancelled already handles a repeated message.

Placing an order

Placing an order has the same problem in two places. The commerce service saves order 93 as PROCESSING and then sends the message to the order queue. If it crashes between the two steps, the order stays PROCESSING, and the storefront keeps polling its status. The order worker sets order 93 to PLACED and then publishes order-placed. If it crashes between those two steps, the order is never shipped, and the shopper gets no confirmation email.

The transactional outbox fixes both problems in the same way. The commerce service saves the message for the order queue in the outbox, in the same transaction as the new order. The order worker saves the order-placed message in the outbox, in the same transaction that sets the order to PLACED. The message relay sends both to the broker.