The Flash Sale

Mugshop sells a new mug at half price. There are 15,000 mugs, and the sale starts at noon on Friday. The merchant announced the sale to a large mailing list, so many shoppers will be on the storefront at noon to buy it.

Our assumptions

About 12,000 shoppers place an order in the first minute. That is $12,000/60=200$ orders a second. I assume the orders are spread evenly over the minute. In a real sale, more of them arrive in the first few seconds. That makes the problem worse.

There are 15,000 mugs. I chose more mugs than there are orders in the first minute. That way, the mug does not sell out during the minute we follow.

Placing an order is the transaction we designed for Shopend. It reads the cart, reduces the stock, creates the order, and marks the cart as ordered. Mugshop’s data is on one shard, and every write goes to the shard’s primary. We keep the whole transaction on the primary. Its reads run there too. So every order of the sale runs on the same server. The shard’s replicas do not help, because none of this work can run on them.

Every order also reduces the stock of the same mug. Two transactions that change the same stock conflict. The database aborts one of them, and it is retried. So the orders run one after another. I assume each transaction takes about 10 ms. The transaction touches several documents, and a purchase also waits for a replica to acknowledge it. One second is 1,000 ms, so the primary places at most $1,000/10=100$ orders a second.

Recall the shop API’s performance requirement: 95 percent of order placements must get a response within two seconds, measured at the API. We kept that requirement for Shopend.

The first minute

Let’s follow the orders from noon. Orders arrive at 200 a second, and the primary places 100 a second. So every second, $200-100=100$ more orders are waiting.

After 10 seconds, $10\times 100=1,000$ orders are waiting. A new order is behind those 1,000 orders. The primary places 100 a second, so the new order waits $1,000/100=10$ seconds. That is far past the two-second requirement.

The requirement fails much earlier than that. An order waits two seconds when 200 orders are ahead of it, because $200/100=2$. The waiting orders grow by 100 a second, so they reach 200 two seconds after noon. Only the orders of the first two seconds get a response in time. That is $2\times 200=400$ of the 12,000 orders. That is about 3 percent. The requirement asks for 95 percent.

After a minute, $60\times 100=6,000$ orders are waiting. The last order of the minute waits $6,000/100=60$ seconds.

Retries

These numbers count each order once. In practice, a storefront that waits too long times out and sends placeOrder again. The idempotency key stops the retry from creating a second order. But the retry is one more request for a primary that is already behind. So the real number of waiting orders is larger. Shoppers see errors and pages that do not load.