What Could Help
Let’s consider four options and check whether each one helps at noon.
More instances of the commerce service
We could run more instances of the commerce service. More instances can handle more requests at the same time. But every order is a transaction that changes the same mug’s stock, and every one of these transactions runs on the shard’s primary. More instances send more of these transactions to the primary at the same time. They conflict on the stock, so they still run one after another. The primary still places 100 orders a second.
A bigger server for the primary
We could move the shard’s primary to a faster server. A faster server shortens each transaction a little. But the transactions on the one stock count still run one after another. To keep up with 200 orders a second, each transaction must take at most ms. That is half of the 10 ms we assumed. A faster server is unlikely to halve it, because part of each transaction is waiting for a replica over the network.
Load shedding
We saw load shedding earlier. When requests arrive faster than the server can process them, the server refuses some requests with 503 Service Unavailable. Here, we would refuse the orders above 100 a second. That protects the primary. But 200 orders arrive each second, so we refuse half of them. Those shoppers cannot complete their purchases. When a client receives a 503, it waits and tries again. So their storefronts send the request again. The retries add to the orders that arrive.
Throttling
We saw throttling earlier too. Instead of refusing a request, the server holds it and sends the response late. Here, we would hold each order until the primary can take it. No shopper is refused. But each held request keeps a connection open, and the primary still places only 100 orders a second. The orders wait as they did in the first minute. An order that arrives 10 seconds into the sale waits 10 seconds for its turn. That is far past the two-second requirement. Its storefront times out and sends the request again.