Handling the Messages
At the peak, 500 games end a second, so 500 messages a second reach the leaderboard queue. The leaderboard service runs as several instances.
Competing consumers
All the instances take messages from the one leaderboard queue. The broker gives each message to one of them. These are competing consumers.
If each instance had a queue of its own, each instance would get every message, and each would set the same scores. That is wasted work.
500 messages a second means 1,000 sorted-set updates a second, two for each game. That is small for one Redis server.
Acknowledging the message
For the rest of this section, we need three ideas from the weekend reading on cancelling an order: acknowledgment, delivery guarantees, and idempotent consumers. A consumer acknowledges a message to tell the broker it is done with it. If a consumer takes a message and does not acknowledge it within the broker’s time limit, the broker gives the message out again.
When should the leaderboard service acknowledge? To decide, we compare what it costs to lose a message with what it costs to handle one twice.
A lost message is bad. The two players’ ranks stay wrong until their next game. For a player who stops playing, the rank stays wrong for good.
A message can also be lost before it reaches the broker. The game service saves the new ratings and then sends the message. If it crashes in between, the message is never sent. The transactional outbox from the weekend reading would prevent this. I do not add one to this design. The nightly repair in “Losing the Leaderboard’s Data” fixes these ratings instead.
A message handled twice does no harm. The message carries the new rating, 2148, not a change like “+8”. Setting the score to 2148 twice gives the same result as setting it once. So the leaderboard service is already an idempotent consumer, with no extra work.
So I choose at-least-once delivery. The leaderboard service acknowledges a message after it has updated the sorted set.
Messages out of order
Suppose the leaderboard service is down for a few minutes, and player 5512 plays two blitz games in that time. After game 1, player 5512’s rating is 2148. After game 2, it is 2156. Both messages wait in the leaderboard queue.
When the leaderboard service comes back, the instances work through the waiting messages:
- Instance A takes the message from game 1. At the same moment, instance B takes the message from game 2.
- Instance B is a little faster. It sets player 5512’s score to 2156.
- Instance A then sets it to 2148.
Player 5512’s rating is 2156, but the leaderboard shows 2148. It stays wrong until player 5512’s next game.
A message that the broker gives out again can cause the same problem. If an instance crashes before it acknowledges the message from game 1, that message comes back after the time limit. By then, the message from game 2 may already have been handled.
The fix is to let the leaderboard service tell an older message from a newer one. I add a count of the player’s games in that time control to the message:
{ "playerId": 5512, "username": "knightowl", "rating": 2148, "games": 1204 }
The game service knows the count, because it saves every game. For each player in each time control, the leaderboard service keeps the count from the last message it applied. It ignores a message whose count is not higher. In the example, the message from game 2 has count 1205. When instance A then handles the message from game 1, with count 1204, it ignores it.
The counts take memory too. I assume about 50 bytes for each count, including the extra memory Redis uses for each entry. For 20 million players in three time controls, that is bytes, about 3 GB. With the 6 GB from the load assumptions, the leaderboard’s data takes about 9 GB. That still fits in the memory of one server.
Checking the count, setting the score, and saving the new count must happen as one step. Otherwise, two instances could both check the count before either saves the new one, and both messages would pass the check. Redis can run several commands as one step, so this is possible. How to write it is beyond the scope of this course.