Losing the Leaderboard’s Data

When the primary fails

When the Redis primary fails, a replica takes over. But Redis replicates asynchronously: the primary reports success for a write before the replicas have it. So the last few updates the primary accepted may be missing on the replica that takes over. We saw the same problem when we covered replication: a purchase the primary had confirmed was missing after a failover.

By then, the leaderboard service has acknowledged those messages, so the broker does not give them out again. The players in those games keep their old ranks until their next game.

When the primary and all its replicas are lost

Now suppose we lose the primary and all its replicas, and the sorted sets and the usernames are lost with them.

Games go on. The game service saves the new ratings in the player database and sends its messages to the broker, as before. It does not depend on the leaderboard.

The messages wait in the leaderboard queue. The broker keeps them until the leaderboard service takes them.

The leaderboard service rebuilds its data from the player database. It reads every rating, with the usernames and the counts of games, across all the shards, once. From those rows, it builds the sorted sets and the usernames again. The rebuild is slow, but we only need it when the primary and all its replicas are lost, and that is rare.

Then the leaderboard service handles the messages that waited. Some of them are for games whose ratings the rebuild already read. The counts of games take care of those: the leaderboard service ignores a message whose count is not higher than the one it already has.

While the leaderboard is being rebuilt, it is down, but players can still play games. That is what the availability requirement asks for.

This is why we keep the player database as the source of truth. We can always build the leaderboard’s data again from it.

A nightly repair

Some ratings in the sorted sets can still be wrong: those lost in a failover, and those whose message the game service never sent because it crashed after saving the ratings. Each stays wrong until that player’s next game. For a player who stops playing, it stays wrong permanently.

A wrong rating gives that player the wrong rank. It also puts other players’ ranks off by one: every player they pass by mistake, and every player they stay below by mistake. So a few wrong ratings can make many ranks wrong.

So every night, the leaderboard service repairs its data in place. It reads every rating from the player database, with each player’s count of games, the same way as the rebuild. For each player, it uses the same one-step check as for the messages: if the player database’s count is higher than the one in Redis, it sets the score and the count. So if a message brings a newer rating while the repair runs, the repair does not overwrite it with the older one. The leaderboard stays up while the repair runs.

So in normal operation, a new rank shows in real time. After a failure, some ranks can be wrong until the next nightly repair. I accept that, because these failures are rare.