The Leaderboard’s Data

A copy of the ratings

The ratings are already in the player database. The sorted sets hold the same ratings, kept in order by rating. So the sorted sets are a copy, and the player database stays the source of truth: the one place whose data is correct by definition. If the two ever disagree, the player database is right.

The copy belongs to the leaderboard service. No other service reads or writes it. The player service and the game service keep using the player database.

We keep one sorted set for each time control, because each time control has its own leaderboard.

graph TD
    G[Gateway] --> PS[Player service]
    G -.->|WebSocket| GS[Game service]
    G --> LS[Leaderboard service]
    PS --> PDB[(Player database: the source of truth)]
    GS --> PDB
    LS --> R
    subgraph R[Redis]
        B1[(bullet sorted set)]
        B2[(blitz sorted set)]
        B3[(rapid sorted set)]
    end

We keep the copy for one purpose, which is ranking players. In that way it is like a materialized view. But a materialized view is refreshed by the database itself. Here, nothing refreshes the copy for us. The leaderboard service has to learn about each new rating and update the sorted set itself.

One server

From the load assumptions, each player takes about 100 bytes, with the username. So one time control takes about 2 GB, and all three take about 6 GB.

That fits in the memory of one server. So we do not shard the sorted sets. Not sharding also keeps every operation on one server, so a rank or a page never needs a query across shards.

A primary and replicas

One Redis server alone has two problems. If it fails, the leaderboard is down until it comes back. And it takes every read.

So we use replication, as we did for Shopend’s database. One server is the primary. It takes all the writes and copies them to the replicas. The replicas take the reads. If the primary fails, one replica takes over as the new primary.

graph TD
    LS[Leaderboard service] -->|writes| P[(Redis primary)]
    P -->|copies| R1[(Redis replica)]
    P -->|copies| R2[(Redis replica)]
    LS -->|reads| R1
    LS -->|reads| R2

At the peak, the writes are 1,000 rating changes a second. One Redis server handles tens of thousands of simple operations a second, so one primary is enough for the writes.

A replica can be a little behind the primary. That is replication lag. For Redis, it is usually well under a second. A new rank may show on a replica a moment later than on the primary. That is fine for a leaderboard.