Putting It Together

Here is the design:

graph TD
    C[Clients: website, apps] -->|leaderboard pages| HC[Shared cache]
    HC --> G[Gateway]
    C -->|other requests| G
    G --> PS[Player service]
    G --> LS[Leaderboard service]
    G -.->|WebSocket| GS[Game service]
    PS --> PDB[(Player database: sharded by player)]
    GS --> PDB
    GS -->|game ended| BR[Broker]
    BR -->|leaderboard queue| LS
    LS -->|writes| RP[(Redis primary)]
    RP -->|copies| RR[(Redis replicas)]
    LS -->|reads| RR

When a game ends

  1. The game service computes both players’ new ratings and saves them in the player database.
  2. It puts a message on the leaderboard queue, with the time control, both players, their new ratings, their usernames, and their counts of games.
  3. One instance of the leaderboard service takes the message. For each player, it checks the count of games. If the count is higher than the count from the last message applied for that player, it sets the player’s score in the sorted set and the username in the map of usernames.
  4. It acknowledges the message.

When someone views a page

  1. Every 2 seconds, the open page sends GET /leaderboards/blitz?page=3.
  2. If the shared cache in front of the gateway has a copy younger than 1 second, it sends that copy.
  3. Otherwise, the request goes through the gateway to the leaderboard service. It gets ranks 101 to 150 from the sorted set on a Redis replica, and their usernames from the map of usernames. The cache keeps the response for 1 second.

The decisions

Each decision applies something from an earlier chapter:

  • Requirements and load. We asked clarifying questions, wrote the requirements with quality attributes, and estimated the load from daily numbers.
  • A REST API. Two GET endpoints, both public, so their responses can be cached.
  • A service of its own. The leaderboard is its own deployment boundary. The reasons are failure, change, capacity, and ownership.
  • A query across shards. It can return the first pages, but it cannot return a player’s rank. So we chose a different structure for the data.
  • A copy of the data. The sorted sets are a copy for one purpose, like a materialized view. The player database is the source of truth.
  • Replication. One Redis primary for the writes, and replicas for the reads and for failover. A failover can lose the last updates. A nightly repair from the player database fixes them.
  • Denormalization. The usernames are copied into Redis.
  • Asynchronous processing. The game service sends a message and does not wait for the leaderboard service. So a slow or failed leaderboard service does not affect games.
  • A queue, not yet a topic. Only the leaderboard needs the message today.
  • Competing consumers. The instances share one queue.
  • At-least-once delivery. Setting a rating is idempotent, and the counts of games handle messages out of order.
  • Polling and HTTP caching. We agreed with the interviewer that a delay of 5 seconds is fine, so polling every 2 seconds is enough. For most of the requests, the shared cache sends the response.

The sorted set is the only new building block in this chapter.