Polling

Marcus sees Priya’s message only when Marcus’s client sends a request for it. The API already has that request: a request with after returns every message after a given one. Team 4’s room is room 12, and the newest message on Marcus’s screen is 61842. So the simplest fix is for Marcus’s client to send this request every 10 seconds, whether or not anyone has posted:

GET /rooms/12/messages?after=61842

When nothing has been posted since 61842, the response holds an empty list of messages. Once Priya has posted, the response holds Priya’s message. The server saved it as 61843. The client adds it to Marcus’s screen and sends after=61843 from then on, because 61843 is now the newest message it has.

The 10 seconds is the interval between polls. It is set to 10 seconds here, but the client could poll more often.

Sending a request again and again like this, to check whether anything has changed, is called polling. Each of those requests is a poll.

sequenceDiagram
    participant P as Priya's client
    participant S as Server
    participant M as Marcus's client
    M->>S: GET /rooms/12/messages?after=61842
    S-->>M: 200, no messages
    P->>S: POST /rooms/12/messages
    S-->>P: 201, message 61843
    Note over M: waits up to 10 seconds
    M->>S: GET /rooms/12/messages?after=61842
    S-->>M: 200, message 61843

The delay

Team 4 sits together in the lab. Priya posts the link to the design doc and tells Marcus, “It is in the chat.” How long until Marcus sees it?

That depends on when the server saves the message, compared with Marcus’s polls. If the message is saved just before Marcus’s client sends a poll, that poll returns it. If it is saved just after a poll, the message waits almost 10 seconds for the next one. A message is equally likely to be saved at any moment between two polls. So a message waits at most 10 seconds, and about 5 seconds on average.

Five seconds is a long time when two people sit next to each other. Marcus looks at the screen right after Priya speaks, and the link is not there. Marcus asks Priya to post it again. The latency requirement says Marcus should see the message within one second.

Polling more often

The interval sets the delay. If Marcus’s client polls every second, a message waits at most one second for the next poll, plus the time the request and the response take on the network.

Setting the interval to one second is a simple way to meet the latency requirement. But then every client with a room open sends a poll every second, and the server must handle all those requests.

Assume at the busiest hour there are 600 open clients. Each client sends one poll per interval:

How many of those polls return a message? To answer that, we go back to the load assumptions in the requirements. The 600 open clients are spread over 60 courses, so each course has about 10 open clients.

We also estimated how many messages are posted at the busiest hour. It came to about one message a second, over all rooms.

Each of the 600 clients polls the room it has open, once a second. A message is posted in one room, so only the clients that have that room open get it. That room’s course has only about 10 open clients. So each second, at most 10 of the 600 polls return a message. The other 590 or more come back empty.

What an empty response costs

An empty response is short, but the server does almost as much work for it as for a full one. The poll is a full HTTP request. It arrives with its headers, including the credentials that identify Marcus. The server authenticates the request to learn which user sent it. It checks that Marcus may read room 12: room 12 is private, so it looks for Marcus’s row in chat_room_members. Then it runs the query that reads every message in room 12 after 61842, finds none, and sends a response with an empty list.

The index on (room_id, message_id) makes that last query fast. The database still runs it 600 times a second, and at most 10 of those queries find a new message.