Long Polling on the Server

A kept request uses resources on the server. Until the server answers it, the server holds an open connection to the client. It also holds the request in memory, with a note of which room the request waits on. Every open client has one kept request, so at the busiest hour the server holds 600 at any moment.

Some servers give each request its own thread until it gets a response. A thread takes memory, and a server has a limited number of them. 600 kept requests could take them all. Then Priya’s POST would have to wait for a free thread.

When nobody posts

If nobody posts for a long time, the kept request stays open with no response. A request can pass through proxies and load balancers on its way to the server. A proxy is a server between the client and the server that passes requests and responses along. A school’s or a company’s network often has one. Many of them close a request that gets no response within a set time. Then Marcus’s client gets an error, not a response.

So the server does not wait for a proxy to close the request. After a while with nothing new, it answers with an empty list. An empty list is a response too, so the client sends a new request right away.

How long should the server wait? If it is too long, a proxy may close the request first. nginx and AWS’s load balancer, for example, close a request after 60 seconds by default. If it is too short, the server sends many empty responses, as it did with polling. 30 seconds is a common choice. RFC 6202, published in 2011, lists the known issues with long polling. It says a 30-second limit is generally safer than longer ones. Firestore’s web SDK, for example, ends its long polls after 30 seconds.

The server’s code

Here is the server’s code. Each handler decides whether to answer now or later.

app.get("/rooms/:roomId/messages", async (req, res) => {
  const messages = await getMessagesAfter(req.params.roomId, req.query.after);

  if (messages.length > 0) {
    res.json({ messages });  // something new: answer now
  } else {
    keepRequest(req.params.roomId, res);  // nothing new: answer later
  }
});

app.post("/rooms/:roomId/messages", async (req, res) => {
  const message = await saveMessage(req.params.roomId, req.body);
  res.status(201).json(message);

  answerKeptRequests(req.params.roomId, { messages: [message] });
});

A GET that finds new messages answers at once. A GET that finds nothing new calls keepRequest, and a POST calls answerKeptRequests after it saves the message.

The server keeps, in memory, a list for each room: the requests waiting on that room. What it stores for each request is res, the object the server uses to answer that request. As long as the server does not call res.json, the request stays open. The client keeps waiting.

  • keepRequest(roomId, res) adds res to the room’s list. It also starts a 30-second timer. When the timer runs out, if res is still in the list, the function takes it out and answers it with an empty list.
  • answerKeptRequests(roomId, body) goes through the room’s list and answers each request with body, which holds the new message. Then it empties the list. When one of those requests’ timers runs out later, res is no longer in the list, so the timer does nothing. No request gets two answers.

This code is simplified. getMessagesAfter and saveMessage stand for the queries on chat_messages. A real server also authenticates each request, checks that the user may read the room, and removes a kept request when its client disconnects.

Messages posted between two requests

The server answers Marcus’s kept request with Priya’s message. Until Marcus’s next request arrives, the server has no request from Marcus’s client. Suppose Priya posts another message before that request arrives. Marcus does not miss it. The message is saved in the database, and Marcus’s next request asks for every message after the last one Marcus has. The GET handler finds the new message and answers at once.

Letting the client set the wait time

The API can also let the client say how long it is willing to wait, with a query parameter:

GET /rooms/12/messages?after=61842&wait=30

The server answers after at most wait seconds.