One Request or Two

Halfway through a reply, Lena’s laptop loses its Wi-Fi connection for a moment. In our design, the reply is the response to Lena’s POST. When the connection drops, the response is gone. The client gets an error partway through the body, and Lena has half a reply on the screen. A new connection cannot pick up the old response.

There is another design that can survive this. It uses two requests.

Two requests

The first request sends Lena’s message:

POST /conversations/31/messages

The server saves Lena’s message as 52817. It also saves an empty reply, 52818, with the status writing, as in the polling option. Then it hands the reply to a background job and responds at once: 201 Created, with Lena’s message and the identifier of the reply, 52818.

The background job sends the conversation to the model provider. As the provider’s events arrive, the job adds their text to the reply’s row. It does not write after every event. It collects the text and writes it a few times a second. When the reply is done, the job sets the status to complete.

The second request reads the reply:

GET /conversations/31/messages/52818/stream

The server responds with a stream of events, as before. It sends the text the row already has. Then, while the status is writing, it reads the row a few times a second and sends the new text. When the status is complete, it sends done and ends the response. If the reply is already complete when the GET arrives, the server sends the whole reply in one text event, and then done.

The difference from one request is where the reply lives. The background job writes it into the database, apart from any request, and the GET only reads it.

Now suppose the connection drops halfway. The job keeps writing the reply. The client sends the GET again, and it gets the reply from the row: the text written so far, and then the rest as the job writes it, or the whole reply if it finished while the client was reconnecting. Sending a GET again does no harm, because it only reads. Sending the POST again would post Lena’s question a second time and pay for a second reply.

The client already has half of the reply, so the server does not need to send that half again. For this, the server gives each event an identifier in an id: field. Here, the identifier is the number of characters of the reply the client has, including this event’s text:

id: 1250
event: text
data: {"text": " the links move"}

When the client sends the GET again, it puts the identifier of the last event it received in the Last-Event-ID header:

GET /conversations/31/messages/52818/stream
Last-Event-ID: 1250

The server sends the text after the first 1,250 characters, and Lena’s client continues from where it stopped. This is the same idea as after=120 in the long poll.

The browser has a built-in reader for this kind of stream, called EventSource:

const source = new EventSource("/conversations/31/messages/52818/stream");

EventSource sends the GET, splits the body into events, and calls our code for each event of a given type. When the connection drops, it sends the GET again on its own, with Last-Event-ID. The server can set how long it waits first, with a retry: field that gives the wait in milliseconds.

What two requests need on the server

This design needs three things on the server that one request does not.

• A background job. The reply has to keep going while no GET is open, so the model’s work cannot run inside a request’s handler. If the server restarts during a reply, the job is lost, and something has to mark the reply as incomplete.
• Writes for the growing reply. The job writes the reply to the database a few times a second, and each open GET reads it a few times a second. This is the cost we noted for polling, without the polls.
• A request to stop the reply. Closing the connection no longer stops the model, because the job runs apart from any request. So the API needs one more request, such as POST /conversations/31/messages/52818/stop. It sets the reply’s status to incomplete, and the job checks the status each time it writes.

My choice: one request

I keep the reply in the response to the POST.

A reply takes about 10 seconds. A connection rarely drops in those seconds. When it does, Lena can send the question again, and the client offers a button for it. That costs one more reply from the model provider. Two requests would save that, but they cost a background job, many writes to the database, and a request of their own to stop a reply.

The choice would change if a reply took minutes, for example an assistant that works on a task for several minutes before it is done. Then a dropped connection is likely, sending the request again wastes minutes of work, and the student may close the page and come back later. In that case, I would use two requests.

OpenAI’s API offers both designs for the same reason. By default, the reply is the response to the request that sends the conversation. Its background mode, meant for tasks that take a long time, works like our two-request design: a client that loses its connection reconnects with a GET and the number of the last event it received.

One request also makes stopping a reply simple. Lena may stop a reply that is going in the wrong direction, or close the tab. Either way, the server should stop its request to the model provider. The provider charges for every token, and nobody will read the rest.

To stop a reply, the client aborts its fetch, and the browser closes the connection. Closing the tab closes it too. On the server, the handler watches for a closed connection. When it sees one, it aborts its own request to the model provider, and the model stops writing.

The server saves Lena’s message as soon as the POST arrives, and the reply when it is complete. When a reply stops partway, Lena has already read part of it. I save that part too, with the status incomplete. So the conversation’s history shows Lena the same part of the reply that Lena saw. The model provider receives that same history with Lena’s next message.