A Service for Running Code
Suppose a student’s method has a bug: it adds to a list in a loop that never ends. The 5-second limit stops the process eventually. But in those 5 seconds, the list keeps growing. By default, Java lets one process use up to a quarter of the server’s memory, so four such runs at once can fill it. Then every request the application handles slows down or fails.
The fix for this is a limit on memory as well as time. Recall that a container runs a process in an isolated environment on a shared host. A container can also be given limits on how much memory and how much of the processor it may use. So we run each submission in its own container. A correct solution to a problem like Balanced Brackets, together with Java itself, needs well under 256 MB, so we limit each container to 256 MB of memory. We also limit it to one core, so that each run uses at most one core. The host stops a container that goes over its memory limit, so one run cannot take the memory the rest of the application needs. The application still stops any run that goes longer than 5 seconds.
Many submissions at once
Here are my assumptions about how much work running code is:
- Starting a container and removing it afterward take about 1 second. Compiling one submission takes about 1 second, and running its tests about 1 more second. Most of each of those two seconds is spent starting Java, which the compiler and the tests each need. So one submission takes about 3 seconds of one processor core.
- On a normal day, students submit fewer than one solution a second.
- The application’s server has 8 cores.
Suppose one submission arrives every second. Each one keeps a core busy for 3 seconds, so at any moment three runs are in progress: the one that just arrived, the one from a second ago, and the one from two seconds ago. That is 3 cores busy. On a normal day, fewer than one submission arrives a second, so running code keeps fewer than 3 of the 8 cores busy, and the rest handle the other requests, such as reading a problem.
Now suppose that on the night before the data structures midterm, about 200 students practice at the same time. Each one submits about once every 40 seconds, the time it takes to read the results, change a line or two, and submit again. So 200 submissions arrive every 40 seconds. That is 5 submissions a second. Each one keeps a core busy for 3 seconds, so at any moment about 15 runs are in progress, and running code needs 15 cores. The server has 8.
The limits do not help here, because no single run is doing anything wrong. The runs need 15 cores and get 8, so submissions arrive almost twice as fast as the server can finish them. The unfinished runs pile up, and the other requests compete with all of them for the same cores. Reading problem 14 takes longer than one second, and Nadia’s results take longer than 10 seconds.
Recall the rule of thumb for scaling: use vertical scaling first, because it needs no change to the design. A server with 24 cores would have room for the 15 cores the runs need on the night before the midterm, with 9 left for the other requests. But the number of runs at once grows with the number of students who submit. If 300 students practice before the next midterm, 300 submissions arrive every 40 seconds, which is 7.5 a second. Each keeps a core busy for 3 seconds, so the runs need about 23 of the 24 cores, and only 1 core is left for the other requests. And for the rest of the term, when the runs keep fewer than 3 cores busy, most of those cores would be idle.
We could also duplicate the whole application behind a load balancer, as we did for the registrar’s system. This has the same problem the registrar’s system had. There, the application and the API grew at different rates. So every instance we added for the API also added a copy of the registrar’s application, and the application’s traffic did not need that copy. Here, the work of running code goes up in the week of an exam and goes down after it. The other requests take about the same work every day. And on every instance, a request to read a problem still competes with the runs on that instance. The two kinds of work are also deployed together. To move submissions to a newer version of Java, we would install it on every instance and redeploy the whole application, though nothing else changed. And a fix to how the application lists past submissions would mean redeploying the code that runs submissions too.
So we use the registrar’s fix, functional decomposition, and split CodeDrill into two services:
- The application handles the REST API, and keeps the problems and the submissions in the database.
- The runner runs submissions, each in its own container with limits on memory and the processor, and stops any run that goes longer than 5 seconds.
Each service is deployed on its own, so each has its own deployment boundary.
The runner
The runner has its own machines. Each runner instance has 4 cores. On a normal day, the runs need fewer than 3 cores, so one instance’s 4 cores are enough. We run two, so that a failure of one does not stop the runs. On the night before the midterm, the runs need 15 cores. We run five instances, 20 cores, so the runs still have 16 cores if one instance fails. The midterm date is known in advance, so whoever operates CodeDrill adds the three instances before the midterm and removes them after it. Recall that on most hosting platforms, this is a configuration change. A load balancer in front of the runner instances spreads the calls across them.
The application stays on its one server, and its 8 cores now handle only the other requests. Moving to a newer version of Java changes only the runner, and a fix to the application does not redeploy the runner.
Who loads the tests
To run Nadia’s code, the runner needs the tests for problem 14. The tests are in the database. There are two options.
The runner could read the tests from the database, given the problem number. The registrar’s two services both use one database, so this would not be new. But then the table that holds the tests becomes part of the interface between the two services, and a change to that table would have to be released in both.
Or the application could send the tests with the code. The application reads problem 14 from the database anyway, to check that it exists and to save the submission.
I choose the second option. The runner then needs no connection to the database, so we can block access to the database from the runner’s machines. This prevents malicious code in a submission from reading or changing CodeDrill’s stored data. And the runner does not depend on how the application stores tests, so a change to that table is released in the application alone. The cost is that every call carries the tests. For a problem like this one, the tests are a few kilobytes.
One service calls another
Recall how the registrar’s two services work. The gateway sends each client request to one service, and that service sends the response. The two services never call each other.
We considered one service calling another service in the same system once before. That was when we thought about moving the shop API’s business logic into a service of its own. The API layer would then send that service a request over the network while it handled the client’s request. I kept the shop API in one backend, so we never designed that request. Here we have to design it.
The application handles Nadia’s POST. While it handles that request, it calls the runner and waits for the results. For that call, the application is the client and the runner is the server.
graph TD
B[Nadia's browser] -->|POST /problems/14/submissions| APP[Application]
APP --> DB[(Database)]
APP -->|run the code| LB[Load balancer]
subgraph "Runner"
LB --> R1[Runner instance 1]
LB --> R2[Runner instance 2]
end