Load Testing Results

Once the load test is done, we can read the results and see if HopPress meets its performance and scalability requirements.

At each load there are thousands of response times. Should all of them be under two seconds? That is one way to read the requirement, and it is a strict one. A few requests may happen during a garbage collection pause or a slow disk read.

So if the two-second requirement is not about every request, what is it about? Is it the average response time? Suppose 95 of every 100 requests return in half a second and 5 take ten seconds. The average is about one second, which is under the target, but five readers out of every hundred waited ten seconds. Is that acceptable?

If you choose to use the average as a measure, keep in mind that it hides the long tail of slow requests.

Percentiles

When we measure response time, we are typically interested in the slowest requests, not the average. So we look at the distribution of response times. A load-testing tool can show a histogram of the thousands of response times, and it can calculate percentiles. A percentile tells you the value that a given percentage of the observations fall below.

The 95th percentile, written p95, is the response time that 95 percent of requests complete within. If p95 is 1.8 seconds, then 95 percent of requests finished in 1.8 seconds or less and 5 percent took longer. p99 is the time that 99 percent of requests complete within, so only the slowest one percent take longer. p50 is the median.

Reading the requirement

I will read the requirement as p95: 95 percent of requests to open a published post return in under two seconds. That is a clarification of the requirement, not a change to it. We should update the requirements specification to say so. Clarifications like this come up often once you start measuring.

The result

Suppose we run the load test and get the following results.

At 50 simultaneous readers, p95 is 1.1 seconds, under the two-second target.

xychart
    title "Response time to open a published post, 50 readers"
    x-axis "Percent of requests finished" [5, 15, 25, 35, 45, 55, 65, 75, 85, 95]
    y-axis "Response time (seconds)" 0 --> 4
    line "response time at this percentile" [0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.55, 0.7, 1.1]

As the load increases to 500 simultaneous readers, p95 rises to 3.3 seconds, past the two-second target.

xychart
    title "Response time to open a published post, 500 readers"
    x-axis "Percent of requests finished" [5, 15, 25, 35, 45, 55, 65, 75, 85, 95]
    y-axis "Response time (seconds)" 0 --> 4
    line "response time at this percentile" [0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.7, 2.0, 2.5, 3.3]

HopPress meets the performance requirement at 50 simultaneous readers and does not meet the scalability requirement at 500.