The Cheap Fixes

HopPress does not meet its scalability requirement. Before changing the design, there are two things to check. Both cost little, and either one is often the reason a load test fails.

Connection pooling

Every query HopPress runs goes over a connection to the database. Opening a connection has a cost. The application and the database perform a network handshake, the database authenticates the application, and the database sets aside memory and often a process or thread to serve that connection. That takes tens of milliseconds, before the first query is sent.

A simple application opens a connection when a request arrives, runs its queries, and closes the connection when the response is sent. Under a light load the cost is too small to matter. Under 500 simultaneous readers, the application is opening and closing hundreds of connections per second, and each one adds its setup cost to a request’s response time.

There is a second problem. A database accepts a limited number of connections at once, often around a hundred by default. When the application tries to open the hundred and first, the database refuses it or makes it wait. At high load, requests fail or queue for a reason that has nothing to do with the query they wanted to run.

A connection pool fixes both. The application opens a fixed number of connections when it starts and keeps them open. When a request needs the database, it borrows a connection from the pool, runs its queries, and returns the connection. No request has to open a connection, and the database never has more connections open than the pool holds.

The pool has a size, and the size is a design decision. It is not set to the number of readers. It is set to what the database can serve at once, which depends on the machine’s cores and disks and is usually a small number, tens rather than hundreds. When every connection in the pool is borrowed, the next request waits for one to be returned. That wait is part of its response time. So a pool that is too small makes requests wait in the application for a free connection, and a pool that is too large makes queries wait inside the database for processor and disk time. Both show up in the load test as rising latency.

Most application frameworks pool connections by default. Check that HopPress’s framework does, and check the pool size. If it does not, adding a pool is a configuration change, not a design change, and it is the first thing to try.

Vertical scaling

The second cheap fix is a bigger machine. Vertical scaling means running the same one server with more hardware: more processor cores, more memory, faster disks. The application and the database do not change. The same monolithic deployment from Chapter 3 runs on better hardware.

For HopPress, more cores let the web server and the database handle more requests at the same time, and more memory lets the database keep more of posts and its indexes in memory instead of reading them from disk. Both lower response time at a given load.

A bigger machine may be enough. If the load test passes at 500 readers after the upgrade, the requirement is met, and that is a reasonable place to stop. You reach the limits of vertical scaling when the load keeps rising and each rise needs another upgrade.

The first limit is price. Hardware prices do not grow in proportion to capacity. A machine with 64 cores costs far more than four machines with 16 cores each. Each upgrade buys less capacity per dollar than the one before it, so at some point you would serve more readers by spending the same money on several smaller machines.

The second limit is the amount of hardware one machine can hold. Once you reach that limit, the only way to get more is to add more machines. That is horizontal scaling, which we will cover in a later chapter.

The other option is to change the design so each request does less work, and meet the requirement on the same hardware.