Load Testing: An Unorthodox Guide

The server, well, runs the web application that you want to load test. You don’t have to think of it as just a single server, in fact, this can range from a tiny web server, a web server + a database, to a whole microservice landscape - more on that in a bit.

A common shortcut is to generate the load on the same machine (i.e. the developer’s laptop), that the server is running on.

What’s problematic about that? Generating load needs CPU/Memory/Network Traffic/IO and that will naturally skew your test results, as to what capacity your server can handle requests.

Hence, you’ll want to introduce the concept of a loader: A loader is nothing more than a machine that runs e.g. an HTTP Client that fires off requests against your server.

A loader sends n-RPS (requests per second) and, of course, you’ll be able to adjust the number across test runs. You can start with a single loader for your load tests, but once that loader struggles to generate the load, you’ll want to have multiple loaders. (Like 3 in the graphic above, though there is nothing magical about 3, it could be 2, it could be 50).

It’s also important that the loader generates those requests at a constant rate, best done asynchronously, so that response processing doesn’t get in the way of sending out new requests. Check out https://github.com/jetty-project/jetty-load-generator for such a load generator.

Bonus points if the loaders aren’t on the same physical machine, i.e. not just adjacent VMs, all sharing the same underlying hardware. Read this article on problems with co-located VMs.

Last but not least, you can split responsibilities even more, by not letting your loaders record latencies (e.g. response ms), instead you’ll use a probe for that.

A probe is a machine that represents, essentially, a single user.

It does not go crazy with generating load, Instead it slowly, but surely, fires off maybe 1-2 RPS and, while doing so, records the latencies that you’ll later on evaluate.

Do note, that there are differing opinions on splitting loaders and probes, but it’s the approach I’m going within this guide. If you made different experiences, do leave a comment in the comment section!

The biggest mistake you can make is to fire off your initial load test against your whole microservice universe - unless you have internalized the process that is yet to follow in this guide.

So, a sensible approach would be to:

Along the way, do try and note who the weakest link in your chain of services might be. It doesn’t help to auto-scale your EC2 web servers into oblivion if your t2.micro database can only handle 3 SQL statements at any given time.

A lot of benchmarks posted online of services and scale are of the synthetic kind: Look, I have this reactive web service and it handles 1.000.000 requests/second.

That’s all nice and dandy, but doesn’t help us when doing real-life load testing.

Unless you’re building a 1-REST-method-service, you’ll need to think of the most important scenarios and workflows your users are going through. For an e-commerce system, that would be browsing product pages and using checkout, including all the backend processing - not just endlessly calling the password-reset REST API.

If you have historical data to use (aka where my users spend most of their time on), great, use it. If not, take an educated guess and adapt as soon as you can.

There is an exception to the rule of "real-life user-journeys", however. As long as you’re still uncomfortable with recording and evaluating the data (from recording and interpreting latencies, flame graphs, and GC logs), try out the load-testing process with a toy example, which only has one REST method and serves bogus JSON data.

How long should your load-tests run for? 5 seconds? 60 minutes? Exactly 42,75 seconds?

Frankly, there are a plethora of opinions out on the web on what the perfect timespan is. You can start with this rule of thumb:

What’s more important than having very specific durations, is to repeat your test runs, and cross-check the results across those runs. If you start getting entirely different results with (almost) the same parameters, that’s a reason to investigate.

Another important point is not just to take a look at your recorded latencies. Instead, you need to verify each load test run.

What does that mean? You’ll need to find an answer to the following questions:

If you answer yes to any of these questions, ignore your test results.

The hard part about this is having the consistency and discipline of validating all the data for every. single. test. run and not just jumping to conclusions because a load test finished and you see a graph printed out somewhere.

To verify the data, we need to first collect the data. Let’s see in the next section, what data specifically, how you can do that, and in what output format you’d generally like to see the data.

You’ll want your probe to record its request-response times for every test run and then figure out how the latencies changed across runs, once you start cranking up the load.

For that, it makes sense to not just record raw data (e.g. calling the /tax_rate endpoint of my REST application took [50ms, 51ms, 37ms, 49ms, …​, etc]), but to visualize those response rates in a High Dynamic Range Histogram.

Here’s a histogram that shows the probe’s latencies across multiple (in microseconds, divide by 1000 to get ms) load test runs, where loaders hammered the server from 1000 RPS → 50000 RPS.

The histogram, with just a single glance, will give you a quick idea of what your response time percentiles (90%, 99% etc..) look like and also how those response times change, with varying loads.

You’ll also want to have a general overview of the throughput of both, your loaders and your server and again, visualize the raw data in a line chart.

If you tell your loaders to send 4000 requests per second to the server, the throughput (i.e. sending those 4000 requests every second ) should, in fact, be 4000 requests per second, and not just 2314 requests. Which will result in a pretty constant line chart like seen below.

If that chart doesn’t look level, it means your loaders had issues generating that many requests and you must reduce the load and possibly spawn new ones.

Similarly, if your loaders are doing just fine, but your servers throughput chart looks like this:

You immediately know that your server has issues responding to that many requests, is definitely at its limit in one regard or another and you’ll need to cross-check with all the CPU/memory/application data below to find out what the culprit is.

This sounds so simple, but if forgotten leads to wrong conclusions. For every HTTP request that is sent during your load test, you should record its HTTP response status code. Was it all 200s? Good.

Were there bad requests (400s), because an API changed, or maybe even server errors (500s), because a pesky bug entered the backend? Then you’ll need to throw away your test results and re-run the test.

Above you’ll see a custom text format (taken from the Jetty Perf Project), that’ll quickly display all the status codes of all HTTP requests that were issued in a specific second of the test.

This is how you read those lines, e.g.

This will depend on your operating system, but if you are e.g. using Linux machines, you can gather data on your CPU in specific intervals with the mpstat command (5s intervals in this case)

Assuming this command ran for 15 seconds (3 intervals a 5s) on a machine with 4 CPUs, you’d get output like this:

Every table above corresponds to a CPU load snapshot, either across all CPUs (averaged), or the individual CPUs (0,1,2,3 = 4 CPUs).

For a quick & dirty start you can simply take the inverse of the %idle column from the all line, to find out how busy your CPUs on average were during that snapshot (e.g. 100%-82.31% = 17,69%).

Though you naturally want to also have a look at individual CPU performance, especially for scenarios where only one of your CPUs was maxed out, while all others were idling.

Again, depending on your operating system you’ll want to gather snapshots of your network traffic with different commands. On Linux, you could be using the sar tool for network snapshots every 5 seconds.

The output will be a file like this, where two tables correspond to one snapshot.

Important are the columns rxkB/s, as well as txkB/s, which, if multiplied by 0.008, give you the megabits of data your network interfaces receives per second, as well as it sends out.

Looking at the second snapshot above that would be

(Do note that networks adapters are usually duplex, hence if you have a 100MBit interface, you’re not only able to do just 50Mbit/50 Mbit.)

All the concepts mentioned above for CPU and network are also valid for memory. On Linux machines, you can capture memory snapshots every 5 seconds with the free tool.

You’ll receive output like this, again, each table corresponding to a specific snapshot.

On a glance, you’ll see how much memory was free/used, from the total available amount.

Will be added in the next revision of the guide.

This depends a bit on the web server & frameworks you are using, but essentially, you’d want to collect response times for requests excluding client latency and excluding parts of the system you cannot influence, e.g. your web server’s HTTP request handling.

In short: How much execution time does your business logic take? How long does your controller need to prepare a JSON response from an SQL query?

Again, you’ll want to visualize the data in a histogram, to get a quick understanding of request latencies and percentiles, across test runs.

In simple terms, Flame Graphs allow you to visualize where your application, or rather its code paths, spends its CPU & memory.

The example below is a Java CPU flame graph, though you can generate them for a variety of programming ecosystems and flame graphs not only show you your application’s code paths (green color), but also where CPU/memory is spent in terms of the JVM/C++ (yellow), operating system (red) and even the kernel (orange).

To learn more about flame graphs, read this excellent page by Brendan Gregg, the inventor of flame graphs.

For garbage-collection platforms like the JVM, it also makes sense to record garbage collection logs and/or use an instrumentation tool like jHiccup, which records JVM stalls.

Graphs/Examples will be added in the next revision of the guide.