Choosing the right load testing tool - JMeter vs. Locust vs. Goose

Load testing is a critical part of the quality assurance process in any business website. Knowing how your website behaves under load, and being sure it will stay up can make the difference between success and failure for your business. Retail websites require this for major events such as Black Friday and Cyber Monday. News outlets need to be sure that their readers can reach them and they can respond during major news cycles.

Choosing the right load testing tool is the first step. Being aware of what your tool does and doesn’t do well, understanding what recurring needs you’ll have for testing, and who will be doing the testing should inform your choice of software. We’re going to look at three different tools: JMeter, Locust, and Goose. All three tools are open source.

JMeter

JMeter is a load testing tool created by the Apache Software Foundation. Written in Java, it is easily one of the most popular open source load testing tools available today. It has the largest community of users and contributors of the tools covered here.

JMeter was originally designed to test web applications' performance and functionality, and has expanded over time to include other features and protocols. This makes it useful for many levels of users. with varying needs. JMeter supports many common internet protocols, giving it one of the widest coverages of any load or performance testing software.

One of the most user-friendly features of JMeter is its interface. Many load testing tools are command-line only, and having a UI can make your tool more approachable for general users. The JMeter UI can also load in logs generated during load tests to provide reports.

However, the user interface has a reputation for being clunky and difficult to use. While the UI makes it easier for users with less experience coding to create and run tests, the tests themselves are written out in XML and it can be difficult to create more complex tests with the limitations imposed by the UI. Many tools are available for working with XML, but the files themselves tend to be large, and it can be problematic for non-developers to make manual changes because they simply can’t find what they need to change. JMeter is designed to be UI-first, which can make it more complicated for devops teams who just want to have a set of quick, consistent tests they can modify quickly to run on every instance, or want to make small changes to their tests. Developers can find it difficult to create the tests they want through a point and click interface.

JMeter takes a lot of system resources to run. It’s a thread bound system, requiring a separate thread for each simulated user. This significantly limits actual load test capabilities in terms of numbers of users. Distributed load testing is possible, but fairly difficult when using the JMeter platform.

Locust

Another popular load testing tool, Locust is a mostly command line tool written in Python. Load tests are distributable and scalable, making it possible to ramp up a test simulating thousands of users quickly. It has the advantage that it’s leaner than JMeter, and doesn’t require large configuration files. For testers who prefer a command line interface, Locust is faster, easier to configure, and it enables testing code to be easily reused between projects. This can be a time and money saver for anyone dealing with testing and infrastructure. Tests are written in Python, and can be stored in your version control system of choice, along with your project, making it easier to re-run the same tests in the future.

While Python is a language that lets you write code quickly, it also has limitations. The major limitation for scalability is Python’s global interpreter lock. Locust makes the effort to work around this limitation by allowing you to spin up a distributed load test, or Swarm, but that adds complexity to running even smaller load tests. The Python language is also slower, relatively, than other languages. This means it requires more CPU power to simulate traffic, and imposes artificial limits on how much traffic a test can easily simulate.

Locust includes a web-based user interface for running and monitoring tests, so you can see your results in real time. You can choose not to use the UI, which is better for continuous integration and delivery (CI/CD) purposes. In these instances, Locust can be used as a library. It uses the Python logging facility for its own logs, enabling you to process them as you would any other log written by a Python program.

Locust doesn’t currently support the same level of testing protocols that JMeter does, but users can write plugins for their own protocol needs. A number of additional plugins are already available.

Locust supports distributed testing, with a manager/worker model. Each manager and worker instance must have its own copy of the test scripts. In this mode, the UI only runs on the manager instance. Locust can scale up with fewer system resources than JMeter, as it supports asynchronous requests. Locust documentation explains, "a common set up is to run a single [manager] on one machine, and then run one worker instance per processor core on the worker machines."

Goose

We created Goose at Tag1 Consulting to improve on what's possible with JMeter and Locust. Goose is inspired by Locust, but as it's written in Rust it makes drastically better use of available system resources. More insight into the design goals and inspiration for creating Goose can be found in the blog post Goose Attack A Locust-inspired Load Testing Tool In Rust.

Goose generates at least 11x as much traffic as Locust per-CPU-core, with even larger gains for more complex load tests (such as those using third-party libraries to scrape form content). While Locust requires you to manage a distributed load test simply to use multiple CPU cores on a single server, Goose leverages all available CPU cores with a single process, drastically simplifying the process for running larger load tests. Ongoing improvements to the codebase continue to bring new features and faster performance. Goose scales far better than Locust, efficiently using available resources to accomplish its goal. It also supports asynchronous processes enabling many more simultaneous processes to ramp up thousands of users from a single server, easily and consistently.

User behavior is defined with standard Rust code, and load tests are built by creating an application with Cargo, and declaring a dependency on the Goose library.

Goose’s distributed testing design is similar to Locust’s, in that it uses a one Manager to many Workers model. However, unlike Locust, you do not need to spin up a distributed load test to leverage all available cores on a single server, as a single Goose process will fully leverage all available cores. Goose distributed load tests scale near-perfectly as once started each Worker performs its load test without any direction from the Manager, and the Manager simply collects statistics from all the Workers for final reporting. In other words, one Manager controlling eight Workers on a single 8-CPU-core server generates the same amount of load as a single standalone Goose process independently leveraging all eight cores.

Goose has a number of unique debugging and logging mechanisms not found in other load testing tools, simplifying the writing of load tests and the analysis of results. At Tag1 this has helped us quickly troubleshoot problems that would be extremely difficult or impossible to detect with tools such as Locust or JMeter. Goose also provides more comprehensive metrics with multiple simple views into the data, and makes it easy to confirm that the load test is doing what you expect it to as you scale it up or down. It exposes the algorithms used to allocate tasks and task sets, giving more granular control over the order and consistency of operations, important for easily repeatable testing.

Goose is also designed to be used as a library, making it easier to integrate into CI/CD deployment schemes. Ensuring performance as part of your regular testing processes can make or break your website.

At this time, the biggest missing feature of Goose is a UI for controlling and monitoring load tests, but this is a work in progress. A recently completed first step toward this goal was the addition of an optional HTML report generated at the end of a load test.

If you've ever found yourself struggling to generate enough load with Locust, adding the complexity of setting up a distributed swarm just to properly leverage all CPU cores, then Goose is worth a look. Take a look at the many included examples to better understand how to write a load test in Rust.

Feature breakdown

This chart compares the distinctive features of the three tools.

There are a lot of JMeter versus Locust comparisons, as both tools have been available for a number of years. Goose is new enough that there aren’t many comparisons between it and other load testing systems, but we hope to see more adoption and comparisons in the future.

Summary

All of these testing tools are powerful additions to your company’s deployment plan. Too regularly, websites go down because some part - from the application, to the database, to the network - wasn’t able to handle the load put on it.

Load testing is a safety measure. It is important to identify bottlenecks during testing, allowing you to fix problems before they affect your users and the bottom line. By integrating load testing into your development process, you can avoid application outages when it matters. Solving problems by optimizing before deployment, ensuring you have optimized the fast paths, and being aware of the need to spin up hardware before the overload point can make the difference between a stressful or (preferably) boring day for your operations team.

If you’re looking to create a simple load test without the need for coding, JMeter may be the choice for you. If you need to generate a moderate amount of traffic, or you need some UI tools to run and monitor your tests, Locust is a good choice. But, if you need to generate a lot of traffic with consistency, and have confidence that you’re generating the exact same amount of load every time, professionals should choose Goose.

Related links:

• JMeter vs. Locust - Which One Should You Choose?
• Let Me List Them Out For You 'Locust Vs JMeter'
• Drupal Load Testing with Locust.io
• Distributed Load Testing With Locust.io

See our other Goose content to learn more about this highly scalable load testing software.