Goose, the load testing software created by Tag1 CEO Jeremy Andrews has had a number of improvements since its creation. One of the most significant improvements is the addition of Gaggles.

A Gaggle is a distributed load test, made up of one Manager process and one or more Worker processes. The comparable concept in Locust is a Swarm, and it's critical for Locust as Python can only make use of a single core: you have to spin up a Swarm to utilize multiple cores with Locust. With Goose, a single process can use all available cores, so the use case is slightly different.

As we discussed in a previous post, Goose Attack: A Locust-inspired Load Testing Tool In Rust, building Goose in Rust increases the scalability of your testing structure. Building Goose in this language enabled the quick creation of safe and performant Gaggles. Gaggles allow you to horizontally scale your load tests, preparing your web site to really take flight.

Distributing workers

Goose is very powerful and fast. Now, it’s so CPU-efficient it is easier to saturate the bandwisth on a network interface; you'll likely need multiple Workers to scale beyond a 1G network interface.

In our Tag1 Team Talk, Introducing Goose, a highly scalable load testing framework written in Rust, we introduced the concept of Workers.

In a Gaggle, each Worker does the same thing as a standalone process with one difference: after the Worker's parent thread aggregates all metrics, it then sends these metrics to the Manager process. The Manager process aggregates these metrics together.

By default, Goose runs as a single process that consists of:

  • a parent thread
  • a thread per simulated "user" (GooseUser), each executing one GooseTaskSet (which is made up of one or more GooseTasks)
  • a Logger thread if metrics logging is enabled with --metrics-file
  • a Throttle thread if limiting the maximum requests per second with --throttle-requests

Distributed in area, not just in number

Sometimes it's useful to generate loads from different places, clouds, and systems. With a Gaggle, you can spin up VMs in multiple clouds, running one or more Workers in each, coordinated by a Manager running anywhere. Typically, the Manager runs on the same VM as one of the Workers -- but it can run anywhere as long as the Workers and Manager have network access to each other. For additional information, see the Goose documentation on creating a distributed load test.

Summarized metrics

While metrics are usually collected on the Workers, a Gaggle summarizes all metrics from its tests by sending them to the Manager. A Goose load test is a custom application written in Rust, and instead of (or in addition to) printing the summary metrics table, it can return the entire structure so your load test application can do anything it wants with them. You can use these load tests and metrics to track the performance of your web application over time.

GooseUser threads track details about every request made and about every task run. The actual load test metrics are defined in the Goose metrics struct. Workers can send metrics to the Manager, which aggregates it all together and summarizes at the end of the load test.

By default Goose displays the metrics collected during the ramp-up period of starting a load test (ie, first there's 1 user, then 2, then 3, etc, until it starts all N users in the test) -- by default it then flushes all metrics and starts a timer so going forward it's only collecting metrics for all simulated users running at the same time. By default it then displays "running metrics" every 15 seconds for the duration of the load test. Finally, when the test finishes (by running a configurable amount of time, or by hitting ctrl-c on the Manager or any Worker) it displays the final metrics.

The parent process merges like information together; instead of tracking each "GET /" request made, it tracks how many of that kind of request was made, how fast the quickest was, how slow the slowest was, total time spent, total requests made, times succeeded, times failed, etc. It stores them in the GooseTaskMetric and GooseRequest structures. Individual GooseUser threads collect metrics instantly, and push them up to the parent thread. The parent thread displays real-time metrics for the running test every 15 seconds.

If the Metrics Log is enabled (by adding --metrics-file=), Goose can record statistics about all requests made by the load test to a metrics file. Currently, each Goose Worker needs to retain their own Metrics log and then they could be manually merged together -- the Manager can not store a Metrics log (if you are interested in the Manager storing the log, follow this issue.) The metrics collected or displayed are the same whether stand alone or in a Gaggle. The documentation on Running the Goose load test explains (and shows) most of the metrics that are collected.

Automated tests confirm that Goose is perfectly tracking actual metrics: the test counts how many requests the web server sees, and confirms it precisely matches the number of requests Goose believes it made, even when split across many Gaggle Workers.

Gaggles in use

During development, Jeremy spun up a Gaggle with 100 Workers. The coordination of the Gaggle itself has no measurable performance impact on the load test because it happens in an additional non-blocking thread. For example, if we start a 5-Worker Gaggle and simulate 1,000 Users, the Manager process splits those users among the Workers: each receives 200 users. From that point on, each Worker is doing its job independently of each other: they're essentially just 5 different stand alone Goose instances applying a load test and collecting metrics.

Communication between the Workers and the Manager is handled in a separate, non-blocking thread. The Manager waits for each Worker to send metrics to it, and aggregates them, showing running metrics and final metrics. For specifics on how the Workers and Manager communicate, read the technical details.

Once a Worker gets its list of users to simulate it tells the Manager "I'm ready". When all Workers tell the Manager they're ready, the Manager sends a broadcast message "Start!" and from that point on Workers aren't ever blocked by the Manager.

Verified consistency of Workers

When running in a Gaggle, each copy of Goose, meaning the Manager and all the Workers, must run from the same compiled code base. We briefly explored trying to push the load test from the Manager to simpler Workers, but Workers run compiled code, making this incredibly non-trivial.

Running different versions of the load test on different Workers can have unexpected side-effects, from panics to inaccurate results. To ensure consistency, Goose performs a checksum to confirm Workers are all running the same load test. Goose includes a --no-hash-check option, to disable this feature, but leaving it enabled is strongly recommended unless you truly know what you're doing.

Taking flight

Using Gaggles with Goose can significantly increase the effectiveness of your load tests.

While standalone Goose can make full use of all cores on a single server, Gaggles can be distributed across a group of servers in a single data center, or across a wide geographic region. Configured properly, they can easily and effectively use whatever resources you have available. Goose tracks the metrics you need in response to those Gaggles, helping you now, and in the future.

Photo by Ian Cumming on Unsplash