Transcript: How to load test with Goose - part 1: D9 Umami on Pantheon with Fastly

This is a transcript. For the video, see How to load test with Goose - part 1: D9 Umami on Pantheon with Fastly.

[00:00:00] Michael Meyers: Hello, and welcome to another Tag1 Team Talk, the podcast and blog of Tag1 Consulting. Today, we're going to be talking about how to load test with Goose. It's going to be running in AWS and we're going to test against Drupal 9 With the default Umami install against a Tag1 Kubernetes cluster fronted by the Fastly CDN. I'm Michael Meyers and managing director of Tag1 Consulting. And I'm joined today by Jeremy Andrews, the Founder and CEO of Tag1, along with Fabian Franz, our VP of Technology. Let's jump right in. I'm excited to see how Goose works. Jeremy, welcome. Fabian, welcome. Thank you guys so much for walking us through this.

[00:00:41] Jeremy Andrews: Thank you. Yeah, I'm going to start off by sharing my screen. So if you're watching this on a video, you'll be able to see what we're doing as we go. To start off quickly looking at Goose itself Goose is written in Rust. And if you go to Goose.rs, it'll take you to the GitHub page, and inside there you'll find a pretty normal looking Rust code base. For today, the part we care about is within the examples folder and we've written several examples, primarily to demonstrate how Goose can be used. The first one was just called ‘simple’ and all that does is it loads an endpoint and shows you how to write the most simple of load tests. It uses a few features like wait times.

[00:01:21] So it'll load a page weight randomly between five and 15 seconds. and then load it index page and about page and whatnot. It does simple stuff. It logs in. Then there's another version of that, where it was written to use closures, a fancy feature of Rust, and Fabian wrote that version. I find closures more difficult to understand, but they're very, very flexible for programmatically doing things.

[00:01:43] So it's, it's a super cool example to have in there.

[00:01:46] Fabian Franz: So essentially what you're doing with the closure is don't even know what a closure is, but what you can see is. There's a pass array at the top or vector, how it's called in Rust. And you can essentially just extend this vector, whatever pages you want.

[00:02:02] You can draw that from a CSV, from wherever, and you can have a programmatic load test, essentially.

[00:02:10] Michael Meyers: I mean, even if you don't know too, too much about Rust, just, just looking at these, you know, you can go to this examples, directory, cut and paste, make some modifications and, you know, simple modifications and use this to test your own Drupal website in a pretty sophisticated fashion.

[00:02:25] Jeremy Andrews: This one is not as helpful yet. Conceptually it helps you with closures, but then there's one called Drupal load test. And that was literally written to load test every release of the memcache module. Originally, we were doing it in JMeter and then we replaced it with Locust and now we use Goose and every time it's a huge step forward, but this one has a much more Drupal examples and much more you can cut and paste. That said, I believe it was Fabian who suggested we should load test against Umami. So if, if your goal is to write a load test for a Drupal 8 or Drupal 9 website, this is absolutely the example to look at it's broken into useful pieces, common.rs are functions that you can use, you just call directly from, from your own load test.

[00:03:10] And it does a lot of things that aren't generic to Drupal websites. Cause you can see here, it's defining a list of all nodes and whatnot in Umami. But in any case, the load test we used when we've written a load test for customers we've borrowed heavily on this code base because it is a great starting place.

[00:03:25] Fabian Franz: I actually love that you, you, you finished this Umami load test because one of the things the Drupal community has been looking for since at least 2015 is essentially a profile which we can, can do for performance, regression, testing and core. And this now would essentially allow that to just spin it up somewhere, load test it, put some New Relic on it and then essentially we could look at what's slow, what's fast, did we regress, et cetera. So it's not only useful for memcache, but with Umami being finally complex enough to resemble a real site I think we could get some meaningful data out of it for future performance work in core. So that's really cool.

[00:04:09] Jeremy Andrews: It's funny you should mention that. I spun this up to test earlier this week, and I found that there are certain paths that are struggling under the amount of load that we're putting against our little install. And so there is some functionality that was being tested that I disabled on the load test to not, to not load those paths.

[00:04:30 For example, I disabled searching. The load test does searching in English and in Spanish, and both are just disabled in our current test filling out the contact form. And I also disable the user that logs in and basically just logs in, edits an article and saves it, flushing the cache.

[00:04:48] Those are the things it'd be great at some point. And maybe on this series, we'll, we'll do that. We could dig into why we had to disable them, how to optimize and fix it so that we don't have to, and everything can be tested.

[00:04:59] Fabian Franz: Essentially what we are testing today and what we're seeing today, as we are doing the big event scenario.

[00:05:05] But essentially you get featured on Slashdot, on Reddit, on wherever, like, and people are just crashing your site and all coming in, like, like angry Geese. And and essentially they are all wanting to get the recipes from Umami because they are so hot right now. I mean, they're, they're perfect.

[00:05:23] And because of that, and we cannot afford at this stage essentially to allow people to do things that are like, really complex. So what often happens in the real world on a scenario when a big day like Black Friday comes in, is that you actually disable or rate limit functionality that is very costly on the site like searching.

[00:05:44] And you'll also tell your editors, Hey, please prepare all content before this date and do as less content changes while it's going on, because it always needs to clear some caches. And obviously that can lead to a worse performance, but that's pretty cool. So, this is the big events scenario. We have set up here,

[00:06:06] Jeremy Andrews: So let's give it a run.

[00:06:08] What we're going to see is, well, first of all, Goose is gonna run from an instance in AWS. We just spun it up here about 15 minutes ago. We'll spin up some other instances as we test to demonstrate how different hardware can perform different tests. But a high-level summary Goose makes really, really good use of available CPU power and.

[00:06:29] Ultimately it consistently tends to bottleneck on bandwidth because it can generate as much load as your uplink supports. So that's, you know, that's a good thing that that's what we're trying to do here and you can of course control it. You can you don't always have to just flood things with as much traffic as possible.

[00:06:45] But for today, that's what we're most interested in is just seeing what we can do, what traffic we can throw at it.

[00:06:50] Michael Meyers: High level specs on this box. Like how many cores is this thing?

[00:06:54] Jeremy Andrews: This is an eight core server. It's Amazon's CFA two X large, which should answer all your questions. The two X large means it's got eight CPU's and the CF line is pretty powerful with CPU.

[00:07:05] I actually chose it. This particular one, because eight cores is what I generally use when I'm, when I'm load testing. And I wanted something with a reasonably fast uplink and this one has, I think it says up to, I think it was 10 gigabits or maybe it's five gigabits. but it's enough to put some significant load and to show that, you know, Goose is actually bottle-necking on uplink, not on available CPU resources. So the public IP, this is up and running. So I already SSH into this and the, in the Goose directory on this server is just the Git checkout of the current head of Goose the, the, the main branch. And I'm going to see if I can quickly find in my history, the commands that I run.

[00:07:48] So we'll start off with just a simple thousand users. This is an SSH into this AWS instance and we're going to just launch a very small Goose load test a Goose attack against Umami website that we're running to quickly talk through the, the flags, cargo run is because instead of compiling a binary and then running the binary I'm using cargo to manage that, which allows me to tweak the code and whatnot and recompile.

[00:08:15] The --release is important to optimize away debug symbols. Any load tests you're running, you're going to want to use --release. --example is because we're just running an existing example that we've provided the Umami one. And then the --space --host is where we actually start configuring Goose itself.

[00:08:33] We're no longer in Cargo, but we're in Goose. We're we're load testing against umami.tag1.io. The -v will give us a little bit of verbosity. You don't want to go past one -v, because then it gets too verbose. It starts showing verbosity from all the libraries and whatnot, and that, that will impact your results because it's a lot of overhead.

[00:08:51] But one, one -v is great. --log file basically takes everything in -v and dumps it to a log file which we're defining as Goose.log. And that's because of the -g that follows, then we have a --debug file to a debug log because if something goes wrong Goose will capture everything, all the headers, everything that was requested and everything that was returned and gives us a ton of insight into what's going wrong.

[00:09:16] Whether there's something's bottle-necking or there's something wrong on the server or, or who knows. but we've found that unbelievably helpful in actually troubleshooting with customers. -t 1m means that this is only gonna run for one minute and then it's going to stop itself.

[00:09:29] -u 100 is we're gonna simulate 100 users and the -r 3.4 means that we're starting 3.4 users per second. So we'll go ahead and give that a try and See what happens? There's a couple of warnings. I've commented some code. So it starting is showing users launching in English and Spanish.

[00:09:49] So now if we go to Fastly momentarily, we should start seeing some traffic and we would expect it. The server is actually located on the West coast of the United States. So we'd expect to see traffic there. It obviously saw something because it flushed it. There we go. So now it's seeing right here traffic coming from Seattle and We're currently doing 140, 160 requests per second.

[00:10:12] It's still ramping up. Down below you can see that it's ramping up to 95 megabits a second. So there's traffic happening. if we look on, on the server itself, we can see that Goose knows it's running. it has successfully ramped everything up. And so it's going to run for 60 seconds. Just generating a little load. 60 seconds is only enough for this demo. But you can see the things generally stabilize and and you know, it's pulling stuff out of the cache. What's really important here. This hit rate going up, this is the cache warming up and that's going to be essential to, to, to scale up this test and put serious load against it.

[00:10:47] You can see here that you know, we're capping out at about 350, 380 requests per second with the a hundred users that we're simulating there's more requests than users, obviously, because there's a, it's not just re requesting the index file, but it's also requesting elements from the page.

[00:11:03] It's the CSS and the images and whatnot. So all of that's loading the load test finished. There's a delay in Fastly. You saw it was a delay to start up. It's also a delay to go down. if, if we look at the results, you can see that everything ran. It prints the final statistics There's, lots of statistics, there's flags you can use to change what it shows you. It breaks it down per task, showing you that, you know, we've been loading the front page. A basic page, different articles, different recipes, different nodes by node ID shows you how many times each was done, shows you the zero failures. it was a total of 39.8 tasks per second.

[00:11:37] And then you can also see how long each took so you can kind of get a sense of which tasks are taking longer. The recipe page is taking about a second to load, which is, which is fairly slow for a webpage, especially coming out of a CDN any ideas on why it's that slow, Fabian, would that be because of the amount of content on the page?

[00:11:52] Fabian Franz: No, it's probably it might've been the first request, like on a cold cache?.

[00:11:57] Jeremy Andrews: We've got a minimum and a max. The minimum is still 972 milliseconds.

[00:12:03] Fabian Franz: You've probably already exhausted the bandwidth of the Amazon instance.

[00:12:07] Jeremy Andrews: Definitely not. we'll we're gonna ramp this up considerably. we can get you about 2000 users before we cap it out.

[00:12:13] Fabian Franz: Then it doesn't make sense.

[00:12:18] Jeremy Andrews: So my thought process is that it's Hmm. Okay. Well, let's see per requests, so sorry. So that's per task. Okay. Actually here it is what I was thinking because per requests for requesting slash it was let's see here. Yeah, it's super, super fast. absolute maximum of 148 milliseconds where it gets.

[00:12:38] The, the reason it's slower is because making so many requests in those tasks. And even though it's fast, that's going to probably include sleeps and that's going to include while it's giving up to let another user use the box. so that's a combination of that user doing the entire page load task and it would be slower.

[00:12:55] The actual requests themselves are very, very fast, like we would expect anyway. it's not very exciting. But it gives you a quick demonstration of what this does. So the general idea here then is, you know, now that now that we've seen what it can do, let's Let's increase it. So all we're going to do is we're going to go from a hundred users to a thousand users, a 10 X speed up, and we're going to go instead of starting 3.4 a second, we're going to go ahead and start 34 second.

[00:13:20] Important to note that you know, the, the resources it takes to start that up it's still has it has no trouble with that. That reminds me one thing that I should be doing here that's useful. I'm, I'm exiting out just so that I can cut and paste this string and connect in with a second window.

[00:13:35] And the reason is on that second window, what's helpful is if we run a SAR command. So like every 15 seconds, when you create a snapshot of what's going on that'll give us a little more insight into what's going on in the box. for that we're only going to be using. The CPU stats. but that was what I found was the most telling.

[00:13:52] So we'll go in and do this again with a thousand starting at 34 seconds. It would help if I was in the Goose directory.

[00:14:00] All right. The first thing it does is it builds a bunch of stuff in memory. now that it's built, it's launching them. If we switched to this view, we can see, we'll see that it starts using more and more CPU as, as this spins up. It only does a snapshot every 15 seconds and soon we should start seeing more considerable traffic hitting the end point here.

[00:14:22] All right. There we go. Traffic starting to hit. It's going to ramp up considerably faster. And what's one thing I noticed when I was testing this earlier, when you throw 10 times as much, 10 times as many users at a server, Goose throws 10 times as much traffic. So it's, it's pretty simple math. You're going to see 10 times or slightly more requests per second and you know, 10 times as much bandwidth happening.

[00:14:45] Fabian Franz: What's really important for me here is that you are, that we can actually see like a line going up for this because this line is so important for me when I'm doing performance testing, it's what I call the scalability test. Because, and we can do that in another [00:15:00] session, if you combine this test with New Relic, essentially, and we then do a scalability test.

[00:15:06] There's a very slow ramp up. You can actually see where a site starts to get keeping trouble, until when it performs, because that is when your performance goes from linear. Or even parallel not impacted to exponential, and that is when you get problems. So that's a great way to do it. That's why we have this ramp up variable because it's so, so useful in testing performance.

[00:15:31] Jeremy Andrews: So the load test just finished. it's sustained 1.3, 1.4 gigabits per second. it didn't, you know, those eight CPU's you know, they're 60% idle, 55, 60% idle. So the server has way, way more, capacity to continue generating more load. But the, the sheer amount of traffic is, is going to be our, our breakdown point when we try to keep ramping up the load test. Let's go ahead and quickly go back and look the per request metrics. You can see that at this point we were making 3,300 requests per second.

[00:16:01] None of them failed. The majority of them are static assets which, you know, we're loading off all the pages that we're loading and then the actual page load times continued to be quite fast. An average of a little over eight milliseconds.

[00:16:13] They stay this fast because they're coming out of Fastly. They're not coming out of out of the end point. But yeah the load test is continuing to look quite good. So the next thing we'll do is we'll double the ramp up again to 64, and we'll go from a thousand users to 2000 users. and see what happens when we double the load.

[00:16:33] As before it takes a moment while it's creating user states and it takes a little longer because it's creating twice as many, it's an area I want to optimize. and now it's ramping them up. And again, the server has no problem. Even though we're doing twice as many per second again, the server can handle this.

[00:16:50] And soon we'll start seeing traffic hitting on Fastly

[00:16:59] And boom, there it goes. there's a delay obviously for the reporting side of things, but once again, you're going to see that nice ramp up and it's going to ramp up even further. you can also see it's using more CPU this time as expected. we're down to 13% idle and it's still ramping up.

[00:17:13] But that's, that's great.

[00:17:14] And then down here, sure enough, we're up to a 2.6 gigabits and still climbing. So as noted, it's capable of doubling as we double how much we throw at it. So Goose is more or less linearly scaling what's available to it, which is great to see. It's using about 90% of the CPU resources sustaining great load.

[00:17:31] And of course, you know, what's, I guess what's also worth seeing is that this website is continuing. It should continue load because Fastly is actually taking most of the traffic, it's not actually hitting the back server. So even though I'm logged in I can still use the website even under this much load.

[00:17:46] With that all said and done. See here, it did finish. we were seeing up to 7,000 requests per second, which added up to, or yes, and, and sure enough well, 6,600 requests per second is what what Goose believed that it was doing. Again, there were zero failures, which is good to see. Fastly is doing a great job.

[00:18:06] When it says zero failures, it's important to note that Goose is. It, it's not guessing on that for every single request it makes for, you know, 6,600 requests a second, it checks the status code to make sure that it's what it expects. So at 200, but then it's also analyzing the page that's returned. And it's looking for key words, it's looking for, you know, the, whether it's the title of the page or whether it's a specific search term and making sure that really does exist on the page.

[00:18:31] And it does it for 100% of the pages returned, not just a subset. Rust is incredibly efficient and a fantastic platform for doing a load test like this. Do you have any questions at this point, Michael or Fabian?

[00:18:44] Michael Meyers: You said that it's doing a hundred percent of the pages and checking.

[00:18:46] Are you implying that other tools and systems don't or?

[00:18:51] Jeremy Andrews: Yeah. What happens at scale is to be able to do to, to be able to scale up a load to this size or even larger, like we're going to do you start having to use performance tricks, where you look at a percentage of them and you have a pretty reasonable assumption that things are working you know, check 25%.

[00:19:07] If they all passed, you probably are okay. but we don't have to do that. And we're able to scale up. I haven't found an uplink big enough that we can't handle that with the available CPU power.

[00:19:16] Fabian Franz: Essentially Michael, a people could also create this much load just by using AB tools or HTP perf, et cetera, which, which is why people could say this is nothing special, but what Jeremy is telling us that this load test, not only is it programmed in Goose, but it does a lot more than just sending traffic and also analyzes the traffic and analyze all the pages. And it's insane how efficient it is using the CPU. Is like usually if you want to create this much traffic you would need several boxes.

[00:19:48] Several AWS boxes to, to even do it, or do some parallelization, et cetera, with this other tools. So it's really fantastic. One Goose, one instance. You're good to go. No complexity. That's great.

[00:20:04] Jeremy Andrews: So you know, the logical thing to do is to double everything again and see what happens. At least that's how it seems logical to me. So we'll go from a 2000 to 4,000 users and we'll create 128 a second. I'm going to restart this SAR because I'm not sure it's going to run long enough otherwise.

[00:20:22] Yeah. And the thing to see here is, again, it can. Launch this many users this fast without problems.

[00:20:30] Jeremy Andrews: Okay. Yes. So it can start them which is great news. what we're going to find is that we're going to hit a different bottleneck this time though and it's not the server CPU. It's going to be our uplink. We stalled for a little while, creating user states, getting everything ready to go, and then boom, all of the user threads start. There's not much to watch here. It's just going to scroll through all the users it creates. So I'll keep it open on our CPU chart to see how much CPU it's using.

[00:20:55] If there's errors it'll become pretty apparent in various places. So for now, we'll just keep an eye on, on on this website.

[00:21:05] Fabian Franz: I love it. I love it to, to just watch this in real time. See the traffic coming in. It's always my favorite part of when Tag1 is doing load tests for various clients. And when we are doing this much load and then watching New Relic, like, like going into insane numbers. Watching Fastly, going to insane numbers.

[00:21:26] Jeremy Andrews: It's fun. it's disappointing here. We're not, we'll have the four 4k a request per second on, on Fastly. So we seem to have capped out here at 6,600 requests a second. And sure enough, interestingly, we, we have the same, even though we're have twice as many users we're seeing the same amount of traffic.

[00:21:46] One thing. That's interesting. If you compare this bandwidth chart to the previous bandwidth chart, it's flatter. and so what's happening is the CPU has sufficient power. It's able to make all of these requests but the way that async works, they’re asynchronous requests.

[00:22:00]Whenever something stalled, it just goes for another request. And so it's able to flatten out how fast these requests are working.

[00:22:06] Jeremy Andrews: So that load test finished nicely, has a relatively flat line because the CPU had enough power to, to keep generating requests. even though the bandwidth was limited. What that shows is that if, if we could start up a server with more power, we could do more interesting things with it, like generate even more load.

[00:22:23] So let's do exactly that. I'm going to exit out of here. I went ahead and started another instance. This one is an R five in eight X large and the eight X large means it has 32 cores. And I believe it's supposed to have like a 25 gigabit uplink, and that's great because we are needing more bandwidth.

[00:22:46] Jeremy Andrews: So we'll connect into the server. And as before we'll connect into it twice so that we can see how much, how much we're stressing it out.

[00:22:57] And there we go.

[00:22:58] So inside the Goose directory we can spin up another load test. Now that we're not capped by bandwidth, we hope let's rerun our 4,000 user tests.

[00:23:07] Jeremy Andrews: Make sure there's no typos in this looks good. Goose debug file one minute, 4,000 users. All right. So yup.

[00:23:16] It's gonna start allocating memory. And that takes a bit of CPU power. We will be able to see on Fastly once this traffic starts hitting. And, and the goal here is that because we have more uplink that we're able to generate more traffic. the expectation absolutely is that that should work.

[00:23:35] Let's see . Start hitting Fastly. The server doesn't have to work very hard because again, it has 32 cores. So even though it's generating quite a bit of users there's plenty of power there. So that's not much of an issue now. We've got the requests starting to come in.

[00:23:52] And already you can see traffic on the Seattle PoP. whoops. There was a spike, which means there's errors and sure enough. So what we have going on here is a perfect storm of too many requests and the server's down. So there's no point in running any further. What's happened is this was able to ramp up so fast and it was able to hit enough pages that were not yet in the cache that it caused some errors.

[00:24:17] It's worth taking a quick look and see here that we've got a, we've got data in our, in our debug log. It has 19 megs of of data there. But if you look, you can get some sense of what's happening and sure enough here, what you can see is that we're not getting anything back. there's just the, the servers not responding.

[00:24:33] Oh, And actually I'm wrong. Okay. It wasn't what I thought it was. The specific problem here is DNS stopper. We killed DNS locally. The solution fortunately is quite easy. if we ping umami.tag1.io and we grab this IP address, we can add it to our host file. And I did have it. Oh, it is there. Okay. Next theory.

[00:25:00] Too many open files. That's our problem. All right. So the other issue is that you need to increase your ulimit, and I go crazy large that we don't run into this. I forgot we've spun up a new server. So I had, I didn't, I didn't make that permanent, so I have to set the limit. Now, if we run again, everything should work a lot better.

[00:25:18 And luckily that's, that should be all we need to do to to be able to run this test properly. So the two things that I've I've ran into when I was testing, this was DNS dying which is for me, the quickest solution was added as the host and not having enough file handles, which is why we increase it with ulimit with those.

[00:25:36] Fabian Franz: it makes perfect sense.

[00:25:37] If you want to load test it can make sense to just put in the IP address. However one needs to be careful when doing that with a distributed load test because then else you would be hitting the Fastly PoP of Seattle from China. That's not what we want to do, obviously. So yeah, but then we also don't want to load test DNS, so it makes sense to cache it locally.

[00:26:03] Jeremy Andrews: I was pinging from each server and then using the response to the ping to set up the host. So whatever resolves for that user, I just cache it in as the host, essentially. and that works pretty well. So now we have what we like to see. We can see a proper ramp up happening and it's continuing to ramp up fairly aggressively.

[00:26:18] That's a lot of requests happening per second, that Goose is making and every single one is validated. And it's the real deal. plus we can see in the global PoP traffic that, you know, we we've hit a new, new level. We went from where it was the topping out at 4k. Now we're talking about 8K so.

[00:26:33] As before we were able to double it. and sure enough bandwidth is up at 4.8 to five gigabytes of traffic pretty consistently.

[00:26:41]Fabian Franz: Can we go even higher?

[00:26:43] Jeremy Andrews: Yes. In fact we can the thing to do, I, I just like doubling and seeing how far we can get. So let's double it again. I'll go ahead and cancel this.

[00:26:51] There's no need to let it run to completion at this point. I will let us do a clean shutdown. actually, you know what, let's just force it to quit. So it's quicker. 128, we'll go to 256 and we'll double 4,000 8,000. So now we're gonna simulate 8,000 users actively loading the page very aggressively and.

[00:27:13] Yeah, . let's start that. we'll see whether or not our uplink can handle 8,000 users. go ahead.

[00:27:19] Fabian Franz: While, we're waiting for that. Could you add like DNS caching to Goose itself like that? it's this only one request or.

[00:27:31] Jeremy Andrews: Yes. I mean, it might get complicated cause there's nothing to prevent your load tests from hitting multiple domains.

[00:27:36] But that's interesting. We should look into that. You want to file

[00:27:41] Fabian Franz: Do you want to take a request live on stage.

[00:27:43] Jeremy Andrews: Nice. I liked the idea. It'd be a good question for Narayan to whether or not he would like that feature, but it seems great to me. Whereas before we were using 50% of our our server to generate traffic, we'll see how much, how much we end up using for this much traffic.

[00:28:00] But our, our Fastly stats should start, start ramping up pretty quickly here

[00:28:11] And there we go. There are, we can see there's no errors because ramping up cleanly like that.

[00:28:18] It's catching up. we're already surpassing the original tests. Now we're at 8K requests per second. It's still going up. 12 K 16K. Sure enough. We were able to double it again. so we're doing 20 to 23,000 requests a second and sustaining 9.2 gigabytes. nine gigabytes of traffic a second with a nice steady line, I should add, you know, it's not, it's not spiky traffic.

[00:28:40] It's very, very consistent traffic which is fantastic when you're trying to do a load test and the, and the server, you know, it's, it's using a CPU's there's only 4% idle but it's, it's generating some impressive some impressive load.

[00:28:54] Michael Meyers: This was, this was really cool to see. Thank you guys so much for walking us through this.

[00:28:58] We're going to do some follow-ups to this. We're going to do some how-tos on testing with authenticated traffic. We're going to show you how to spin up Gaggles and do distributed load testing with Goose and Fabian, I'm going to take you up on your scalability webinar, show people how all that works.

[00:29:15] So please stay tuned for more. We'll put all of the links that Jeremy and Fabian mentioned in the show notes. Do you have any questions for us? Please reach out to us at tag1teamtalks@tag1.com. We'd love your feedback on this, questions about Goose, or even better hit us up in the issue queues on GitHub.

[00:29:33] And we love feature suggestions. You know, what, what should we cover in the future on this? You can see our past Tag1 Team Talks at tag1.com/TTT for Tag1 Team Talks, and you can go to Tag1.com/goose to get a lot of links and information about Goose Jeremy, Fabian. Again, a huge thank you for walking us through this today and to our listeners.

[00:29:55] Thanks for tuning in for anotherTag1 Team talk.