Performance Soak Testing: Faster Loop, More Convenience

[blt]

The Vector project today relies on "soak tests" for its all-up, integrated performance testing. What we have today is the "Run Vector Continuously" option from RFC 6531. These tests are run 24/7 on the current nightly build for a set list of configurations. Load is generated with lading and sinks are real. Results from the soak tests have been promising to this point. In the following table we summarize the worked configurations so far:

Soak Test	Throughput MB/s (initial)	Throughput MB/s (current)	Delta
Datadog agent => remap => Datadog logs	3	46	+43
Kafka => JSON parser => Elasticsearch	3	3	45
Syslog => regex parser => Logs to Metrics => Datadog metrics	12	12	+0
Splunk forwarder => remap => Datadog logs	27	27	+0
File => remap => Blackhole	23	257	+234

Each configuration has seen a steep drop in CPU consumption overall, as well as memory consumption in our soak rig. This work has started us down architectural and operational improvements to Vector -- for instance #9261, #9480, #9477, #9165 -- that would not have been as clearly motivated without these soak tests.

However, the soak tests as they exist today are very much a first pass on an idea and have some serious drawbacks.

How Soak Tests Are Made

A "soak test" is a VM that runs 24/7 in project infrastructure. The VM is specially rigged to run nightly Vector, lading and has access to services we've spun up in AWS, where the VMs run. Each soak testing VM uses the same base image -- built with ansible -- and all that differs between machines are configuration files for Vector and lading. The VMs and services are managed with terraform. Deploying a new soak test requires permission to build the ansible image and permission to run terraform, of which only a subset of the paid Vector team is practically capable. Telemetry is rigged to emit into Datadog and we have a dashboard for viewing the results one test at a time. It takes about 15 minutes to build a new image with ansible and then run the terraform apply before any changes are present in the soak testing infra.

Debugging issues with a new soak test is awkward and there's no convenient local testing -- especially if you work from a Mac -- meaning it can take a day or two to build a new soak test. which is convenient for the project since it's free but does make sharing results a hard problem. This whole setup effectively makes our soak tests closed-source, which, uh, we don't love.

What's Tough About our Soak Tests

The biggest drawbacks with our current soak test approach can be summarized pretty simply:

• they are too hard to add to,
• they don't allow ad-hoc experiments,
• they take too long to get results and
• they don't attribute regressions well.

Vector developers create new soak tests with a combination of ansible and terraform -- more on this below -- which then wait to be rolled out into our soak infra, telemetry from these tests being terminated into Datadog for dashboarding. In order for changes to be soaked they must be merged to master: it is inevitable that we'll commit regressions and see them in nightly before they are detected by our current approach. Asking everyone on the team to learn ansible and terraform is a Big Ask, not to mention we have no way of sharing this with the broader community. Because of the way our soak tests are built we can't run them local to our dev environments, meaning PR work is done in an ad hoc process that is similar but different from how soak tests run. You can see this in my investigation issues, example. PR authors have to take special, extra steps to interrogate their work and may accidentally regress important use cases outside the scope of their work. The nightly-only nature of our soak tests mean changes have, generally, a 24 hour wait for feedback, by which time you may have already mentally moved on to new work. Moreover, because of this nightly roll-up regression attribution is challenging today; a developer must notice that performance has dropped, figure out which nightly is at cause and then look up which commits are present in that nightly and work out which might be involved.

What We Know Now

When RFC 6531 was written we didn't appreciate a handful of important factors. They are:

1. In the present state of the project relatively brief runtimes are indicative of throughput performance.
2. Ad-hoc, uniform performance experiments -- PR based, say -- would be a huge productivity boon.
3. Waiting a day for feedback is too long.
4. Active feedback is better than passive feedback.

So What Do?

We can build on our existing soak testing work and make it better for everyone. In this project we see rustc's perf as a kind of northstar for where we want to get to, especially in its integration with PRs via bors. We've had a great deal of success with our clippy / Github integration and there's every indication we should be able to pull off the same trick with soak testing. We can also, I believe, reduce the cognitive burden of adding a soak test on our developers by eschewing ansible in favor of containerized vector and support infra, where appropriate. If we flesh out our terraform support for soak testing we end up needing to ask people to follow examples, for the most part. Avoiding AMIs as the basic image for soak testing especially opens up the possibility of local soak testing in a project uniform way. This makes it easier for us to improve our soak test infrastructure, since we no longer have to test live, not to mention the benefit to day to day development on Vector itself. Much like rustc's perf we want to capture Vector's performance relevant telemetry, offer a simple, comparative display of that data and provide PR feedback in addition to the nightly rollup feedback.

To that end we need:

• On demand container images of release + debug symbols Vector #9531
• Demonstration of localhost running environment for time-bounded soak #9616
• Demonstration of performance telemetry capture from localhost soak #9617
• Demonstration of remote running environment for time-bounded soak with telemetry capture #9619
• Triggering of remote soak for nightly Vector with telemetry capture #9620
• Port soak tests without serious infrastructure dependencies #9618
• On-demand trigger of remote soak for Vector PR with telemetry capture, PR feedback with extant and PR introduced soaks #9622
• Port soak tests with serious infrastructure dependencies #9623
• Active feedback on nightly soak comparisons, display of #9624
• OS X users report #9699 requires minikube tunnel, breaking soaks/soak.sh #9752
• Add events metrics on soak reports #10001

We've seen recent indications that collecting heap information from vector offers useful optimization paths, so being able to backfill soaks with a new methodology or to fill in data for new work would be valuable if it's straightforward to achieve. The biggest wins will be for us to get local soak testing available for the project and, second, for PR feedback to be good and useful.

[blt]

With #9926 closed the major work for this epic is now completed. PRs get statistically significant feedback, we have a CI check for serious regressions and there's a path forward to completed #9623, #9624 as a matter of typing, both of which are low priority. #10001 is also feasible if we find an interested party to do the work.