Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(datadog_metrics sink): incrementally encode sketches #17764

Merged
merged 7 commits into from
Jun 30, 2023
Merged

chore(datadog_metrics sink): incrementally encode sketches #17764

merged 7 commits into from
Jun 30, 2023

Conversation

tobz
Copy link
Contributor

@tobz tobz commented Jun 26, 2023

Context

When support was added for encoding/sending sketches in #9178, logic was added to handle "splitting" payloads if a metric exceeded the (un)compressed payload limits. As we lacked (at the time) the ability to encode sketch metrics one-by-one, we were forced to collect all of them, and then attempt to encode them all at once, which had a tendency to grow the response size past the (un)compressed payload limits. This "splitting" mechanism allowed us to compensate for that.

However, in order to avoid getting stuck in pathological loops where payloads were too big, and thus required multiple splits (after already attempting at least one split), the logic was configured such that a batch of metrics would only be split once, and if the two subsequent slices couldn't be encoded without also exceeding the limits, they would be dropped and we would give up trying to split further.

Despite the gut feeling during that work that it should be exceedingly rare to ever need to split further, real life has shown otherwise: #13175

Solution

This PR introduces proper incremental encoding of sketches, which doesn't eliminate the possibility of needing to split (more below) but significantly reduces the likelihood that splitting will need to happen down to a purely theoretical level.

We're taking advantage of hidden-from-docs methods in prost to encode each SketchPayload object and append the bytes into a single buffer. This is possible due to how Protocol Buffers functions. Additionally, we're now generating "file descriptors" for our compiled Protocol Buffers definitions. We use this to let us programmatically query the field number of the "sketches" field in the SketchPayload message, which is a slightly more robust way than just hardcoding it and hoping it doesn't ever change in the future.

In Protocol Buffers, each field in a message is written out such that the field data is preceded by the field number. This is part and parcel to its ability to allow for backwards compatible changes to a definition. Further, for repeated fields -- i.e. Vec<Sketch> -- the repetitive nature is determined simply by write the same field multiple times rather than needing to write everything all together. Practically speaking, this means that we can encode a vector of two messages, or encode those two messages individually, and end up with the same encoded output of [field N][field data][field N][field data].

Ancillary changes

We've additionally fixed a bug with the "bytes sent" metric being reported for the datadog_metrics sink due to some very tangled and miswired code around how compressed/uncompressed/event bytes/etc sizes were being shuttled from the request builder logic down to Driver.

We've also reworked some of the encoder error types just to clean them up and simplify things a bit.

Reviewer notes

Still needing to handle splits

The encoder still does need to care about splits, in a theoretical sense, because while we can accurately track and avoid ever exceeding the uncompressed payload limit, we can't know the final compressed payload size until we finalize the builder/payload.

Currently, the encoder does a check to see if adding the current metric would cause us to exceed the compressed payload limit, assuming the compressor couldn't actually compress the encoded metric at all. This is a fairly robust check since it tries to optimally account for the overhead of an entirely incompressible payload, and so on... but we really want to avoid dropping events if possible, obviously, and that's why the splitting code is still in place.

@tobz tobz requested a review from a team June 26, 2023 18:17
@netlify
Copy link

netlify bot commented Jun 26, 2023 •
edited
Loading

✅ Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit f533dd7
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/649d833967f5a7000945b865

@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Jun 26, 2023
@netlify
Copy link

netlify bot commented Jun 26, 2023 •
edited
Loading

✅ Deploy Preview for vrl-playground canceled.

Name Link
🔨 Latest commit f533dd7
🔍 Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/649d8339f814da0008d12481

spencergilbert
spencergilbert reviewed Jun 26, 2023
View reviewed changes
Copy link
Contributor

@spencergilbert spencergilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments as I scanned through this, need to take more time to really digest the bigger changes.

src/sinks/datadog/metrics/encoder.rs Outdated Show resolved Hide resolved
src/sinks/datadog/metrics/encoder.rs Outdated Show resolved Hide resolved
src/sinks/datadog/metrics/encoder.rs Show resolved Hide resolved
src/sinks/datadog/metrics/request_builder.rs Outdated Show resolved Hide resolved
@jszwedko jszwedko requested review from bruceg and neuronull June 26, 2023 20:26
tobz
tobz commented Jun 26, 2023
View reviewed changes
src/sinks/util/metadata.rs
@@ -21,21 +20,19 @@ pub struct RequestMetadataBuilder {
impl RequestMetadataBuilder {
pub fn from_events<E>(events: &[E]) -> Self
where
E: ByteSizeOf + EventCount + GetEventCountTags + EstimatedJsonEncodedSizeOf,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bound was entirely extraneous, because we already know exactly how many events we have, and in the case of this PR, I'm generating the request metadata for a bunch of Metrics, which doesn't have EventCount implemented... but Event and TraceEvent do, and the count is always 1, because, well, duh... so yeah, this was extraneous.

@datadog-vectordotdev
Copy link

datadog-vectordotdev bot commented Jun 26, 2023 •
edited
Loading

Datadog Report

Branch report: tobz/datadog-metrics-sink-incremental-sketches
Commit report: 1d16c08

✅ vector: 0 Failed, 0 New Flaky, 2102 Passed, 0 Skipped, 13m 52.14s Wall Time

@neuronull
Copy link
Contributor

We've additionally fixed a bug with the "bytes sent" metric being reported for the datadog_metrics sink due to some very tangled and miswired code around how compressed/uncompressed/event bytes/etc sizes were being shuttled from the request builder logic down to Driver.

:cant-hide:
😬

neuronull
neuronull reviewed Jun 27, 2023
View reviewed changes
Copy link
Contributor

@neuronull neuronull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the Agent + Vector e2e tests are in flight still , was any manual testing done with the agent -> vector -> DD , performed?

src/sinks/datadog/metrics/request_builder.rs
Comment on lines -223 to -231
let builder = RequestMetadataBuilder::new(
metrics.len(),
raw_bytes_written,
json_size,
);
let bytes_len = NonZeroUsize::new(payload.len())
.expect("payload should never be zero length");
let request_metadata = builder.with_request_size(bytes_len);
results.push(Ok(((metadata, request_metadata), payload)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The incorrectness here, this would be something the component validation framework would detect?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would have been caught, yeah: network bytes out was using the event bytes size, rather than the payload size.

src/sinks/datadog/metrics/encoder.rs Outdated Show resolved Hide resolved
@tobz
Copy link
Contributor Author

tobz commented Jun 28, 2023 •
edited
Loading

As the Agent + Vector e2e tests are in flight still , was any manual testing done with the agent -> vector -> DD , performed?

It wasn't, because I'm on a version of the Linux kernel that the Datadog Agent can't deal with. I did test using internal_metrics as the source, though, sending to DD, which did work: we convert histograms to sketches in the datadog_metrics sink and so this should have sufficiently exercised the code.

spencergilbert
spencergilbert approved these changes Jun 28, 2023
View reviewed changes
Copy link
Contributor

@spencergilbert spencergilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll defer to @neuronull with regards to the final approval around testing given he's the owner of the integration, but otherwise the changes look good to me.

@neuronull
Copy link
Contributor

It wasn't, because I'm on a version of the Linux kernel that the Datadog Agent can't deal with. I did test using internal_metrics as the source, though, sending to DD, which did work: we convert histograms to sketches in the datadog_metrics sink and so this should have sufficiently exercised the code.

Ok, my only concern would be if there could be differences between the internal_metrics source and the datadog_agent source in terms of the metrics going into the datadog_metrics sink that could cause it to behave differently. I admittedly don't know whether that is the case so would defer to your judgment here.

@tobz
Copy link
Contributor Author

tobz commented Jun 28, 2023

There shouldn't be.

The essence of this change is that for each sketch that comes into the sink, instead of encoding them all together, we encode them incrementally. Nothing has changed about how we interpret the Metric itself, or any of that... we're just writing out the Protocol Buffers data in a more direct way.

One of the new unit tests accounts for this by encoding a fixed set of sketches in the old all-at-once way, and then encoding them incrementally, and making sure the resulting buffers are identical.

@tobz tobz added this pull request to the merge queue Jun 28, 2023
bruceg
bruceg approved these changes Jun 28, 2023
View reviewed changes
Copy link
Member

@bruceg bruceg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just a few nit picks below.

build.rs Outdated Show resolved Hide resolved
src/proto/mod.rs Outdated Show resolved Hide resolved
src/proto/mod.rs Outdated Show resolved Hide resolved
src/sinks/datadog/metrics/encoder.rs Outdated Show resolved Hide resolved
@github-actions
Copy link

Regression Detector Results

Run ID: ef64cf18-21d6-45b9-821a-be76d8803a86
Baseline: 24cabe1
Comparison: 08c5e94
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
datadog_agent_remap_blackhole ingress throughput +1.65 [+1.57, +1.74] 100.00%
http_to_http_acks ingress throughput +0.95 [-0.29, +2.19] 67.34%
http_to_http_noack ingress throughput +0.04 [-0.01, +0.10] 66.58%
enterprise_http_to_http ingress throughput +0.03 [-0.01, +0.07] 65.84%
splunk_hec_route_s3 ingress throughput +0.02 [-0.09, +0.14] 18.50%
splunk_hec_indexer_ack_blackhole ingress throughput +0.01 [-0.03, +0.05] 18.42%
fluent_elasticsearch ingress throughput -0.00 [-0.00, +0.00] 10.77%
splunk_hec_to_splunk_hec_logs_acks ingress throughput -0.00 [-0.06, +0.06] 1.60%
http_to_http_json ingress throughput -0.01 [-0.05, +0.03] 24.10%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.02 [-0.06, +0.03] 35.07%
otlp_http_to_blackhole ingress throughput -0.15 [-0.31, +0.01] 77.49%
syslog_regex_logs2metric_ddmetrics ingress throughput -0.53 [-0.74, -0.33] 99.92%
datadog_agent_remap_blackhole_acks ingress throughput -0.60 [-0.69, -0.51] 100.00%
otlp_grpc_to_blackhole ingress throughput -0.77 [-0.87, -0.67] 100.00%
syslog_splunk_hec_logs ingress throughput -1.10 [-1.17, -1.04] 100.00%
syslog_log2metric_humio_metrics ingress throughput -1.12 [-1.20, -1.04] 100.00%
syslog_humio_logs ingress throughput -1.91 [-1.98, -1.85] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput -2.33 [-2.44, -2.23] 100.00%
syslog_log2metric_splunk_hec_metrics ingress throughput -2.42 [-2.50, -2.33] 100.00%
http_text_to_http_json ingress throughput -3.03 [-3.11, -2.95] 100.00%
datadog_agent_remap_datadog_logs ingress throughput -3.25 [-3.34, -3.15] 100.00%
socket_to_socket_blackhole ingress throughput -3.28 [-3.32, -3.23] 100.00%
syslog_loki ingress throughput -3.79 [-3.84, -3.73] 100.00%
file_to_blackhole egress throughput -7.90 [-11.67, -4.13] 99.27%

@github-actions
Copy link

Regression Detector Results

Run ID: 530b15af-7c97-4c00-8226-5a1547ddea2f
Baseline: d7bc531
Comparison: f60fe00
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
file_to_blackhole egress throughput +4.55 [+0.57, +8.53] 85.73%
http_to_http_acks ingress throughput +0.37 [-0.85, +1.59] 30.49%
syslog_log2metric_humio_metrics ingress throughput +0.26 [+0.18, +0.34] 100.00%
enterprise_http_to_http ingress throughput +0.01 [-0.02, +0.04] 46.55%
http_to_http_noack ingress throughput +0.01 [-0.05, +0.07] 22.96%
fluent_elasticsearch ingress throughput +0.00 [-0.00, +0.00] 78.33%
splunk_hec_to_splunk_hec_logs_acks ingress throughput -0.00 [-0.06, +0.06] 1.72%
http_to_http_json ingress throughput -0.00 [-0.04, +0.03] 11.31%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.01 [-0.05, +0.04] 13.91%
splunk_hec_indexer_ack_blackhole ingress throughput -0.01 [-0.05, +0.04] 17.35%
otlp_http_to_blackhole ingress throughput -0.01 [-0.16, +0.14] 7.39%
splunk_hec_route_s3 ingress throughput -0.09 [-0.22, +0.03] 65.41%
syslog_regex_logs2metric_ddmetrics ingress throughput -0.17 [-0.42, +0.08] 61.03%
datadog_agent_remap_blackhole ingress throughput -0.21 [-0.29, -0.12] 99.75%
otlp_grpc_to_blackhole ingress throughput -0.68 [-0.79, -0.56] 100.00%
datadog_agent_remap_blackhole_acks ingress throughput -1.52 [-1.59, -1.45] 100.00%
syslog_splunk_hec_logs ingress throughput -1.70 [-1.76, -1.63] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput -1.73 [-1.82, -1.64] 100.00%
datadog_agent_remap_datadog_logs ingress throughput -2.63 [-2.75, -2.50] 100.00%
syslog_humio_logs ingress throughput -2.72 [-2.78, -2.65] 100.00%
syslog_loki ingress throughput -2.80 [-2.87, -2.73] 100.00%
syslog_log2metric_splunk_hec_metrics ingress throughput -3.35 [-3.41, -3.29] 100.00%
socket_to_socket_blackhole ingress throughput -4.72 [-4.76, -4.68] 100.00%
http_text_to_http_json ingress throughput -4.76 [-4.82, -4.70] 100.00%

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 29, 2023
@tobz tobz added this pull request to the merge queue Jun 29, 2023
@dsmith3197 dsmith3197 removed this pull request from the merge queue due to a manual request Jun 29, 2023
@dsmith3197 dsmith3197 added this pull request to the merge queue Jun 29, 2023
@github-actions
Copy link

Regression Detector Results

Run ID: 542404db-b986-4f17-a24f-fa007e175f95
Baseline: e66e285
Comparison: 0ac94d4
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment goal Δ mean % confidence
http_text_to_http_json ingress throughput -6.84 100.00%
Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
syslog_regex_logs2metric_ddmetrics ingress throughput +1.67 [+1.44, +1.91] 100.00%
datadog_agent_remap_blackhole ingress throughput +1.11 [+1.01, +1.21] 100.00%
http_to_http_acks ingress throughput +0.38 [-0.85, +1.61] 30.90%
enterprise_http_to_http ingress throughput +0.03 [-0.01, +0.07] 65.76%
http_to_http_noack ingress throughput +0.01 [-0.05, +0.07] 22.54%
http_to_http_json ingress throughput +0.01 [-0.02, +0.05] 33.94%
fluent_elasticsearch ingress throughput +0.00 [-0.00, +0.00] 48.66%
splunk_hec_to_splunk_hec_logs_acks ingress throughput -0.00 [-0.06, +0.06] 2.98%
splunk_hec_indexer_ack_blackhole ingress throughput -0.01 [-0.05, +0.04] 14.13%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.02 [-0.07, +0.02] 50.94%
datadog_agent_remap_datadog_logs ingress throughput -0.06 [-0.17, +0.05] 51.94%
syslog_loki ingress throughput -0.37 [-0.45, -0.30] 100.00%
syslog_log2metric_humio_metrics ingress throughput -1.02 [-1.10, -0.94] 100.00%
otlp_http_to_blackhole ingress throughput -1.66 [-1.80, -1.51] 100.00%
syslog_log2metric_splunk_hec_metrics ingress throughput -2.06 [-2.12, -2.00] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput -2.25 [-2.34, -2.16] 100.00%
syslog_splunk_hec_logs ingress throughput -2.51 [-2.58, -2.43] 100.00%
datadog_agent_remap_blackhole_acks ingress throughput -2.58 [-2.66, -2.50] 100.00%
splunk_hec_route_s3 ingress throughput -2.62 [-2.76, -2.49] 100.00%
otlp_grpc_to_blackhole ingress throughput -2.67 [-2.78, -2.56] 100.00%
syslog_humio_logs ingress throughput -3.77 [-3.83, -3.71] 100.00%
file_to_blackhole egress throughput -3.85 [-7.51, -0.18] 82.14%
socket_to_socket_blackhole ingress throughput -4.51 [-4.56, -4.46] 100.00%
http_text_to_http_json ingress throughput -6.84 [-6.90, -6.77] 100.00%

@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 30, 2023
@spencergilbert spencergilbert added this pull request to the merge queue Jun 30, 2023
@github-actions
Copy link

Regression Detector Results

Run ID: 26c504cf-d931-433b-825c-170bf02eaa25
Baseline: 671aa79
Comparison: 3f6df61
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

  1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

  2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean % Δ mean % CI confidence
http_text_to_http_json ingress throughput +3.16 [+3.09, +3.23] 100.00%
syslog_humio_logs ingress throughput +2.10 [+2.04, +2.17] 100.00%
datadog_agent_remap_datadog_logs ingress throughput +2.04 [+1.94, +2.14] 100.00%
syslog_loki ingress throughput +1.82 [+1.75, +1.88] 100.00%
datadog_agent_remap_datadog_logs_acks ingress throughput +1.35 [+1.25, +1.46] 100.00%
datadog_agent_remap_blackhole ingress throughput +0.97 [+0.90, +1.05] 100.00%
http_to_http_acks ingress throughput +0.62 [-0.62, +1.87] 48.00%
socket_to_socket_blackhole ingress throughput +0.44 [+0.40, +0.49] 100.00%
file_to_blackhole egress throughput +0.29 [-3.74, +4.33] 7.45%
datadog_agent_remap_blackhole_acks ingress throughput +0.23 [+0.12, +0.33] 99.37%
http_to_http_noack ingress throughput +0.03 [-0.03, +0.09] 51.35%
splunk_hec_indexer_ack_blackhole ingress throughput +0.01 [-0.04, +0.05] 16.96%
fluent_elasticsearch ingress throughput -0.00 [-0.00, +0.00] 18.81%
enterprise_http_to_http ingress throughput -0.00 [-0.03, +0.03] 3.08%
http_to_http_json ingress throughput -0.01 [-0.04, +0.03] 13.44%
splunk_hec_to_splunk_hec_logs_acks ingress throughput -0.01 [-0.07, +0.05] 11.11%
splunk_hec_to_splunk_hec_logs_noack ingress throughput -0.01 [-0.05, +0.04] 17.92%
syslog_log2metric_splunk_hec_metrics ingress throughput -0.16 [-0.22, -0.09] 99.56%
splunk_hec_route_s3 ingress throughput -0.30 [-0.43, -0.16] 99.47%
syslog_splunk_hec_logs ingress throughput -0.69 [-0.77, -0.61] 100.00%
otlp_grpc_to_blackhole ingress throughput -1.46 [-1.57, -1.35] 100.00%
syslog_regex_logs2metric_ddmetrics ingress throughput -1.71 [-1.88, -1.54] 100.00%
otlp_http_to_blackhole ingress throughput -1.71 [-1.88, -1.55] 100.00%
syslog_log2metric_humio_metrics ingress throughput -3.71 [-3.80, -3.62] 100.00%

Merged via the queue into master with commit 3f6df61 Jun 30, 2023
@spencergilbert spencergilbert deleted the tobz/datadog-metrics-sink-incremental-sketches branch June 30, 2023 14:26
@jszwedko jszwedko mentioned this pull request Jun 30, 2023
Closed
@lukesteensen lukesteensen mentioned this pull request Sep 11, 2023
Merged
aholmberg pushed a commit to aholmberg/vector that referenced this pull request Feb 14, 2024
@logdnabot
chore(release): 1.30.0 [skip ci]
64f9fcd 
# [1.30.0](answerbook/vector@v1.29.2...v1.30.0) (2023-12-19)

### Bug Fixes

* `aws_ec2_metadata` transform when using log namespacing (vectordotdev#17819) [4786743](answerbook/vector@4786743) - GitHub
* **auth**: Vector does not put the Proxy-Authorization header on the wire (vectordotdev#17353) (vectordotdev#17363) [6705bdd](answerbook/vector@6705bdd) - GitHub
* **buffers**: deadlock when seeking after entire write fails to be flushed (vectordotdev#17657) [37a662a](answerbook/vector@37a662a) - GitHub
* **ci**: add missing env var (vectordotdev#17872) [4f67695](answerbook/vector@4f67695) - Jesse Szwedko
* **ci**: add missing logic to mark required checks failed (vectordotdev#17543) [3b87e00](answerbook/vector@3b87e00) - GitHub
* **ci**: change command to find baseline sha from issue comment trigger (vectordotdev#17622) [5791083](answerbook/vector@5791083) - GitHub
* **ci**: checkout a greater depth in regression workflow (vectordotdev#17604) [baa04e5](answerbook/vector@baa04e5) - GitHub
* **ci**: post failed status to PR and isolate branch checkout on comment trigger (vectordotdev#17544) [e2c0255](answerbook/vector@e2c0255) - GitHub
* **ci**: reg workflow alt approach to getting baseline sha (vectordotdev#17645) [f1e1ae3](answerbook/vector@f1e1ae3) - GitHub
* **ci**: use correct ID for Triage in Gardener Board (vectordotdev#17647) [2638cca](answerbook/vector@2638cca) - GitHub
* **ci**: use correct secret for gardener board comment (vectordotdev#17605) [9395eba](answerbook/vector@9395eba) - GitHub
* **config**: Fix preloading log_schema (vectordotdev#17759) [659e1e6](answerbook/vector@659e1e6) - GitHub
* **databend sink**: use get for page request (vectordotdev#17373) [c7d7cf8](answerbook/vector@c7d7cf8) - GitHub
* **datadog_agent source**: remove duplicate internal metrics emission (vectordotdev#17720) [48ec2e8](answerbook/vector@48ec2e8) - GitHub
* **distribution**: Fix architecture detection for ARMv7 (vectordotdev#17484) [78fb469](answerbook/vector@78fb469) - GitHub
* **docs**: fix copy-paste issue in component spec (vectordotdev#17616) [b400acc](answerbook/vector@b400acc) - GitHub
* **file source**: Fix tailing problem when source number greater than 512 (vectordotdev#17717) [23a3e0e](answerbook/vector@23a3e0e) - GitHub
* **fluent source**: fix ack message format (vectordotdev#17407) [d194992](answerbook/vector@d194992) - GitHub
* **http_client source**: adapt int test to use breaking change of dep (vectordotdev#17583) [d7df520](answerbook/vector@d7df520) - GitHub
* **http_client source**: remove utf8 lossy conversion (vectordotdev#17655) [59e2cbf](answerbook/vector@59e2cbf) - GitHub
* **install.sh**: Correctly `shift` all parsed arguments (vectordotdev#17684) [f883575](answerbook/vector@f883575) - GitHub
* **loki sink, observability**: Drop non-fatal template render errors to warnings (vectordotdev#17746) [4ebc3e1](answerbook/vector@4ebc3e1) - GitHub
* **loki sink**: use json size of unencoded event (vectordotdev#17572) [25e7699](answerbook/vector@25e7699) - GitHub
* **observability**: correct emitted metrics (vectordotdev#17562) [7a4f1f7](answerbook/vector@7a4f1f7) - GitHub
* **observability**: issues with event_cache PR (vectordotdev#17768) [fdf02d9](answerbook/vector@fdf02d9) - GitHub
* remap behavior for root types when using the `Vector` namespace (vectordotdev#17807) [c19938c](answerbook/vector@c19938c) - GitHub
* **sinks**: Add missing component span for sink building (vectordotdev#17765) [219883e](answerbook/vector@219883e) - GitHub

### Chores

* Add docker config to dependabot (vectordotdev#17696) [079d895](answerbook/vector@079d895) - GitHub
* add sink prelude (vectordotdev#17595) [da939ca](answerbook/vector@da939ca) - GitHub
* Add submodules to all checkouts (vectordotdev#17770) [7196622](answerbook/vector@7196622) - GitHub
* **administration**: add domain label for vdev (vectordotdev#17748) [c35ebd1](answerbook/vector@c35ebd1) - GitHub
* **aws_s3 sink**: Update metadata to match the editorial review for the schema. (vectordotdev#17475) [c1262cd](answerbook/vector@c1262cd) - GitHub
* Bump version to 0.31.0 (vectordotdev#17466) [78bbfbc](answerbook/vector@78bbfbc) - GitHub
* **ci**: Add apt retries to cross builds (vectordotdev#17683) [ab1169b](answerbook/vector@ab1169b) - GitHub
* **ci**: Add schedule to component features workflow conditional check (vectordotdev#17816) [708b7f6](answerbook/vector@708b7f6) - GitHub
* **ci**: Bump aws-actions/configure-aws-credentials from 2.0.0 to 2.1.0 (vectordotdev#17565) [8a741d5](answerbook/vector@8a741d5) - GitHub
* **ci**: Bump aws-actions/configure-aws-credentials from 2.1.0 to 2.2.0 (vectordotdev#17697) [12bc4a7](answerbook/vector@12bc4a7) - GitHub
* **ci**: Bump docker/build-push-action from 4.0.0 to 4.1.0 (vectordotdev#17656) [cb9a3a5](answerbook/vector@cb9a3a5) - GitHub
* **ci**: Bump docker/build-push-action from 4.1.0 to 4.1.1 (vectordotdev#17687) [bce5e65](answerbook/vector@bce5e65) - GitHub
* **ci**: Bump docker/metadata-action from 4.4.0 to 4.5.0 (vectordotdev#17624) [a54a12f](answerbook/vector@a54a12f) - GitHub
* **ci**: Bump docker/metadata-action from 4.5.0 to 4.6.0 (vectordotdev#17686) [71273df](answerbook/vector@71273df) - GitHub
* **ci**: Bump docker/setup-buildx-action from 2.5.0 to 2.6.0 (vectordotdev#17625) [15bc42a](answerbook/vector@15bc42a) - GitHub
* **ci**: Bump docker/setup-buildx-action from 2.6.0 to 2.7.0 (vectordotdev#17685) [8006987](answerbook/vector@8006987) - GitHub
* **ci**: Bump docker/setup-buildx-action from 2.7.0 to 2.8.0 (vectordotdev#17786) [dbdff9e](answerbook/vector@dbdff9e) - GitHub
* **ci**: Bump docker/setup-qemu-action from 2.1.0 to 2.2.0 (vectordotdev#17623) [3005141](answerbook/vector@3005141) - GitHub
* **ci**: bump myrotvorets/set-commit-status-action from 1.1.6 to 1.1.7 (vectordotdev#17460) [bca45eb](answerbook/vector@bca45eb) - GitHub
* **ci**: Bump up OSX runners for release builds (vectordotdev#17823) [fe730ad](answerbook/vector@fe730ad) - GitHub
* **ci**: bump xt0rted/pull-request-comment-branch from 1 to 2 (vectordotdev#17461) [c425006](answerbook/vector@c425006) - GitHub
* **ci**: correctly validate comment author in k8s e2e job (vectordotdev#17818) [b8e3dbe](answerbook/vector@b8e3dbe) - GitHub
* **ci**: Drop VRL license exceptions (vectordotdev#17529) [aa01452](answerbook/vector@aa01452) - GitHub
* **ci**: fix a few logic bugs and more strict comment parsing (vectordotdev#17502) [bf372fd](answerbook/vector@bf372fd) - GitHub
* **ci**: fix comment author validation (vectordotdev#17794) [75ae967](answerbook/vector@75ae967) - GitHub
* **ci**: fix failure notify job conditional in publish workflow (vectordotdev#17468) [3699842](answerbook/vector@3699842) - GitHub
* **ci**: fix gardener issues comment workflow (vectordotdev#17825) [47c3da1](answerbook/vector@47c3da1) - GitHub
* **ci**: fix team membership action (vectordotdev#17791) [13c3c78](answerbook/vector@13c3c78) - GitHub
* **ci**: int test yaml file detection (vectordotdev#17590) [fa8a553](answerbook/vector@fa8a553) - GitHub
* **ci**: minor fixes to workflows post merge queue enabling  (vectordotdev#17462) [9f6f6ec](answerbook/vector@9f6f6ec) - GitHub
* **ci**: move component features check out of merge queue (vectordotdev#17773) [e6e776d](answerbook/vector@e6e776d) - GitHub
* **ci**: Move most CI checks to merge queue (vectordotdev#17340) [060399a](answerbook/vector@060399a) - GitHub
* **ci**: reduce runner sizing to 4 core and free tier (vectordotdev#17785) [53a575f](answerbook/vector@53a575f) - GitHub
* **ci**: remove /ci-run-install comment trigger (vectordotdev#17803) [a3770d8](answerbook/vector@a3770d8) - GitHub
* **ci**: Remove remaining Discord notification (vectordotdev#17805) [ed59f37](answerbook/vector@ed59f37) - GitHub
* **ci**: Remove upload of config schema (vectordotdev#17740) [ff6a1b4](answerbook/vector@ff6a1b4) - GitHub
* **ci**: Retry `make check-component-docs` check (vectordotdev#17718) [e8e7e04](answerbook/vector@e8e7e04) - GitHub
* **ci**: revert fix gardener issues comment workflow (vectordotdev#17829) [ee10b8c](answerbook/vector@ee10b8c) - GitHub
* **ci**: Set HOMEBREW_NO_INSTALL_FROM_API in CI (vectordotdev#17867) [00cc584](answerbook/vector@00cc584) - Jesse Szwedko
* **ci**: temporarily disable comment_trigger workflow (vectordotdev#17480) [58d7f3d](answerbook/vector@58d7f3d) - GitHub
* **ci**: temporarily disable flakey `aws_s3` integration test case `handles_errored_status`  (vectordotdev#17455) [8e40b68](answerbook/vector@8e40b68) - GitHub
* **ci**: update comment_trigger note about concurrency groups (vectordotdev#17491) [7699f4d](answerbook/vector@7699f4d) - GitHub
* **ci**: Update publish workflow test Ubuntu versions (vectordotdev#17781) [20d62f1](answerbook/vector@20d62f1) - GitHub
* **clickhouse sink**: refactor to new style (vectordotdev#17723) [77ac63c](answerbook/vector@77ac63c) - GitHub
* **codecs**: consolidate enum types (vectordotdev#17688) [9c45394](answerbook/vector@9c45394) - GitHub
* Codify flag naming including sentinel values (vectordotdev#17569) [134578d](answerbook/vector@134578d) - GitHub
* Codify the use of abbreviate time units in config option names (vectordotdev#17582) [8823561](answerbook/vector@8823561) - GitHub
* **config**: Convert top-level sinks enum to typetag (vectordotdev#17710) [9cd5404](answerbook/vector@9cd5404) - GitHub
* **config**: Make config schema output ordered (vectordotdev#17694) [9606353](answerbook/vector@9606353) - GitHub
* **config**: Update field labels for commonly used sources and transforms  (vectordotdev#17517) [f523f70](answerbook/vector@f523f70) - GitHub
* **config**: Update field labels for sinks (vectordotdev#17560) [e1ddd0e](answerbook/vector@e1ddd0e) - GitHub
* **config**: Update field labels for the rest of the sources and transforms fields (vectordotdev#17564) [6e45477](answerbook/vector@6e45477) - GitHub
* **datadog_archives sink**: Remove this component (vectordotdev#17749) [53f8bff](answerbook/vector@53f8bff) - GitHub
* **datadog_metrics sink**: incrementally encode sketches (vectordotdev#17764) [3f6df61](answerbook/vector@3f6df61) - GitHub
* **datadog_traces sink**: Add additional warning around APM stats for `peer.service` (vectordotdev#17733) [9a899c5](answerbook/vector@9a899c5) - GitHub
* **deps, releasing**: Update to Alpine 3.18 (vectordotdev#17695) [2263756](answerbook/vector@2263756) - GitHub
* **deps**: Bump async-graphql from 5.0.8 to 5.0.9 (vectordotdev#17486) [077a294](answerbook/vector@077a294) - GitHub
* **deps**: Bump async-graphql from 5.0.9 to 5.0.10 (vectordotdev#17619) [2931542](answerbook/vector@2931542) - GitHub
* **deps**: Bump async-graphql-warp from 5.0.8 to 5.0.9 (vectordotdev#17489) [ac81fc1](answerbook/vector@ac81fc1) - GitHub
* **deps**: Bump async-graphql-warp from 5.0.9 to 5.0.10 (vectordotdev#17642) [b3885f6](answerbook/vector@b3885f6) - GitHub
* **deps**: bump aws-sigv4 from 0.55.1 to 0.55.3 (vectordotdev#17481) [2ad5b47](answerbook/vector@2ad5b47) - GitHub
* **deps**: bump base64 from 0.21.0 to 0.21.1 (vectordotdev#17451) [95cbba9](answerbook/vector@95cbba9) - GitHub
* **deps**: Bump base64 from 0.21.1 to 0.21.2 (vectordotdev#17488) [f261781](answerbook/vector@f261781) - GitHub
* **deps**: bump bstr from 1.4.0 to 1.5.0 (vectordotdev#17453) [7554d9c](answerbook/vector@7554d9c) - GitHub
* **deps**: Bump cached from 0.43.0 to 0.44.0 (vectordotdev#17599) [7a55210](answerbook/vector@7a55210) - GitHub
* **deps**: Bump chrono to 0.4.26 (vectordotdev#17537) [0dfa09c](answerbook/vector@0dfa09c) - GitHub
* **deps**: Bump chrono-tz from 0.8.2 to 0.8.3 (vectordotdev#17789) [f79947c](answerbook/vector@f79947c) - GitHub
* **deps**: bump clap_complete from 4.2.3 to 4.3.0 (vectordotdev#17447) [05bf262](answerbook/vector@05bf262) - GitHub
* **deps**: Bump clap_complete from 4.3.0 to 4.3.1 (vectordotdev#17586) [8549809](answerbook/vector@8549809) - GitHub
* **deps**: bump criterion from 0.4.0 to 0.5.0 (vectordotdev#17477) [84f0ada](answerbook/vector@84f0ada) - GitHub
* **deps**: Bump criterion from 0.5.0 to 0.5.1 (vectordotdev#17500) [da7bc95](answerbook/vector@da7bc95) - GitHub
* **deps**: Bump crossbeam-utils from 0.8.15 to 0.8.16 (vectordotdev#17674) [714ccf8](answerbook/vector@714ccf8) - GitHub
* **deps**: Bump csv from 1.2.1 to 1.2.2 (vectordotdev#17555) [bcc5b6c](answerbook/vector@bcc5b6c) - GitHub
* **deps**: bump data-encoding from 2.3.3 to 2.4.0 (vectordotdev#17452) [9aaf864](answerbook/vector@9aaf864) - GitHub
* **deps**: Bump getrandom from 0.2.9 to 0.2.10 (vectordotdev#17613) [bd880f5](answerbook/vector@bd880f5) - GitHub
* **deps**: Bump gloo-utils from 0.1.6 to 0.1.7 (vectordotdev#17707) [53e1785](answerbook/vector@53e1785) - GitHub
* **deps**: Bump graphql_client from 0.12.0 to 0.13.0 (vectordotdev#17541) [ecb707a](answerbook/vector@ecb707a) - GitHub
* **deps**: Bump h2 from 0.3.19 to 0.3.20 (vectordotdev#17767) [ac3bc72](answerbook/vector@ac3bc72) - GitHub
* **deps**: Bump hashbrown from 0.13.2 to 0.14.0 (vectordotdev#17609) [154e393](answerbook/vector@154e393) - GitHub
* **deps**: Bump hyper from 0.14.26 to 0.14.27 (vectordotdev#17766) [a2a3609](answerbook/vector@a2a3609) - GitHub
* **deps**: Bump indexmap from 1.9.3 to 2.0.0 (vectordotdev#17755) [248ccb8](answerbook/vector@248ccb8) - GitHub
* **deps**: Bump indicatif from 0.17.3 to 0.17.4 (vectordotdev#17532) [1565985](answerbook/vector@1565985) - GitHub
* **deps**: Bump indicatif from 0.17.4 to 0.17.5 (vectordotdev#17597) [a164952](answerbook/vector@a164952) - GitHub
* **deps**: Bump infer from 0.13.0 to 0.14.0 (vectordotdev#17737) [326ad08](answerbook/vector@326ad08) - GitHub
* **deps**: Bump itertools from 0.10.5 to 0.11.0 (vectordotdev#17736) [6e1878b](answerbook/vector@6e1878b) - GitHub
* **deps**: Bump lalrpop to 0.19.12 (vectordotdev#17457) [1f54415](answerbook/vector@1f54415) - GitHub
* **deps**: bump lapin from 2.1.1 to 2.1.2 (vectordotdev#17439) [a8b7899](answerbook/vector@a8b7899) - GitHub
* **deps**: bump lapin from 2.1.2 to 2.2.0 (vectordotdev#17443) [b639422](answerbook/vector@b639422) - GitHub
* **deps**: bump lapin from 2.2.0 to 2.2.1 (vectordotdev#17448) [618379a](answerbook/vector@618379a) - GitHub
* **deps**: Bump libc from 0.2.144 to 0.2.146 (vectordotdev#17615) [10cfd0a](answerbook/vector@10cfd0a) - GitHub
* **deps**: Bump libc from 0.2.146 to 0.2.147 (vectordotdev#17753) [1a75ec6](answerbook/vector@1a75ec6) - GitHub
* **deps**: Bump log from 0.4.17 to 0.4.18 (vectordotdev#17526) [5a2fea1](answerbook/vector@5a2fea1) - GitHub
* **deps**: Bump log from 0.4.18 to 0.4.19 (vectordotdev#17662) [e1b3357](answerbook/vector@e1b3357) - GitHub
* **deps**: Bump lru from 0.10.0 to 0.10.1 (vectordotdev#17810) [96e68f7](answerbook/vector@96e68f7) - GitHub
* **deps**: bump memmap2 from 0.6.1 to 0.6.2 (vectordotdev#17482) [79f7dfb](answerbook/vector@79f7dfb) - GitHub
* **deps**: Bump memmap2 from 0.6.2 to 0.7.0 (vectordotdev#17641) [593ea1b](answerbook/vector@593ea1b) - GitHub
* **deps**: Bump memmap2 from 0.7.0 to 0.7.1 (vectordotdev#17752) [4236e32](answerbook/vector@4236e32) - GitHub
* **deps**: Bump mock_instant from 0.3.0 to 0.3.1 (vectordotdev#17574) [1c1beb8](answerbook/vector@1c1beb8) - GitHub
* **deps**: Bump mongodb from 2.5.0 to 2.6.0 (vectordotdev#17726) [c96e3be](answerbook/vector@c96e3be) - GitHub
* **deps**: Bump notify from 6.0.0 to 6.0.1 (vectordotdev#17700) [cd6d154](answerbook/vector@cd6d154) - GitHub
* **deps**: Bump once_cell from 1.17.1 to 1.17.2 (vectordotdev#17531) [8e113ad](answerbook/vector@8e113ad) - GitHub
* **deps**: Bump once_cell from 1.17.2 to 1.18.0 (vectordotdev#17596) [dc6bef2](answerbook/vector@dc6bef2) - GitHub
* **deps**: bump opendal from 0.34.0 to 0.35.0 (vectordotdev#17471) [ebf958b](answerbook/vector@ebf958b) - GitHub
* **deps**: Bump opendal from 0.35.0 to 0.36.0 (vectordotdev#17540) [dbd7151](answerbook/vector@dbd7151) - GitHub
* **deps**: Bump opendal from 0.36.0 to 0.37.0 (vectordotdev#17614) [b5bd85f](answerbook/vector@b5bd85f) - GitHub
* **deps**: Bump opendal from 0.37.0 to 0.38.0 (vectordotdev#17777) [ec4785a](answerbook/vector@ec4785a) - GitHub
* **deps**: Bump openssl from 0.10.52 to 0.10.53 (vectordotdev#17534) [078de66](answerbook/vector@078de66) - GitHub
* **deps**: Bump openssl from 0.10.53 to 0.10.54 (vectordotdev#17573) [4af5e6d](answerbook/vector@4af5e6d) - GitHub
* **deps**: Bump openssl from 0.10.54 to 0.10.55 (vectordotdev#17716) [dd2527d](answerbook/vector@dd2527d) - GitHub
* **deps**: Bump percent-encoding from 2.2.0 to 2.3.0 (vectordotdev#17602) [8e04259](answerbook/vector@8e04259) - GitHub
* **deps**: Bump pin-project from 1.1.0 to 1.1.1 (vectordotdev#17806) [0b32626](answerbook/vector@0b32626) - GitHub
* **deps**: Bump PR limit for Dependabot to 100 (vectordotdev#17459) [85703e7](answerbook/vector@85703e7) - GitHub
* **deps**: bump proc-macro2 from 1.0.57 to 1.0.58 (vectordotdev#17426) [ae656c7](answerbook/vector@ae656c7) - GitHub
* **deps**: Bump proc-macro2 from 1.0.58 to 1.0.59 (vectordotdev#17495) [4ce3278](answerbook/vector@4ce3278) - GitHub
* **deps**: Bump proc-macro2 from 1.0.59 to 1.0.60 (vectordotdev#17643) [f20eb2f](answerbook/vector@f20eb2f) - GitHub
* **deps**: Bump proc-macro2 from 1.0.60 to 1.0.63 (vectordotdev#17757) [63ba2a9](answerbook/vector@63ba2a9) - GitHub
* **deps**: bump proptest from 1.1.0 to 1.2.0 (vectordotdev#17476) [9235fc2](answerbook/vector@9235fc2) - GitHub
* **deps**: bump pulsar from 5.1.1 to 6.0.0 (vectordotdev#17587) [3395cfd](answerbook/vector@3395cfd) - GitHub
* **deps**: Bump pulsar from 6.0.0 to 6.0.1 (vectordotdev#17673) [8d98bb8](answerbook/vector@8d98bb8) - GitHub
* **deps**: Bump quanta from 0.11.0 to 0.11.1 (vectordotdev#17524) [2388c2f](answerbook/vector@2388c2f) - GitHub
* **deps**: Bump quote from 1.0.27 to 1.0.28 (vectordotdev#17496) [cc30746](answerbook/vector@cc30746) - GitHub
* **deps**: Bump quote from 1.0.28 to 1.0.29 (vectordotdev#17798) [cba983e](answerbook/vector@cba983e) - GitHub
* **deps**: Bump quote from 1.0.28 to 1.0.29 (vectordotdev#17815) [bf9828d](answerbook/vector@bf9828d) - GitHub
* **deps**: bump rdkafka from 0.30.0 to 0.31.0 (vectordotdev#17428) [e7fa8d3](answerbook/vector@e7fa8d3) - GitHub
* **deps**: Bump rdkafka from 0.31.0 to 0.32.2 (vectordotdev#17664) [ac68a7b](answerbook/vector@ac68a7b) - GitHub
* **deps**: bump regex from 1.8.1 to 1.8.2 (vectordotdev#17469) [897e45d](answerbook/vector@897e45d) - GitHub
* **deps**: bump regex from 1.8.2 to 1.8.3 (vectordotdev#17494) [5d90cff](answerbook/vector@5d90cff) - GitHub
* **deps**: Bump regex from 1.8.3 to 1.8.4 (vectordotdev#17601) [657758d](answerbook/vector@657758d) - GitHub
* **deps**: bump reqwest from 0.11.17 to 0.11.18 (vectordotdev#17420) [2ed8ec7](answerbook/vector@2ed8ec7) - GitHub
* **deps**: bump security-framework from 2.9.0 to 2.9.1 (vectordotdev#17441) [ac0c7e8](answerbook/vector@ac0c7e8) - GitHub
* **deps**: Bump serde from 1.0.163 to 1.0.164 (vectordotdev#17632) [e35150e](answerbook/vector@e35150e) - GitHub
* **deps**: Bump serde_json from 1.0.96 to 1.0.97 (vectordotdev#17701) [25131ef](answerbook/vector@25131ef) - GitHub
* **deps**: Bump serde_json from 1.0.97 to 1.0.99 (vectordotdev#17754) [e07158c](answerbook/vector@e07158c) - GitHub
* **deps**: Bump serde_yaml from 0.9.21 to 0.9.22 (vectordotdev#17756) [e164b36](answerbook/vector@e164b36) - GitHub
* **deps**: Bump sha2 from 0.10.6 to 0.10.7 (vectordotdev#17698) [d122d32](answerbook/vector@d122d32) - GitHub
* **deps**: Bump tempfile from 3.5.0 to 3.6.0 (vectordotdev#17617) [c55c9ec](answerbook/vector@c55c9ec) - GitHub
* **deps**: Bump tokio from 1.28.1 to 1.28.2 (vectordotdev#17525) [cc703da](answerbook/vector@cc703da) - GitHub
* **deps**: Bump tokio from 1.28.2 to 1.29.0 (vectordotdev#17776) [e26e8b8](answerbook/vector@e26e8b8) - GitHub
* **deps**: bump toml from 0.7.3 to 0.7.4 (vectordotdev#17440) [91ba052](answerbook/vector@91ba052) - GitHub
* **deps**: Bump toml from 0.7.4 to 0.7.5 (vectordotdev#17751) [35c4581](answerbook/vector@35c4581) - GitHub
* **deps**: Bump tower-http from 0.4.0 to 0.4.1 (vectordotdev#17711) [e5e6b96](answerbook/vector@e5e6b96) - GitHub
* **deps**: Bump url from 2.3.1 to 2.4.0 (vectordotdev#17608) [d956092](answerbook/vector@d956092) - GitHub
* **deps**: Bump uuid from 1.3.3 to 1.3.4 (vectordotdev#17682) [c97d619](answerbook/vector@c97d619) - GitHub
* **deps**: Bump uuid from 1.3.4 to 1.4.0 (vectordotdev#17775) [935babf](answerbook/vector@935babf) - GitHub
* **deps**: Bump wasm-bindgen from 0.2.86 to 0.2.87 (vectordotdev#17672) [19c4d4f](answerbook/vector@19c4d4f) - GitHub
* **deps**: Bump wiremock from 0.5.18 to 0.5.19 (vectordotdev#17618) [460bbc7](answerbook/vector@460bbc7) - GitHub
* **deps**: Bump xml-rs from 0.8.4 to 0.8.14 (vectordotdev#17607) [a932489](answerbook/vector@a932489) - GitHub
* **deps**: Drop use of `hashlink` crate (vectordotdev#17678) [41ee394](answerbook/vector@41ee394) - GitHub
* **deps**: Export more common bits for components (vectordotdev#17788) [062224b](answerbook/vector@062224b) - GitHub
* **deps**: Update fs_extra to 1.3.0 (vectordotdev#17458) [299fd6a](answerbook/vector@299fd6a) - GitHub
* **deps**: Upgrade Ruby version to 3.1.4 (vectordotdev#17722) [ddebde9](answerbook/vector@ddebde9) - GitHub
* **deps**: Upgrade rust to 1.70.0 (vectordotdev#17585) [6c48565](answerbook/vector@6c48565) - GitHub
* **dev**: Add @dsmith3197 to CODEOWNERS (vectordotdev#17729) [a08443c](answerbook/vector@a08443c) - GitHub
* **docs**: Add info about Vector Operator to Kubernetes instalation page (vectordotdev#17432) [54d9c99](answerbook/vector@54d9c99) - GitHub
* **docs**: add instructions for regenerating component docs and licenses (vectordotdev#17828) [93ef6c3](answerbook/vector@93ef6c3) - GitHub
* **docs**: Add Log Namespacing docs (vectordotdev#16571) [7d098e4](answerbook/vector@7d098e4) - GitHub
* **docs**: add note about const strings (vectordotdev#17774) [d7bc531](answerbook/vector@d7bc531) - GitHub
* **docs**: Clarify `bytes` framing for streams (vectordotdev#17745) [7d10fc9](answerbook/vector@7d10fc9) - GitHub
* **docs**: Clarify when component received and sent bytes events should be emitted (vectordotdev#17464) [547783d](answerbook/vector@547783d) - GitHub
* **docs**: Move CONTRIBUTING.md to top-level (vectordotdev#17744) [7a0dec1](answerbook/vector@7a0dec1) - GitHub
* Download submodules in the CI checkouts (vectordotdev#17760) [5417a06](answerbook/vector@5417a06) - GitHub
* Dropped error field from StreamClosed Error (vectordotdev#17693) [ee480cd](answerbook/vector@ee480cd) - GitHub
* **enrichment**: avoid importing vector-common in enrichment module (vectordotdev#17653) [45a28f8](answerbook/vector@45a28f8) - GitHub
* **enterprise**: Extend library functionality for secret scanning (vectordotdev#17483) [541bb00](answerbook/vector@541bb00) - GitHub
* **external docs**: fix reference to supported aarch64 architecture (vectordotdev#17553) [247bb80](answerbook/vector@247bb80) - GitHub
* **external docs**: update fluentd link (vectordotdev#17436) [187f142](answerbook/vector@187f142) - GitHub
* Fix publish workflow for older OS images (vectordotdev#17787) [ab39c6a](answerbook/vector@ab39c6a) - GitHub
* **flush on shutdown**: validate s3 sink flushes (vectordotdev#17667) [c21f892](answerbook/vector@c21f892) - GitHub
* **kubernetes_logs source**: Add warning about Windows support (vectordotdev#17762) [a53c7a2](answerbook/vector@a53c7a2) - GitHub
* **kubernetes**: Bump k8s manifests to 0.22.0 (vectordotdev#17467) [f547871](answerbook/vector@f547871) - GitHub
* **observability**: emit `component_sent` events by `source` and `service` (vectordotdev#17549) [dcf7f9a](answerbook/vector@dcf7f9a) - GitHub
* **observability**: ensure `sent_event` and `received_event` metrics are estimated json size (vectordotdev#17465) [3b2a2be](answerbook/vector@3b2a2be) - GitHub
* **observability**: Have `tower_limit` use configured log level (vectordotdev#17715) [08099a8](answerbook/vector@08099a8) - GitHub
* **observability**: remove deprecated internal metrics + massive cleanup to vector top and graphql API (vectordotdev#17516) [98c54ad](answerbook/vector@98c54ad) - GitHub
* **observability**: remove more deprecated internal metrics (vectordotdev#17542) [b0ed167](answerbook/vector@b0ed167) - GitHub
* **observability**: set source fields to mean service (vectordotdev#17470) [670bdea](answerbook/vector@670bdea) - GitHub
* **releasing**: Prepare v0.30.0 release [af2b2af](answerbook/vector@af2b2af) - Jesse Szwedko
* **releasing**: Prepare v0.31.0 release [0f13b22](answerbook/vector@0f13b22) - Jesse Szwedko
* remove custom async sleep impl (vectordotdev#17493) [b28d915](answerbook/vector@b28d915) - GitHub
* Remove links to roadmap (vectordotdev#17554) [349c718](answerbook/vector@349c718) - GitHub
* Revert all submodule introductions to fix CI (vectordotdev#17800) [d8d57e5](answerbook/vector@d8d57e5) - GitHub
* RFC for Data Volume Insights (vectordotdev#17322) [a551f33](answerbook/vector@a551f33) - GitHub
* **sinks**: Drop the custom `SinkContext::default` implementation (vectordotdev#17804) [e66e285](answerbook/vector@e66e285) - GitHub
* **sinks**: mark VectorSink::from_event_sink as deprecated (vectordotdev#17649) [0dc450f](answerbook/vector@0dc450f) - GitHub
* **statsd sink**: refactor `statsd` sink to stream-based style (vectordotdev#16199) [2a76cac](answerbook/vector@2a76cac) - GitHub
* update `vrl` to `0.4.0` (vectordotdev#17378) [426d660](answerbook/vector@426d660) - GitHub
* Update the NOTICE file (vectordotdev#17430) [9a44e6e](answerbook/vector@9a44e6e) - GitHub
* Upgrade aws-smithy and aws-sdk crates (vectordotdev#17731) [6a6b42b](answerbook/vector@6a6b42b) - GitHub
* **website**: Fix upgrade guide dates [80de738](answerbook/vector@80de738) - Jesse Szwedko

### Features

* add metadata support to `set_semantic_meaning` (vectordotdev#17730) [44be378](answerbook/vector@44be378) - GitHub
* **codecs**: add lossy option to `gelf`, `native_json`, and `syslog` deserializers (vectordotdev#17680) [2dfa850](answerbook/vector@2dfa850) - GitHub
* **codecs**: Add lossy option to JSON deserializer (vectordotdev#17628) [bf7d796](answerbook/vector@bf7d796) - GitHub
* **configurable shutdown duration**: make shutdown duration configurable (vectordotdev#17479) [23ed0e3](answerbook/vector@23ed0e3) - GitHub
* **error code when shutdown fails**: set exit flag to non-zero when shutdown times out (vectordotdev#17676) [cc52c0e](answerbook/vector@cc52c0e) - GitHub
* **internal telemetry at shutdown**: close internal sources after external ones (vectordotdev#17741) [812929b](answerbook/vector@812929b) - GitHub
* **journald source**: add journal_namespace option (vectordotdev#17648) [a324a07](answerbook/vector@a324a07) - GitHub
* **kinesis sinks**: implement full retry of partial failures in firehose/streams (vectordotdev#17535) [bebac21](answerbook/vector@bebac21) - GitHub
* **prometheus**: add more compression algorithms to Prometheus Remote Write (vectordotdev#17334) [380d7ad](answerbook/vector@380d7ad) - GitHub
* track runtime schema definitions for log events (vectordotdev#17692) [6eecda5](answerbook/vector@6eecda5) - GitHub

### Miscellaneous

* Merge pull request vectordotdev#371 from answerbook/feature/LOG-18200 [491ea31](answerbook/vector@491ea31) - GitHub [LOG-18200](https://logdna.atlassian.net/browse/LOG-18200)
* Merge pull request vectordotdev#374 from answerbook/dominic/LOG-18535-temp-revert [df87df7](answerbook/vector@df87df7) - GitHub [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535) [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535)
* Revert "Merge pull request vectordotdev#370 from answerbook/dominic/LOG-18535" [73a8b28](answerbook/vector@73a8b28) - dominic-mcallister-logdna [LOG-18535](https://logdna.atlassian.net/browse/LOG-18535)
* Merge branch 'master' into feature/LOG-18200 [fcd21f3](answerbook/vector@fcd21f3) - Darin Spivey [LOG-18200](https://logdna.atlassian.net/browse/LOG-18200)
* Merge tag 'v0.31.0' into upstream-0.31.0 [9443fb1](answerbook/vector@9443fb1) - Darin Spivey
* Update VRL to `0.5.0` (vectordotdev#17793) [671aa79](answerbook/vector@671aa79) - GitHub
* fix `demo_logs` metadata source name (vectordotdev#17689) [83af7ea](answerbook/vector@83af7ea) - GitHub
* enhancement(s3 source) Add minimal support to unwrap an S3-SQS event from an SNS event (vectordotdev#17352) [7a7bc9a](answerbook/vector@7a7bc9a) - GitHub
* Additional notes on proposing new integrations (vectordotdev#17658) [2ad964d](answerbook/vector@2ad964d) - GitHub
* **ci**: reduce billable time of Test Suite (vectordotdev#17714) [bc69255](answerbook/vector@bc69255) - GitHub
* **ci**: refactor logic for int test file path changes detection (vectordotdev#17725) [92a36e0](answerbook/vector@92a36e0) - GitHub
* **compression**: zstd compression support (vectordotdev#17371) [ced219e](answerbook/vector@ced219e) - GitHub
* **dev**:  move blocked/waiting gardener issues to triage on comment (vectordotdev#17588) [6b34868](answerbook/vector@6b34868) - GitHub
* explain how to run tests locally (vectordotdev#17783) [3b67a80](answerbook/vector@3b67a80) - GitHub
* remove aggregator beta warning (vectordotdev#17750) [94e3f15](answerbook/vector@94e3f15) - GitHub
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spencergilbert spencergilbert spencergilbert approved these changes

@neuronull neuronull neuronull approved these changes

@bruceg bruceg bruceg approved these changes

Assignees
No one assigned
Labels
domain: sinks Anything related to the Vector's sinks
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tobz @neuronull @bruceg @spencergilbert