enhancement(console sink): Add end-to-end acknowledgement support

[bruceg]

This is split into two commits. The first completes what is required of #13020, the second is for #9857. This is likely the most trivial example of both of those.

[netlify]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	51f629a
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62a0ad1b5a2857000806d128

[github-actions]

Soak Test Results

Baseline: 4c00211
Comparison: f0de4f2
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_loki	255.06KiB	1.73	100.00%	14.41MiB	983.31KiB	20.04KiB	0	0.0666232	14.66MiB	915.14KiB	18.71KiB	0	0.060951	False	False
datadog_agent_remap_datadog_logs	772.77KiB	1.02	100.00%	73.88MiB	2.32MiB	48.47KiB	0	0.0314035	74.63MiB	3.74MiB	78.13KiB	0	0.0500756	False	False
syslog_log2metric_splunk_hec_metrics	130.65KiB	0.69	100.00%	18.38MiB	909.13KiB	18.5KiB	0	0.0482817	18.51MiB	605.92KiB	12.36KiB	0	0.0319574	False	False
syslog_splunk_hec_logs	124.02KiB	0.67	100.00%	18.16MiB	701.08KiB	14.31KiB	0	0.0376875	18.28MiB	522.67KiB	10.68KiB	0	0.0279106	False	False
syslog_humio_logs	94.74KiB	0.52	100.00%	17.67MiB	631.16KiB	12.94KiB	0	0.0348711	17.76MiB	646.48KiB	13.23KiB	0	0.0355316	False	False
datadog_agent_remap_blackhole_acks	237.45KiB	0.33	97.59%	69.94MiB	4.23MiB	88.14KiB	0	0.0604186	70.17MiB	2.75MiB	57.53KiB	0	0.0392306	False	False
syslog_regex_logs2metric_ddmetrics	42.35KiB	0.33	98.66%	12.35MiB	527.31KiB	10.74KiB	0	0.0416721	12.4MiB	654.99KiB	13.32KiB	0	0.0515896	False	False
http_to_http_noack	75.85KiB	0.31	99.98%	23.77MiB	962.55KiB	19.61KiB	0	0.0395362	23.84MiB	250.14KiB	5.11KiB	0	0.0102423	False	False
datadog_agent_remap_datadog_logs_acks	53.82KiB	0.08	65.13%	69.65MiB	707.28KiB	14.44KiB	0	0.00991534	69.7MiB	2.66MiB	55.58KiB	0	0.0381285	False	False
syslog_log2metric_humio_metrics	5.25KiB	0.04	43.47%	14.58MiB	403.19KiB	8.22KiB	0	0.0270009	14.58MiB	193.08KiB	3.95KiB	0	0.0129257	False	False
fluent_elasticsearch	16.38KiB	0.02	88.25%	79.46MiB	516.48KiB	10.41KiB	0	0.00634643	79.47MiB	51.95KiB	1.05KiB	0	0.000638194	False	False
http_to_http_json	527.38B	0	4.16%	23.84MiB	341.49KiB	6.98KiB	0	0.0139841	23.84MiB	342.69KiB	7.0KiB	0	0.0140332	False	False
splunk_hec_to_splunk_hec_logs_acks	161.57B	0	0.61%	23.78MiB	711.88KiB	14.49KiB	0	0.0292327	23.78MiB	711.16KiB	14.48KiB	0	0.0292031	False	False
splunk_hec_to_splunk_hec_logs_noack	865.92B	0	6.93%	23.84MiB	335.58KiB	6.87KiB	0	0.0137443	23.84MiB	335.8KiB	6.88KiB	0	0.013753	False	False
splunk_hec_indexer_ack_blackhole	-8.99KiB	-0.04	32.13%	23.77MiB	732.0KiB	14.9KiB	0	0.0300645	23.76MiB	774.98KiB	15.77KiB	0	0.0318417	False	False
http_pipelines_blackhole_acks	-3.26KiB	-0.07	15.56%	4.41MiB	572.28KiB	11.7KiB	0	0.126687	4.41MiB	577.99KiB	11.81KiB	0	0.128044	False	False
http_pipelines_blackhole	-8.54KiB	-0.18	38.60%	4.66MiB	577.97KiB	11.84KiB	0	0.12098	4.66MiB	594.22KiB	12.11KiB	0	0.124605	False	False
file_to_blackhole	-186.04KiB	-0.19	95.06%	95.31MiB	3.12MiB	64.79KiB	0	0.0327029	95.13MiB	3.34MiB	68.99KiB	0	0.0350895	False	False
http_to_http_acks	-54.52KiB	-0.3	18.26%	17.91MiB	7.93MiB	165.72KiB	0	0.442506	17.86MiB	8.05MiB	168.18KiB	0	0.450411	True	True
splunk_hec_route_s3	-109.65KiB	-0.55	90.85%	19.61MiB	2.18MiB	45.54KiB	0	0.111058	19.51MiB	2.22MiB	46.32KiB	0	0.113575	False	False
http_pipelines_no_grok_blackhole	-118.07KiB	-0.58	99.76%	19.83MiB	1.27MiB	26.56KiB	0	0.0639418	19.71MiB	1.36MiB	28.41KiB	0	0.0690345	False	False
socket_to_socket_blackhole	-172.17KiB	-1.23	100.00%	13.7MiB	572.31KiB	11.65KiB	0	0.0407946	13.53MiB	519.13KiB	10.58KiB	0	0.0374639	False	False
splunk_transforms_splunk3	-169.07KiB	-1.23	99.00%	13.38MiB	2.22MiB	46.36KiB	0	0.165784	13.22MiB	2.22MiB	46.49KiB	0	0.168028	False	False
datadog_agent_remap_blackhole	-883.24KiB	-1.39	100.00%	62.16MiB	6.39MiB	133.83KiB	0	0.10283	61.3MiB	6.22MiB	130.25KiB	0	0.101384	False	False

[github-actions]

Soak Test Results

Baseline: da11de5
Comparison: 51f629a
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_loki	260.19KiB	1.71	100.00%	14.85MiB	542.99KiB	11.06KiB	0	0.0357066	15.1MiB	215.61KiB	4.4KiB	0	0.0139398	False	False
datadog_agent_remap_datadog_logs_acks	990.2KiB	1.36	100.00%	70.93MiB	1.16MiB	24.28KiB	0	0.0163716	71.9MiB	2.52MiB	52.64KiB	0	0.0349823	False	False
datadog_agent_remap_datadog_logs	833.15KiB	1.18	100.00%	69.1MiB	1.6MiB	33.45KiB	0	0.0231674	69.91MiB	3.57MiB	74.67KiB	0	0.0511067	False	False
syslog_regex_logs2metric_ddmetrics	120.97KiB	0.93	100.00%	12.74MiB	706.01KiB	14.37KiB	0	0.054108	12.86MiB	759.03KiB	15.44KiB	0	0.0576372	False	False
syslog_log2metric_splunk_hec_metrics	165.17KiB	0.89	100.00%	18.06MiB	1.13MiB	23.46KiB	0	0.0623679	18.22MiB	1.08MiB	22.6KiB	0	0.0594778	False	False
socket_to_socket_blackhole	107.06KiB	0.77	100.00%	13.54MiB	564.14KiB	11.48KiB	0	0.0406824	13.64MiB	487.59KiB	9.95KiB	0	0.0348929	False	False
syslog_splunk_hec_logs	124.83KiB	0.69	100.00%	17.59MiB	721.91KiB	14.74KiB	0	0.0400613	17.72MiB	715.63KiB	14.63KiB	0	0.0394394	False	False
datadog_agent_remap_blackhole_acks	478.67KiB	0.69	100.00%	68.02MiB	3.82MiB	79.77KiB	0	0.0561988	68.48MiB	2.6MiB	54.37KiB	0	0.037991	False	False
syslog_humio_logs	97.07KiB	0.52	100.00%	18.21MiB	146.32KiB	3.0KiB	0	0.00784449	18.31MiB	152.91KiB	3.13KiB	0	0.00815518	False	False
http_to_http_noack	95.02KiB	0.39	100.00%	23.75MiB	1.03MiB	21.49KiB	0	0.043377	23.85MiB	247.21KiB	5.05KiB	0	0.0101213	False	False
fluent_elasticsearch	126.48KiB	0.16	98.47%	79.35MiB	2.53MiB	52.1KiB	0	0.0318858	79.47MiB	54.64KiB	1.1KiB	0	0.00067126	False	False
splunk_hec_to_splunk_hec_logs_noack	-350.83B	-0	2.77%	23.83MiB	341.05KiB	6.98KiB	0	0.0139706	23.83MiB	341.38KiB	6.99KiB	0	0.0139844	False	False
http_to_http_json	-1.19KiB	-0	9.65%	23.85MiB	337.38KiB	6.89KiB	0	0.013814	23.84MiB	340.64KiB	6.95KiB	0	0.0139483	False	False
http_pipelines_blackhole_acks	-1.26KiB	-0.03	6.24%	4.43MiB	550.34KiB	11.25KiB	0	0.121234	4.43MiB	558.82KiB	11.41KiB	0	0.123135	False	False
splunk_hec_to_splunk_hec_logs_acks	-9.7KiB	-0.04	32.25%	23.76MiB	792.86KiB	16.13KiB	0	0.0325844	23.75MiB	829.04KiB	16.86KiB	0	0.0340848	False	False
splunk_hec_indexer_ack_blackhole	-14.75KiB	-0.06	51.21%	23.78MiB	707.77KiB	14.41KiB	0	0.0290637	23.76MiB	768.47KiB	15.63KiB	0	0.0315755	False	False
file_to_blackhole	-105.13KiB	-0.11	90.85%	95.3MiB	1.95MiB	40.53KiB	0	0.0204452	95.2MiB	2.29MiB	47.29KiB	0	0.0240558	False	False
http_pipelines_blackhole	-10.93KiB	-0.23	46.70%	4.64MiB	599.66KiB	12.29KiB	0	0.126226	4.63MiB	613.94KiB	12.51KiB	0	0.129531	False	False
splunk_hec_route_s3	-67.01KiB	-0.33	70.71%	19.69MiB	2.15MiB	44.9KiB	0	0.109105	19.62MiB	2.16MiB	45.2KiB	0	0.110162	False	False
splunk_transforms_splunk3	-62.08KiB	-0.46	65.38%	13.25MiB	2.23MiB	46.59KiB	0	0.168361	13.19MiB	2.23MiB	46.61KiB	0	0.168794	False	False
datadog_agent_remap_blackhole	-361.5KiB	-0.53	100.00%	66.7MiB	1.79MiB	37.44KiB	0	0.0267654	66.35MiB	1.5MiB	31.35KiB	0	0.0225593	False	False
http_pipelines_no_grok_blackhole	-128.35KiB	-0.63	99.62%	19.79MiB	1.49MiB	31.09KiB	0	0.0750719	19.67MiB	1.52MiB	31.61KiB	0	0.0770118	False	False
syslog_log2metric_humio_metrics	-107.42KiB	-0.73	100.00%	14.43MiB	592.45KiB	12.08KiB	0	0.0400855	14.33MiB	447.53KiB	9.16KiB	0	0.0305021	False	False
http_to_http_acks	-139.29KiB	-0.76	44.91%	17.89MiB	7.85MiB	164.05KiB	0	0.438627	17.75MiB	7.95MiB	166.16KiB	0	0.44766	True	True