Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(http sink): cert verification with proxy enabled #13759

Merged

Conversation

ntim
Copy link
Contributor

@ntim ntim commented Jul 29, 2022

As described in #13683, when enabling the proxy for the http sink, the hyper ProxyConnector is not configured properly with the user supplied tls settings. Therefore, certificate verification using a private PKI or client cert authentication are broken.

@bits-bot
Copy link

bits-bot commented Jul 29, 2022

CLA assistant check
All committers have signed the CLA.

@netlify
Copy link

netlify bot commented Jul 29, 2022

Deploy Preview for vector-project canceled.

Name Link
🔨 Latest commit 60becc5
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/62eab06b27afbd00085edae7

Copy link
Contributor

@tobz tobz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me and it's nice that's such a simple change. Left one change request as, despite the simplicity of the change itself, I think we just need to document it better to disambiguate what's actually happening.

src/http.rs Outdated Show resolved Hide resolved
@tobz tobz self-assigned this Aug 1, 2022
@ntim ntim requested a review from tobz August 3, 2022 15:51
@tobz
Copy link
Contributor

tobz commented Aug 3, 2022

@ntim Thanks for that. Given that it looks like the tests are passing/should pass without issue, the only remaining bit is signing our CLA. 🖊️ 📜

@tobz tobz added the ci-condition: integration tests enable Run integration tests on this PR label Aug 3, 2022
@github-actions
Copy link

github-actions bot commented Aug 3, 2022

Soak Test Results

Baseline: 4a14553
Comparison: a5d5ddf
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole 898.79KiB 1.43 100.00% 61.27MiB 4.65MiB 96.97KiB 0 0.075923 62.14MiB 3.81MiB 79.54KiB 0 0.0613519 False False
syslog_splunk_hec_logs 199.18KiB 1.17 100.00% 16.63MiB 747.12KiB 15.22KiB 0 0.0438719 16.82MiB 659.18KiB 13.44KiB 0 0.0382604 False False
syslog_log2metric_humio_metrics 148.26KiB 1.16 100.00% 12.5MiB 667.89KiB 13.62KiB 0 0.0521573 12.65MiB 640.65KiB 13.05KiB 0 0.049457 False False
datadog_agent_remap_blackhole_acks 679.93KiB 1.08 100.00% 61.22MiB 5.13MiB 106.71KiB 0 0.0837263 61.88MiB 3.85MiB 80.58KiB 0 0.0622499 False False
http_pipelines_blackhole_acks 9.07KiB 0.77 99.96% 1.15MiB 102.91KiB 2.1KiB 0 0.087461 1.16MiB 72.27KiB 1.47KiB 0 0.0609489 False False
syslog_humio_logs 99.47KiB 0.57 100.00% 17.19MiB 145.52KiB 2.97KiB 0 0.00826626 17.29MiB 185.67KiB 3.8KiB 0 0.0104877 False False
syslog_regex_logs2metric_ddmetrics 68.02KiB 0.53 99.98% 12.43MiB 590.08KiB 12.03KiB 0 0.0463396 12.5MiB 663.87KiB 13.52KiB 0 0.051858 False False
syslog_log2metric_splunk_hec_metrics 91.23KiB 0.5 100.00% 18.0MiB 563.41KiB 11.49KiB 0 0.0305689 18.08MiB 830.95KiB 16.92KiB 0 0.0448626 False False
splunk_hec_to_splunk_hec_logs_noack 20.54KiB 0.08 91.36% 23.82MiB 483.34KiB 9.87KiB 0 0.0198146 23.84MiB 332.39KiB 6.78KiB 0 0.013615 False False
splunk_hec_indexer_ack_blackhole 5.09KiB 0.02 15.99% 23.74MiB 882.73KiB 17.95KiB 0 0.0362975 23.75MiB 869.46KiB 17.69KiB 0 0.0357447 False False
splunk_hec_to_splunk_hec_logs_acks 1.23KiB 0.01 4.25% 23.76MiB 801.59KiB 16.31KiB 0 0.0329377 23.76MiB 797.14KiB 16.22KiB 0 0.0327531 False False
enterprise_http_to_http 714.33B 0 7.61% 23.85MiB 252.25KiB 5.15KiB 0 0.0103287 23.85MiB 253.07KiB 5.18KiB 0 0.010362 False False
file_to_blackhole -56.75KiB -0.06 40.25% 95.34MiB 3.25MiB 67.42KiB 0 0.0341058 95.28MiB 4.03MiB 83.69KiB 0 0.0422491 False False
http_to_http_json -33.92KiB -0.14 99.23% 23.84MiB 351.2KiB 7.17KiB 0 0.0143807 23.81MiB 514.84KiB 10.51KiB 0 0.0211108 False False
fluent_elasticsearch -175.57KiB -0.22 100.00% 79.47MiB 54.49KiB 1.1KiB 0 0.000669444 79.3MiB 1.52MiB 31.23KiB 0 0.0191298 False False
http_pipelines_blackhole -7.33KiB -0.44 99.88% 1.62MiB 11.36KiB 237.75B 0 0.00683804 1.62MiB 110.66KiB 2.26KiB 0 0.0668909 False False
http_to_http_noack -122.59KiB -0.5 100.00% 23.84MiB 408.05KiB 8.34KiB 0 0.0167142 23.72MiB 1.23MiB 25.68KiB 0 0.0519508 False False
splunk_hec_route_s3 -107.47KiB -0.55 90.00% 18.94MiB 2.25MiB 46.84KiB 0 0.118781 18.84MiB 2.18MiB 45.52KiB 0 0.115548 False False
datadog_agent_remap_datadog_logs_acks -422.4KiB -0.65 99.99% 63.18MiB 2.62MiB 54.93KiB 0 0.0415194 62.77MiB 4.5MiB 93.59KiB 0 0.0716128 False False
http_pipelines_no_grok_blackhole -86.78KiB -0.74 99.98% 11.39MiB 53.17KiB 1.09KiB 0 0.00455745 11.31MiB 1.12MiB 23.37KiB 0 0.0992493 False False
datadog_agent_remap_datadog_logs -521.79KiB -0.79 100.00% 64.14MiB 449.6KiB 9.2KiB 0 0.00684366 63.63MiB 4.28MiB 89.23KiB 0 0.0673201 False False
http_to_http_acks -206.3KiB -1.11 58.83% 18.18MiB 8.43MiB 176.19KiB 0 0.46348 17.98MiB 8.58MiB 179.14KiB 0 0.477212 True True
syslog_loki -178.12KiB -1.2 100.00% 14.45MiB 546.38KiB 11.2KiB 0 0.0369092 14.28MiB 871.57KiB 17.72KiB 0 0.0595939 False False
socket_to_socket_blackhole -525.38KiB -2.17 100.00% 23.62MiB 324.89KiB 6.63KiB 0 0.01343 23.11MiB 272.12KiB 5.55KiB 0 0.0114982 False False
http_text_to_http_json -1.15MiB -2.89 100.00% 39.79MiB 715.26KiB 14.6KiB 0 0.0175522 38.64MiB 821.58KiB 16.77KiB 0 0.0207616 False False

@github-actions
Copy link

github-actions bot commented Aug 3, 2022

Soak Test Results

Baseline: 7d9e3e6
Comparison: 60becc5
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_pipelines_blackhole 108.61KiB 7.31 100.00% 1.45MiB 132.25KiB 2.7KiB 0 0.0889531 1.56MiB 143.48KiB 2.92KiB 0 0.0899413 False False
socket_to_socket_blackhole 906.99KiB 4.09 100.00% 21.66MiB 422.66KiB 8.63KiB 0 0.0190563 22.54MiB 489.7KiB 10.0KiB 0 0.0212115 False False
syslog_humio_logs 355.6KiB 2.21 100.00% 15.68MiB 1.02MiB 21.23KiB 0 0.0647627 16.03MiB 1.02MiB 21.33KiB 0 0.0634211 False False
syslog_log2metric_splunk_hec_metrics 352.98KiB 1.93 100.00% 17.88MiB 783.6KiB 15.96KiB 0 0.0428012 18.22MiB 924.81KiB 18.83KiB 0 0.0495585 False False
syslog_splunk_hec_logs 303.14KiB 1.79 100.00% 16.51MiB 738.74KiB 15.05KiB 0 0.043694 16.8MiB 718.41KiB 14.64KiB 0 0.0417427 False False
http_text_to_http_json 643.81KiB 1.71 100.00% 36.86MiB 1.09MiB 22.84KiB 0 0.0296337 37.49MiB 1.13MiB 23.71KiB 0 0.0302471 False False
datadog_agent_remap_datadog_logs_acks 655.13KiB 1.05 100.00% 61.04MiB 3.39MiB 70.8KiB 0 0.0554554 61.68MiB 4.73MiB 98.56KiB 0 0.0767458 False False
syslog_loki 134.85KiB 0.93 100.00% 14.22MiB 522.47KiB 10.7KiB 0 0.0358751 14.35MiB 844.58KiB 17.17KiB 0 0.0574607 False False
datadog_agent_remap_blackhole_acks 604.58KiB 0.92 100.00% 63.91MiB 4.55MiB 94.76KiB 0 0.0711827 64.5MiB 2.83MiB 59.17KiB 0 0.0437998 False False
datadog_agent_remap_datadog_logs 516.74KiB 0.81 100.00% 62.29MiB 724.29KiB 14.84KiB 0 0.0113529 62.79MiB 4.25MiB 88.42KiB 0 0.0676052 False False
datadog_agent_remap_blackhole 296.34KiB 0.45 99.35% 64.22MiB 4.14MiB 86.25KiB 0 0.0644233 64.51MiB 3.18MiB 66.38KiB 0 0.0493061 False False
http_pipelines_blackhole_acks 4.91KiB 0.44 89.67% 1.08MiB 118.74KiB 2.42KiB 0 0.107085 1.09MiB 88.62KiB 1.81KiB 0 0.079563 False False
syslog_log2metric_humio_metrics 47.64KiB 0.36 99.91% 12.86MiB 361.98KiB 7.39KiB 0 0.0274817 12.91MiB 600.53KiB 12.22KiB 0 0.0454276 False False
splunk_hec_to_splunk_hec_logs_noack 26.72KiB 0.11 96.29% 23.81MiB 535.93KiB 10.94KiB 0 0.0219753 23.84MiB 326.65KiB 6.67KiB 0 0.0133796 False False
http_pipelines_no_grok_blackhole 2.21KiB 0.02 7.11% 10.9MiB 303.41KiB 6.19KiB 0 0.0271696 10.91MiB 1.15MiB 23.94KiB 0 0.105423 False False
splunk_hec_to_splunk_hec_logs_acks -12.88KiB -0.05 38.88% 23.75MiB 854.31KiB 17.38KiB 0 0.0351203 23.74MiB 906.8KiB 18.44KiB 0 0.0372978 False False
enterprise_http_to_http -13.56KiB -0.06 83.72% 23.85MiB 261.29KiB 5.33KiB 0 0.0106972 23.84MiB 396.92KiB 8.12KiB 0 0.0162592 False False
file_to_blackhole -62.31KiB -0.06 41.99% 95.34MiB 3.79MiB 78.54KiB 0 0.0397709 95.28MiB 3.89MiB 80.71KiB 0 0.0408117 False False
splunk_hec_indexer_ack_blackhole -17.88KiB -0.07 51.56% 23.76MiB 845.86KiB 17.21KiB 0 0.0347585 23.74MiB 929.58KiB 18.9KiB 0 0.0382268 False False
http_to_http_json -43.92KiB -0.18 99.88% 23.85MiB 349.95KiB 7.14KiB 0 0.0143286 23.8MiB 564.08KiB 11.51KiB 0 0.0231372 False False
fluent_elasticsearch -162.35KiB -0.2 100.00% 79.47MiB 53.19KiB 1.08KiB 0 0.000653458 79.31MiB 1.45MiB 29.87KiB 0 0.0182996 False False
http_to_http_noack -72.31KiB -0.3 99.58% 23.82MiB 587.67KiB 12.01KiB 0 0.0240899 23.75MiB 1.06MiB 22.19KiB 0 0.0448064 False False
splunk_hec_route_s3 -89.16KiB -0.47 82.07% 18.66MiB 2.28MiB 47.44KiB 0 0.12214 18.57MiB 2.22MiB 46.44KiB 0 0.119429 False False
syslog_regex_logs2metric_ddmetrics -104.85KiB -0.83 100.00% 12.4MiB 718.42KiB 14.63KiB 0 0.0565843 12.29MiB 650.32KiB 13.26KiB 0 0.0516468 False False
http_to_http_acks -946.14KiB -5.05 99.98% 18.29MiB 8.3MiB 173.55KiB 0 0.45374 17.37MiB 8.6MiB 179.37KiB 0 0.494835 True True

@tobz
Copy link
Contributor

tobz commented Aug 3, 2022

The failed integration tests we can ignore: they depend on secret credentials that aren't injected into CI workflows based on outside forks.

I'm just rerunning the unit tests for macOS, as there was a weird test failure that had some TLS-related output. Might be a red herring, since it didn't fail for any other platforms, but we'll see.

@tobz
Copy link
Contributor

tobz commented Aug 4, 2022

Gonna try the macOS unit tests one more time then try and debug it locally... still not sure what's up since those unit tests don't appear to use proxying or TLS in any way. 🤔

@tobz tobz merged commit 57db12a into vectordotdev:master Aug 4, 2022
@tobz
Copy link
Contributor

tobz commented Aug 4, 2022

Looks like it was indeed a bit flaky.

Thanks again for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants