fix(deps): load default and legacy openssl providers

[dsmith3197]

The OpenSSL documentation is unclear about whether or not the default provider remains loaded when calling try_load with retain_fallbacks=true. We now explicitly load the default provider to be safe.

This also fixes the --openssl-legacy-provider cli arg, which could previously only be set to true.

The following now work

vector --openssl-legacy-provider
vector --openssl-legacy-provider=true
vector --openssl-legacy-provider=false
VECTOR_OPENSSL_LEGACY_PROVIDER=true ./target/debug/vector
VECTOR_OPENSSL_LEGACY_PROVIDER=false ./target/debug/vector

But the below still result in an error

vector --openssl-legacy-provider true
vector --openssl-legacy-provider false

In the future, we will refactor this to allow the user to specify a list of providers to load (#18261).

[netlify]

✅ Deploy Preview for vrl-playground canceled.

Name	Link
🔨 Latest commit	de0df1f
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/64dd074bae051000086b5978

[netlify]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	de0df1f
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/64dd074b1afbcf00089659c2

[jszwedko]

Great! I want to play around with this a bit locally but left one comment for now.

[jszwedko]

I realize this is roughly the same as the previous implementation in terms of error handling, but I'm wondering if failing to load a provider should be a hard error. What do you think?

[dsmith3197]

Good question, I went back and forth on this when writing the original PR.

These providers should never fail to load because they are shipped with openssl. In the case that they do fail to load, openssl would continue to function (just without the given provider).

I'm open to making it a hard error if you think that is preferable.

[hhromic]

From the tests in my other PR, sometimes a provider can fail.
For example the FIPS provider did not load because it requires a third party library in the system.
So, yes, loading providers can fail. The question is how tolerant Vector should be about it.

If any requested provider fails to load, the user would not get what he/she wanted and therefore maybe is a good idea to make it a hard fail indeed.

[datadog-vectordotdev]

Datadog Report

Branch report: dougsmith/fix-openssl-legacy-provider
Commit report: 1aca479

✅ vector: 0 Failed, 0 New Flaky, 1939 Passed, 0 Skipped, 1m 18.35s Wall Time

[jszwedko]

Played around with this locally and it looks good to me! I'm still thinking about whether failing to load providers should be a hard error or not 🤔

[jszwedko]

I think this can be merged regardless though and we can follow up to make provider loading a hard error if we decide to go that route.

[dsmith3197]

I think this can be merged regardless though and we can follow up to make provider loading a hard error if we decide to go that route.

I updated this to make it a hard error.

[hhromic]

Ha! Neat trick to keep using the same bool but accept true/false values 👍.
I humbly accept that I'm not really well versed in clap :)

[jszwedko]

🎉

[jszwedko]

I'd suggest we refactor this in #18261 to not be an Option and just always load the default provider if none are specified.

[hhromic]

Taking note of this comment for #18261 ✍️

[hhromic]

Quick question, this loads the providers in the same order as declared in the array, right?
Shouldn't the default provider be loaded first?

If the implementations in multiple providers have non-overlapping algorithms, order should not matter, but if they overlap, I guess the order matters for overriding.

This would be important to make it clear in the documentations of the other PR.

[hhromic]

From the openssl repository provider README about the FIPS provider:

In some cases, there may be minor behavioural differences between algorithm implementations in this provider compared to the equivalent algorithm in the default provider. This is typically in order to conform to FIPS standards.

This implies that the FIPS provider overrides algorithms in the default provider, therefore the order of loading them definitively should matter. Will take this into account for the other PR.

[dsmith3197]

The example in the documentation loads the legacy provider and then the default provider, so this should be okay. With respect to the FIPS provider, we would have to better understand how the ordering works..

[github-actions]

Regression Detector Results

Run ID: b8b37da8-1559-40f4-ad3f-4293b25046b4
Baseline: 4ec6c11
Comparison: fc17fba
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval.

We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:

1. The estimated |Δ mean %| ≥ 5.00%. This criterion intends to answer the question "Does the estimated change in mean optimization goal performance have a meaningful impact on your customers?". We assume that when |Δ mean %| < 5.00%, the impact on your customers is not meaningful. We also assume that a performance change in optimization goal is worth investigating whether it is an increase or decrease, so long as the magnitude of the change is sufficiently large.

2. Zero is not in the 90.00% confidence interval "Δ mean % CI" about "Δ mean %". This statement is equivalent to saying that there is at least a 90.00% chance that the mean difference in optimization goal is not zero. This criterion intends to answer the question, "Is there a statistically significant difference in mean optimization goal performance?". It also means there is no more than a 10.00% chance this criterion reports a statistically significant difference when the true difference in mean optimization goal is zero -- a "false positive". We assume you are willing to accept a 10.00% chance of inaccurately detecting a change in performance when no true difference exists.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.

experiment	goal	Δ mean %	Δ mean % CI	confidence
syslog_log2metric_humio_metrics	ingress throughput	+1.50	[+1.42, +1.59]	100.00%
syslog_regex_logs2metric_ddmetrics	ingress throughput	+0.94	[+0.68, +1.19]	100.00%
syslog_log2metric_splunk_hec_metrics	ingress throughput	+0.82	[+0.69, +0.94]	100.00%
splunk_hec_route_s3	ingress throughput	+0.37	[-0.05, +0.79]	73.77%
datadog_agent_remap_blackhole	ingress throughput	+0.23	[+0.15, +0.31]	99.99%
datadog_agent_remap_datadog_logs_acks	ingress throughput	+0.10	[-0.01, +0.21]	75.20%
splunk_hec_to_splunk_hec_logs_acks	ingress throughput	+0.01	[-0.10, +0.12]	6.91%
splunk_hec_indexer_ack_blackhole	ingress throughput	-0.00	[-0.13, +0.12]	2.39%
http_to_http_json	ingress throughput	-0.01	[-0.07, +0.06]	7.64%
splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.01	[-0.12, +0.10]	9.22%
syslog_loki	ingress throughput	-0.09	[-0.14, -0.03]	96.04%
syslog_humio_logs	ingress throughput	-0.18	[-0.28, -0.09]	98.78%
fluent_elasticsearch	ingress throughput	-0.22	[-0.47, +0.02]	75.36%
http_to_s3	ingress throughput	-0.26	[-0.47, -0.06]	90.31%
syslog_splunk_hec_logs	ingress throughput	-0.99	[-1.04, -0.94]	100.00%
file_to_blackhole	egress throughput	-1.00	[-1.98, -0.01]	80.40%
http_text_to_http_json	ingress throughput	-1.02	[-1.12, -0.91]	100.00%
otlp_grpc_to_blackhole	ingress throughput	-1.34	[-1.42, -1.27]	100.00%
datadog_agent_remap_datadog_logs	ingress throughput	-1.45	[-1.58, -1.31]	100.00%
http_to_http_acks	ingress throughput	-1.54	[-2.86, -0.23]	86.66%
socket_to_socket_blackhole	ingress throughput	-2.07	[-2.13, -2.01]	100.00%
datadog_agent_remap_blackhole_acks	ingress throughput	-2.12	[-2.20, -2.03]	100.00%
http_to_http_noack	ingress throughput	-2.37	[-2.47, -2.27]	100.00%
otlp_http_to_blackhole	ingress throughput	-2.44	[-2.61, -2.27]	100.00%
enterprise_http_to_http	ingress throughput	-2.71	[-2.81, -2.62]	100.00%