Support SSL_CERT_FILE/SSL_CERT_DIR

[pimterry]

Moved from rustls/rustls#540:

OpenSSL supports SSL_CERT_FILE and SSL_CERT_DIR to allow configuration of CA certificates in an application without making code changes. It would be very helpful if rustls (via rustls-native-certs) also supported this by default.

This is a pretty widely used feature (a quick search shows them cumulatively appearing about 200,000 times in scripts, sources & docs on github alone). It's especially useful when you want to reconfigure an application that's not your own, where needing to change the code itself or reconfiguring trust for the entire OS would be very inconvenient, insecure, or impractical (e.g. if you don't have admin privileges but want to reconfigure your own processes). By using these env vars, you can automatically reconfigure applications so they work happily in enterprise environments with internal cert infrastructure, with HTTPS debugging tools, or when using self-signed certs in development.

For a general purpose reconfiguration like this, it's quite inconvenient to require every end application author to check for and use env vars themselves (and in practice, when they do, they each use a different env var - e.g. Cargo supports this manually via its own CARGO_HTTP_CAINFO - where a single global env var would be much better).

I can imagine this might be an OpenSSL feature that's considered risky, since it takes TLS configuration from the environment, but if an attacker can inject arbitrary environment variables they can already do things like setting LD_PRELOAD or even just PATH to exert almost complete control over the application being run regardless. Notably almost all other platforms allow SSL configuration in env vars in some form, e.g. the JVM via options with JAVA_TOOL_OPTIONS, node via NODE_EXTRA_CA_CERTS, anything using OpenSSL via SSL_CERT_FILE, etc etc.

Providing the same here would effectively add this widely used & supported capability to the rust ecosystem too, matching other platforms and working out of the box with existing OpenSSL configurations.

[ctz]

AFAIK this should already work -- this crate calls https://github.com/alexcrichton/openssl-probe/blob/master/src/lib.rs#L88 which appears to be preferentially paying attention to these environment variables.

However, note #9 and also note that this doesn't happen in Windows/Mac -- these have their own root cert configuration surfaces.

[pimterry]

Ah ok, interesting! That's really useful, thanks, I hadn't realised that. Would it be possible to expand this to windows & mac too though, so it's supported consistently everywhere? I think it's equally useful on all platforms and having a single universal env var to set the cert file just like OpenSSL would be great.

I think specifying a single file with SSL_CERT_FILE is by far the most common use case, both by GH references and anecdotally in my own usage. It's also much simpler - no hardcoded paths or heuristics or #9 required, so just supporting that globally rather than both would be a reasonable option imo if you don't want to support the more complicated behaviours of SSL_CERT_DIR.

[pimterry]

@ctz any thoughts on the above? I.e. expanding SSL_CERT_FILE support to Windows & Mac too.

I'm happy to put a PR together for the implementation if you're open to it.

[djc]

On Mac and Windows, the current search routines do not look in any particular directories. I feel like starting to do so maybe doesn't make sense in the scope of rustls-native-certs, which is explicitly about using the OS trust stores. The fact that non-darwin Unices just use files/directories for this is kind of an implementation detail.

If you want to load your trust roots from a particular location in the file system, rustls should already provide decent APIs to do so (though perhaps we can make that scenario easier there).

[pimterry]

If you want to load your trust roots from a particular location in the file system, rustls should already provide decent APIs to do so

APIs in rustls aren't sufficient - the use case for this (like all the other OS configuration) is to reconfigure the certificates used from outside an application, without changing the code.

This is useful because there's many cases where the person executing the application knows which certificates to trust, and the author does not (most enterprise environments, network traffic debugging, self-signed development services, etc).

Solving this with OS store configuration usually requires root/admin permissions, is persistent, affects the system globally, and is more difficult to automate (e.g. in automated test environments). Executing an application with an environment variable set is temporary, isolated, easy to do, and does not require permissions beyond controlling how the application is launched.

On Mac and Windows, the current search routines do not look in any particular directories. I feel like starting to do so maybe doesn't make sense in the scope of rustls-native-certs, which is explicitly about using the OS trust stores. The fact that non-darwin Unices just use files/directories for this is kind of an implementation detail.

Using these variables to find certs on disk isn't a new convention for Windows or Mac though - AFAICT it's the de facto cross-platform standard for certificate configuration, often used today on those platforms too:

• It's widely documented (1, 2, 3, 4, 5) as the main way to externally configure certificate trust for Ruby on Windows & Mac
• Python's PEP on certificate trust formally defines these env vars (in addition to OS stores) as the standard mechanism for externally configuring Python certificates on all platforms.
• Node.js supports this on all platforms, when OpenSSL is enabled, which is documented as the official way to support external configuration of the trust store instead of using the fixed set of certs from Mozilla.
• Curl supports this on all platforms.
• OpenSSL supports this on all platforms, so its supported everywhere by anything built on OpenSSL that doesn't override cert configuration, which covers a lot of existing software.
• It's commonly used in scripting for all platforms. GitHub has many public examples of appveyor CI configurations (a CI platform used almost exclusively for Windows support), batch scripts and powershell scripts that set this explicitly on Windows, and lots of scripts that include both ssl_cert_file and "brew install" (because they're setting this in scripts on mac).
• There's 200,000 other code results on GitHub in total for these two env vars, plus a lot of related issues etc - you can skim through those and quickly see a lot of other usage of community usage of these env vars on non-Linux platforms.

Imo, users of this library want to easily support the most common external mechanisms for configuring certificate trust on each platform - not necessarily just to support the OS trust stores specifically. SSL_CERT_FILE is very commonly used for configuring cert trust today on all platforms, and so it would be useful to support that everywhere.

To be clear, I'm just suggesting reading from SSL_CERT_FILE if set, I'm not suggesting that Windows & Mac follow all the Linux openssl-probe logic or similar. I agree it's not reasonable to look up best-guess paths like we do on Linux, but reading a cert explicitly specified by the SSL_CERT_FILE location only if that variable is set is very easy to do I think, would be consistent with the rest of the Windows & Mac ecosystem, and would be a really useful feature for many users.

[djc]

I'm saying that I think it would be useful and good to support SSL_CERT_FILE/SSL_CERT_DIR; I just think that arguably this might not within the defined scope for rustls-native-certs, in that this crate is intended to provide access specifically to the OS trust store. As soon as the user is using these environment variables to specify a particular location to override the OS trust store, I'm suggesting that maybe there should be a convenience API in rustls or that this might be supported by yet another crate.

[pimterry]

this crate is intended to provide access specifically to the OS trust store

What's the definition of "OS trust store" that means that using these variables on Linux is sensible, but using them on Windows & Mac is not?

I think there's a good argument that "the OS trust store" is "the set of commonly used mechanisms for managing trust on each OS". I think SSL_CERT_FILE fits into the definition on all platforms.

---

Theory aside, a more practical argument: this crate seems to be marketed & used elsewhere as the official one-stop shop for automatically detecting the certificates that should be trusted on a system when using Rustls. That's a clearly valuable use case! SSL_CERT_FILE is also a useful feature in that use case though, and I'm not aware of any situations where you would ever actively not want to support SSL_CERT_FILE if you're trying to detect the appropriate cert configuration for an environment. That means that supporting this feature is strictly better for all use cases (I think? Concrete counter examples would be very interesting).

Given that, if this crate doesn't support this, and that functionality moves into a tiny separate crate instead, then we probably need a 3rd crate that includes this crate plus the SSL_CERT_FILE crate together, and it becomes sensible to recommend that everybody uses that 3rd crate as their one-stop shop for detecting certificate configuration, and to recommend that nobody uses this crate directly.

That situation is definitely a solution, I guess we could aim towards wrapping this crate elsewhere to add that functionality and recommending against using it directly, sure. Splitting into multiple tiny crates that you'd want to use together 100% of the time doesn't seem very efficient for anybody though, especially when the whole implementation of the SSL_CERT_FILE detection is present here already, but just disabled for 2 of 3 platforms.

Would that model work better for you? It seems like more hassle all round to me, but I'd be interested to know what the benefits are from your POV, if that is what you'd prefer.

[jsha]

I support the idea of an environment-based trust override for rustls. One important benefit: If it's hard to set a custom root bundle, many people will turn to publicly-trusted WebPKI certificates for their service-to-service authentication. It's much better for their own security and the health of the WebPKI if they use a local CA for those purposes.

There are a few possible crates where this override could go:

• rustls-native-certs
• rustls
• downstream libraries like ureq, hyper, etc. There's precedent for this in the Python requests library: REQUESTS_CA_BUNDLE.

@pimterry's goal, which is a good one, is that there should be a consistent environment variable across all rustls-using programs. For that purpose, putting the override in rustls itself would be better than rustls-native-certs. Many rustls-using programs use webpki-roots. For webpki-roots programs it's even more important to have an environment-based override, because you can't update the roots without recompiling, which is often not an option in emergency situations.

[jsha]

One other question to figure out: should we use the same environment variables as OpenSSL or different ones? I think probably different ones, so it's clear when looking at a script that it is configuring rustls' trust rather than OpenSSL's.

[Ralith]

so it's clear when looking at a script that it is configuring rustls' trust rather than OpenSSL's.

Why is this desirable? Seems like compatibility with existing scripts would be much more valuable.

[djc]

I support the idea of an environment-based trust override for rustls.

To be clear, I also support this.

My doubt here is with the notion of a program's user overriding a program's (or library's) author. Is it always unequivocally a good thing if a program's user can override the trust roots intentionally selected by the program's author? For example in rustls we have dangerous_configuration to make sure users are explicit in enabling more options for themselves in potentially unsafe ways. While I guess SSL_CERT_FILE/SSL_CERT_DIR will be used most often to restrict the set of accepted roots rather than widen it, that is not necessarily the case.

Additionally, rustls-native-certs advertises itself to its direct users (that is, program authors selecting a library for initializing a trust base) as giving access to "the platform's native certificate store". While the usage of these environment variables is already supported on non-Darwin Unix because the platform cert store is kind of a mess there, it's not entirely obvious to me that users of this crate are automatically sold on implicitly opting into allowing users to override that core idea.

Maybe this is as easy to resolve as creating an additional function load_native_certs_or_user_overrides() or certs_from_env() in this crate's root to make this explicitly opt-in? Would still likely also require some changes to the README.

[pimterry]

One other question to figure out: should we use the same environment variables as OpenSSL or different ones? I think probably different ones, so it's clear when looking at a script that it is configuring rustls' trust rather than OpenSSL's.

I agree with @Ralith - I would expect that compatibility is more valuable. There could be an argument for a separate RUSTLS_CERT_FILE env var that takes precedence, if that is important, but I'm not sure it's necessary. If you need to target cert config to specific OpenSSL/Rustls processes then you can just set it SSL_CERT_FILE only for some processes and unset it for others.

it's not entirely obvious to me that users of this crate are automatically sold on implicitly opting into allowing users to override that core idea

I'm not 100% sure what 'core idea' refers to here, but if you mean "allowing users to override the certificates in the application for themselves", then I think users of this crate are already opting into that.

For any app using the crate, most users executing the app can already freely override the certificates that are trusted. On Linux they can use SSL_CERT_FILE today, and on Windows & Mac the current user's own trust store is used (not the system root store) which AFAIK is generally user-modifiable.

The difference with using an env var is that it's much more convenient for various use cases, and it's isolated and temporary - it's not that it really changes the control the end user has over the certificates used.

Is it always unequivocally a good thing if a program's user can override the trust roots intentionally selected by the program's author?

I'm sure there will be some application authors (a very small minority, I think) who do need to specifically configure fixed TLS settings, and who actively don't want SSL_CERT_FILE or other system configuration to override those. I can imagine cases where this helps ensure things work correctly despite OS configuration, e.g. building internal tools that always trust the relevant self-signed CAs, where users who accidentally overriding that settings with some SSL_CERT_FILE would break the application, or enforcing a standard set of CAs to make customer support easier.

That is a reasonable use case, and we definitely shouldn't unexpectedly enable SSL_CERT_FILE for those authors. Those authors should never be using rustls-native-certs though - if they want direct control over certificate trust, they can't simultaneously delegate that to the OS.

It's important to note that either way this is an end user UX issue, not a security issue. It's always been possible for an end user running any application using Rustls or anything else to change the certificates that are trusted if they're sufficiently determined (using LD_PRELOAD, Frida, modifying the binary directly, or whatever else). The only difference is that this makes it much easier. There's no world in which we can or should try to enforce program behaviour in the face of the person who's running the application and really really wants to change what it's doing.

Anyway, given all that, imo there's a good case for:

• Not enabling anything by default in Rustls, to ensure this isn't accidentally enabled by authors who do want to configure a fixed set of certificates.
• Enabling this by default in Rustls-Native-Certs (or in an OS+env var crate that will replace this crate, if that's preferable)
• Recommending in Rustls that people should use Rustls-Native-Certs/the replacement crate by default unless they have a specific reason not to.

I think that's better than making this an opt-in feature in Rustls, because I think the set of use cases where you should use Rustls-Native-Certs and SSL_CERT_FILE is extremely similar if not identical, and enabling the two in separate ways just increases the odds people incorrectly use one without the other.

This doesn't fix overriding certs with webpki, but I'm not sure that's possible without a big breaking change that causes issues for the users who really do want to pin the webpki certs.

[djc]

Okay, I think I'm convinced that implicit support for SSL_CERT_FILE/SSL_CERT_DIR is useful. @pimterry would you be able to/interested in submitting a PR?

[pimterry]

Sure! PR opened: #32