ssl - tls verify on Windows fails

[christian-korneck]

BPO	36011
Nosy	@pfmoore, @tiran, @tjguk, @zware, @zooba, @tianon, @christian-korneck, @teeks99

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2019-02-16.18:36:12.057>
labels = ['expert-SSL', '3.7', '3.8', 'OS-windows']
title = 'ssl - tls verify on Windows fails'
updated_at = <Date 2021-04-19.20:04:50.465>
user = 'https://github.com/christian-korneck'

bugs.python.org fields:

activity = <Date 2021-04-19.20:04:50.465>
actor = 'tianon'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Windows', 'SSL']
creation = <Date 2019-02-16.18:36:12.057>
creator = 'chris-k'
dependencies = []
files = []
hgrepos = []
issue_num = 36011
keywords = []
message_count = 4.0
messages = ['335708', '335850', '335984', '381059']
nosy_count = 8.0
nosy_names = ['paul.moore', 'christian.heimes', 'tim.golden', 'zach.ware', 'steve.dower', 'tianon', 'chris-k', 'teeks99']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue36011'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

Linked PRs

• gh-80192: Use windows api if ssl cert verification fails #127622

[christian-korneck]

Hello,

I have the impression that there's a general issue with how the Python stdlib module ssl uses the Windows certificate store to read the "bundle" of trusted Root CA certificates. At a first look, I couldn't find this issue documented elsewhere, so I'm trying to describe it below (apologies if its a duplicate).

This issue leads to that on a standard Windows 10 installation with a standard Python 2.x or 3.x installation TLS verification for many webservers fails out of the box, including for common domains/webservers with a highly correct TLS setup like https://google.de or https://www.verisign.com/ .

In short: On a vanilla Win 10 with a vanilla Python 2/3 installation, HTTPS connections to "commonly trusted" domain names fail out of the box. Example with Python 2.7.15:

>>> import urllib2
>>> response = urllib2.urlopen("https://google.de")
[...]
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)

Expected Behavior: TLS verify succeeds
Actual Behavior: TLS verify fails

Affected Python version/environment: I believe every Python version that uses the Windows certificate store is affected (since 3.4 / 2.7.9). However, I've only tested 2.7.11, 2.7.15, 3.7.2 (all 64 bit). I did test on Windows 10 1803, 1809, Windows Server 2019 1809 (all Pro x64 with latest patchlevel, i.e. the Jan 2019 cumulative update). All tested Python versions on all tested Windows 10 versions show the same behavior.

--------

Details:

1.) Background

• Factor1: Python's "ssl" std lib
  Since Python 3.4 / 2.7.9 the ssl lib uses the Windows certificate store to get a "bundle" of the trusted root CA certificates. (Some Python libraries like requests bring their own ca bundle though, usually through certifi. These libs are not affected). However, the ssl lib is not using the Windows SCHANNEL library but instead bundles its own copy of openssl.

• Factor2: Windows 10 behavior
  Windows provides a certificate store, a vendor managed and updated "bundle" of Trusted Root CA certificates and a library for TLS operations called SCHANNEL (the native Windows openssl equivalent).

On Windows 10, the list of pre-installed Trusted Root CA certificates is very minimal. On Windows 10 1809 only 12 Root CAs are known by the certificate store. In comparison certifi (Mozilla cabundle) currently lists 134 trusted RootCAs. Many widely trusted RootCAs are missing out of the box in the Windows certstore. Instead there's an online download mechanism used by the SCHANNEL library to download additional trusted root CA certificates from a Microsoft server when they are needed for the first time.

Example: The certificate currently used for https://google.de was signed by an IntermediateCA which was signed by the RootCA "GlobalSign Root CA - R2". The cert for this RootCA is not out of the box present in the Windows certstore and therefore not trusted. When I make a HTTPS connection to this domain with a client that uses the SCHANNEL library (i.e. Microsoft Edge or Internet Explorer browser), the connection succeeds and is shown as "trusted". Afterwards the previously missing RootCA certificate appears in the windows certstore. (The Windows certstores can get inspected with the GUIs certml.msc (Machine store) and certmgr.msc (User store)).

2.) Behavior

• install a vanilla Windows 10 1809 with default settings
• install a vanilla Python 2.7.15 and/or 3.7.2

In Python:

c:\python27\python.exe
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket, ssl
>>> context = ssl.SSLContext(ssl.PROTOCOL_TLS)
>>> context.verify_mode = ssl.CERT_REQUIRED
>>> context.check_hostname = True
# by default there are no cacerts in the context
>>> len(context.get_ca_certs())
0
>>> context.load_default_certs()
>>> len(context.get_ca_certs())
# after loading the cacerts from the Windows cert store "ROOT", we are seeing some - but it's only 12 root cacerts in a vanilla Windows 10 (compared to 134 in the certifi / mozilla cabundle!)
12
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> ssl_sock = context.wrap_socket(s, server_hostname='www.google.de')
>>> ssl_sock.connect(('www.google.de', 443))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\python27\lib\ssl.py", line 882, in connect
    self._real_connect(addr, False)
  File "c:\python27\lib\ssl.py", line 873, in _real_connect
    self.do_handshake()
  File "c:\python27\lib\ssl.py", line 846, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
>>> ssl_sock.close()

This first attempt to make a HTTPS connection to https://google.de failed because the required RootCA for this domain is not part of the very minimal Windows out-of-the-box ca bundle.

Now let's make a https request against this domain with an application that uses the Windows SCHANNEL library (for example by typing https://google.de/ into the address bar in Internet Explorer / Edge browser). I will use the experimental pySchannelSSL here:

$ git clone https://github.com/lsowen/pySchannelSSL.git
$ "c:\Program Files\Python37\python.exe"
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
>>> import pySchannelSSL.httpshandler
>>> h = pySchannelSSL.httpshandler.SSLConnection("google.de", port=443)
>>> h.connect()
>>> h.close()

As part of processing the above request, the Windows SCHANNEL library has magically fetched the missing trusted RootCA certificate from a Microsoft server and has stored it permanently in the Windows "Trusted Root CAs" certstore.

We can verify this with:

>>> import socket, ssl
>>> context = ssl.SSLContext(ssl.PROTOCOL_TLS)
>>> context.load_default_certs()
>>> len(context.get_ca_certs())
15

Note that in our first attempt there were only 12 root cacerts in the Windows certstore. Now it's 15. And the only difference is that in between we've made an SCHANNEL-based https connection to the google.de domain. (You can also see the additional root certificates via the certificates mmc consoles certlm.msc and certmgr.msc).

From now on all non-SCHANNEL based HTTPS connections via the Python 2/3 ssl standard lib work, as SCHANNEL has permanently placed the RootCA cert in the windows certstore:

c:\python27\python.exe
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket, ssl
>>> context = ssl.SSLContext(ssl.PROTOCOL_TLS)
>>> context.verify_mode = ssl.CERT_REQUIRED
>>> context.check_hostname = True
>>> context.load_default_certs()
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> ssl_sock = context.wrap_socket(s, server_hostname='www.google.de')
# got a certificate verify failed error here in the first try, this time the verify is successfull
>>> ssl_sock.connect(('www.google.de', 443))
>>> ssl_sock.close()

3.) Conclusion
I believe the way how the Python "ssl" stdlib uses the Windows Certificate Store is not ideal. Windows seems to expects all TLS connections to be made through the SCHANNEL library. The Trusted Root CA store in Windows 10 only seems to function as some sort of cache for SCHANNEL but is not as a complete source of truth. Maybe letting the "ssl" stdlib make a minimal SCHANNEL call before handing over to openssl could provide a minimal invasive fix?

(Side note: I would still advocate for not bypassing the Windows certstore, as having a certstore per application is a security issue and big pain for deploying/updating own "Intranet" RootCA certificates).

--------

Best,
Chris

[christian-korneck]

quick addition: It looks like all recent Windows versions (Win8/Server 2012, Win8.1/Server 2012R2, Win10 (older versions)/Server 2016, Win10-1809/Server 2019 behave the same (= only very few RootCAs are pre-installed out of the box, additional ones are added on the fly when HTTPS requests are being made via the SCHANNEL api).

Possible workaround for Windows admins:

Import the RootCA certs from "certifi" into the Windows local machine Trusted RootCA store.

To do so, first download and convert the certifi cabundle (https://certifi.io) to a pfx container, i.e. with something like:

wget -O certs.pem https://mkcert.org/generate/
openssl pkcs12 -export -nokeys -out certs.pfx -in certs.pem

Then import the pfx via the certlm.msc GUI or the certutil.exe cmdline tool. This imports all certs at once. This can also be centralized for a larger number of machines via an Active Directory Group Policy (Local Machine -> Windows Settings -> Security Settings -> PKI).

This isn't ideal as it puts the admin into the responsibility to update the certstore/GPO whenever there's a change in the certifi cabundle, but works well for me besides that.

[zooba]

Thanks.

This is a well-known and long-standing issue between OpenSSL and Windows, and the best workaround right now is to use the Mozilla certs directly.

One day when OpenSSL is no longer part of the CPython public API, then we can consider switching to an HTTP implementation that uses the operating system support (which in my experimentation is 2-3x faster than using OpenSSL anyway, but a *big* breaking change for a lot of code). Until then, use the options provided by OpenSSL to enable it to verify what you need.

[teeks99]

Christian's message indicated that a workaround was possible by adding mozilla's certs to windows cert store.

I'm sure there are sysadmins who will really hate this idea, but I've successfully implemented it in a windows docker image, and wanted to document here.

Powershell commands, requires OpenSSL to be installed on the system:

cd $env:USERPROFILE;
Invoke-WebRequest https://curl.haxx.se/ca/cacert.pem -OutFile $env:USERPROFILE\cacert.pem;
$plaintext_pw = 'PASSWORD';
$secure_pw = ConvertTo-SecureString $plaintext_pw -AsPlainText -Force;
& 'C:\Program Files\OpenSSL-Win64\bin\openssl.exe' pkcs12 -export -nokeys -out certs.pfx -in cacert.pem -passout pass:$plaintext_pw;
Import-PfxCertificate -Password $secure_pw  -CertStoreLocation Cert:\LocalMachine\Root -FilePath certs.pfx;

Once mozilla's store is imported into the microsoft trusted root store, python has everything it needs to access files directly.

[TTimo]

(not sure why this is tagged 3.7 / 3.8 - this is a problem in 3.10 as well)

[zooba]

The tags were current at the time it was filed, and they only get updated when someone manually updates it. I think we also may have changed out version tagging approach to make this less of an issue, but issues migrated from bpo just kept their data.

[segevfiner]

I think the underlying function used by SCHANNEL to fetch the cert chain is CertGetCertificateChain. But I'm not sure how one would go about integrating that with OpenSSL.

[tobil4sk]

I think the underlying function used by SCHANNEL to fetch the cert chain is CertGetCertificateChain. But I'm not sure how one would go about integrating that with OpenSSL.

We recently applied patch like this for Haxe. I've implemented a similar patch for CPython using OpenSSL's SSL_CTX_set_verify, see: #127622.