Make Python compatible with OpenSSL 3.0.0

[tiran]

BPO	38820
Nosy	@tiran, @mcepl, @ambv, @stratakis, @hroncok, @miss-islington, @iritkatriel
PRs	bpo-38820: OpenSSL 3.0.0 compatibility. #17190 [3.8] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) #17499 [3.7] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) #17500 bpo-38820: Run CI with OpenSSL 3.0.0-alpha14 #25316 bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319) #25319 [3.9] bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319) #25320 [3.8] bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319) #25321 bpo-38820: Test with OpenSSL 3.0.0-alpha15 (GH-25537) #25537 bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) #25624 [3.9] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) #25626 [3.8] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) #25627 bpo-38820: Add ssl, hashlib, and hmac changes to whatsnew 3.10 (GH-25817) #25817 bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) #25942 [3.10] bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) #25943 [3.9] bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) #25944 bpo-38820: Test with OpenSSL 3.0.0-alpha17 #26266 [3.10] bpo-38820: Test with OpenSSL 3.0.0-alpha17 (GH-26266) #26269 bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769) #26769 [3.10] bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769) #26799 bpo-38820: Test with OpenSSL 3.0.0 final #28205 [3.10] bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205) #28216 [3.9] bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205) #28217
Dependencies	bpo-43788: OpenSSL 3.0.0: Make ssl_data.h version specific bpo-43789: OpenSSL 3.0.0: password callback called multiple times bpo-43791: OpenSSL 3.0.0: TLS 1.0 / 1.1 connections fail with TLSV1_ALERT_INTERNAL_ERROR bpo-43794: OpenSSL 3.0.0: Handle UNEXPECTED_EOF_WHILE_READING / wrap SSL_OP_IGNORE_UNEXPECTED_EOF bpo-43799: OpenSSL 3.0.0: define OPENSSL_API_COMPAT 1.1.1 bpo-43811: Run GHA CI with multiple OpenSSL versions bpo-43920: OpenSSL 3.0.0: handle empty cadata consistently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2019-11-16.15:06:59.538>
labels = ['expert-SSL', 'type-feature', '3.9', '3.10', '3.11']
title = 'Make Python compatible with OpenSSL 3.0.0'
updated_at = <Date 2021-09-08.17:05:14.504>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-09-08.17:05:14.504>
actor = 'lukasz.langa'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['SSL']
creation = <Date 2019-11-16.15:06:59.538>
creator = 'christian.heimes'
dependencies = ['43788', '43789', '43791', '43794', '43799', '43811', '43920']
files = []
hgrepos = []
issue_num = 38820
keywords = ['patch']
message_count = 27.0
messages = ['356750', '356759', '357979', '357980', '357981', '378790', '378807', '390485', '390650', '390662', '390664', '390666', '391688', '391887', '391892', '391940', '392701', '393098', '394031', '396120', '396123', '397326', '401311', '401355', '401407', '401408', '401409']
nosy_count = 8.0
nosy_names = ['christian.heimes', 'mcepl', 'lukasz.langa', 'cstratak', 'hroncok', 'miss-islington', 'iritkatriel', 'bweeks']
pr_nums = ['17190', '17499', '17500', '25316', '25319', '25320', '25321', '25537', '25624', '25626', '25627', '25817', '25942', '25943', '25944', '26266', '26269', '26769', '26799', '28205', '28216', '28217']
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38820'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[tiran]

OpenSSL 3.0.0 is currently development [1]. I'm expecting a first beta release in December. Final release is scheduled for Q2 2020. OpenSSL 3.0.0 is API and feature compatible to OpenSSL 1.1.0 and 1.1.1. Only minor changes are required:

• OpenSSL version number is >= 3.0.0, which breaks test_openssl_version
• GENERAL_NAME_print() no longer adds trailing newline to IPv6 address strings.
• ERR_func_error_string is deprecated

[1] https://www.openssl.org/blog/blog/2019/11/07/3.0-update/

[tiran]

PR #61392 fixes test_openssl_version and removes the trailing newline from IPv6 addresses on all OpenSSL versions. I prefer to have the output consistent on all OpenSSL versions. The newline was silly any way.

[miss-islington]

New changeset 2b7de66 by Miss Islington (bot) (Christian Heimes) in branch 'master':
bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190)
2b7de66

[miss-islington]

New changeset 9d3cacd by Miss Islington (bot) in branch '3.8':
[3.8] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) (GH-17499)
9d3cacd

[miss-islington]

New changeset a197f8a by Miss Islington (bot) in branch '3.7':
[3.7] bpo-38820: OpenSSL 3.0.0 compatibility. (GH-17190) (GH-17500)
a197f8a

[iritkatriel]

Can this be closed?

[tiran]

No, this is still work in progress.

[hroncok]

Python 3.10.0a7 with OpenSSL 3.0 from https://copr.fedorainfracloud.org/coprs/saprasad/openssl-3.0/ in https://copr.fedorainfracloud.org/coprs/g/python/openssl-3.0/package/python3.10/ (full logs available there).

3 tests failed:
test_imaplib test_ssl test_urllib2_localnet

Many:

ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2628)

Also:

Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.10.0a7/Lib/test/test_ssl.py", line 1413, in test_load_cert_chain
    ctx.load_cert_chain(CERTFILE_PROTECTED, password=getpass_huge)
SystemError: _PyEval_EvalFrameDefault returned a result with an exception set

And:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1122)

ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1122)

ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1122)

[tiran]

Miro,

I have pushed several fixes for OpenSSL 3.0.0

• bpo-43788 addresses wrong library and error reason codes (e.g. KRB5_S_TKT_NYV)
• bpo-43789 fixes an issue with exception state in password callbacks (_PyEval_EvalFrameDefault returned a result with an exception set)
• bpo-43791 disables TLS 1.0 and 1.1 testing with OpenSSL 3.0.0. I'll have to talk to upstream and figure out a better solution.
• bpo-43794 adds OP_IGNORE_UNEXPECTED_EOF and sets it by default. This makes the code behave like OpenSSL 1.1.0 and 1.0.2.

I'll look into the other issues next week.

[tiran]

New changeset 2d7fdc9 by Christian Heimes in branch 'master':
bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319)
2d7fdc9

[miss-islington]

New changeset ffb05bb by Miss Islington (bot) in branch '3.8':
bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319)
ffb05bb

[miss-islington]

New changeset 7c8796a by Miss Islington (bot) in branch '3.9':
bpo-38820: OpenSSL 3.0.0: Use supported hashing algos in doc test (GH-25319)
7c8796a

[tiran]

New changeset dcf6581 by Christian Heimes in branch 'master':
bpo-38820: Test with OpenSSL 3.0.0-alpha15 (GH-25537)
dcf6581

[tiran]

New changeset 3c586ca by Christian Heimes in branch 'master':
bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624)
3c586ca

[miss-islington]

New changeset 10ee266 by Miss Islington (bot) in branch '3.8':
[3.8] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) (GH-25627)
10ee266

[miss-islington]

New changeset 3b917d1 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-38820: Old OpenSSL 3.0.0 releases are in /old/3.0/ (GH-25624) (GH-25626)
3b917d1

[tiran]

New changeset d8389e3 by Christian Heimes in branch 'master':
bpo-38820: Add ssl, hashlib, and hmac changes to whatsnew 3.10 (GH-25817)
d8389e3

[miss-islington]

New changeset f8778f9 by Miss Islington (bot) in branch '3.10':
bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942)
f8778f9

[miss-islington]

New changeset 36843f7 by Miss Islington (bot) in branch '3.10':
bpo-38820: Test with OpenSSL 3.0.0-alpha17 (GH-26266)
36843f7

[tiran]

New changeset 44fb551 by Christian Heimes in branch 'main':
bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769)
44fb551

[miss-islington]

New changeset c6cd2ec by Miss Islington (bot) in branch '3.10':
[3.10] bpo-38820: Test with OpenSSL 3.0.0-beta1 (GH-26769) (GH-26799)
c6cd2ec

[ambv]

New changeset c92b391 by Christian Heimes in branch '3.9':
[3.9] bpo-38820: Test with OpenSSL 3.0.0-alpha16 (GH-25942) (bpo-25944)
c92b391

[ambv]

New changeset cc7c680 by Christian Heimes in branch 'main':
bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205)
cc7c680

[miss-islington]

New changeset 2fe15db by Miss Islington (bot) in branch '3.10':
bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205)
2fe15db

[ambv]

New changeset 7a6178a by Łukasz Langa in branch '3.9':
[3.9] bpo-38820: Test with OpenSSL 3.0.0 final (GH-28205) (GH-28217)
7a6178a

[ambv]

Christian, Python is now tested with 3.0.0 final in 3.9, 3.10, and 3.11. Looks like we can close this!

Thank you for this big body of work ✨ 🍰 ✨

[ambv]

(I'll let you close this yourself when you determine that the two remaining open dependencies can be closed as well.)

[Steve123p]

Is this going to be officially closed?

[kumaraditya303]

I am closing this as it is fixed, if required it can reopened.