OpenSSL 3.0 support

[gcsideal]

Botan 2.18.1 (latest release ATM) fails to test with OpenSSL 3.0:
Thread_Pool ran 100 tests all ok
tls:
3DES ECDH ran 2 tests 2 FAILED
Failure 1: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
Failure 2: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
3DES RSA ran 2 tests 2 FAILED
Failure 1: EVP_EncryptInit_ex failed: error:03000086:digital envelope routines::initialization error
Failure 2: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
AES-128 DH ran 1 tests 1 FAILED
Failure 1: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
AES-128 DHE_PSK ran 1 tests 1 FAILED
Failure 1: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
AES-128 ECDH ran 2 tests 2 FAILED
Failure 1: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
Failure 2: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
AES-128 ECDHE_PSK ran 1 tests 1 FAILED
Failure 1: EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported
[...]

For your information, there's a migration guide: https://www.openssl.org/docs/man3.0/man7/migration_guide.html

[randombit]

Really unclear what's happening here since the tests that are failing are AES-128 and 3DES ciphersuites, which were not moved to the legacy provider.

[reneme]

I can reproduce this on macOS with the OpenSSL 3.0 build provided by brew. AES tests work fine, as well as TripleDES. The DES tests fail with

'EVP_EncryptInit_ex failed: error:0308010C:digital envelope routines::unsupported'

I configured the botan build like:

./configure.py --minimized-build --with-openssl --without-documentation --no-optimization --extra-cxxflags="-O1" --enable-modules="aes,aes_ni,aes_vperm,des" --with-external-includedir=/usr/local/opt/openssl@3/include --with-external-libdir=/usr/local/opt/openssl@3/lib  --compiler-cache=$(which ccache) --build-targets="static,tests"

(--no-optimization --extra-cxxflags="-O1" is because of #2802)

Even without a special module configuration tests related to Blowfish, DES, CTR-BE(DES), MD4, RIPEMD-160, CBC-MAC(DES), Whirlpool, CMAC(Blowfish), X9.19-MAC, Blowfish/EAX, DES/EAX, passhash9, and more DES, Blowfish related. Also the mentioned "3DES ECDH" tests fail.

Full log: openssl3_testoutput.txt

[randombit]

Right, certainly we should remove support (within the OpenSSL provider) for DES, Blowfish, MD4, and everything else OpenSSL has moved to the legacy provider in 3.0.

I'm just not seeing why the TLS tests fail, since these tests should not be using any of those algorithms.

[randombit]

Alternately take this opportunity to kill the OpenSSL provider entirely since it seems to cause a lot of problems and at this point besides P-256 I think Botan is within +/- 10% of OpenSSL for most operations.

[randombit]

Issue has been fixed in master and also in forthcoming 2.18.2 release - closing. Thank you for reporting this.