Unable to create SSLContext and no proper reason code for failure.

[kushalagrawal]

Hi, we are running netty server on Alma 9.2 (FIPS version) with OpenSSL 3.0.7+TuxCare_FIPS 1 Nov 2022. while I am trying to create the server SSLContext in netty using SslContextBuilder contextBuilder = SslContextBuilder.forServer

javax.net.ssl.SSLException: unable to setup accepted issuers for trustmanager io.netty.handler.ssl.EnhancingX509ExtendedTrustManager@1fd77205
	at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:166) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:356) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:336) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:478) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:610) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]

Error is seen in the line: ReferenceCountedOpenSSLServerContext.java line#165 where it fails but the native call returns false value and does not provide a reasonable explanation to identify a possible way to rectify the issue.

if (!SSLContext.setCACertificateBio(ctx, bio)) {
                            throw new SSLException("unable to setup accepted issuers for trustmanager " + manager);
                        }

Is there any way to debug and find out the reason for failure?

[ramtech123]

Hi team,

Any thoughts on this issue?

[sej7278]

Also reproducible on RHEL 9.4 - also you only see the error in FIPS mode

[normanmaurer]

I wonder if you could log SSL.getLastError() and see if there is some meaningful errors message ?

[ramtech123]

We did some further research on this issue and below are the findings.

1. The error was on the native code was error:0A080106:SSL routines::passed invalid argument, it did not indicate what was the invalid parameter.
2. Our initial observation was that error was already present when invocation of SSLContext.setCACertificateBio(ctx, bio) happened inside ReferenceCountedOpenSslServerContext.java. We did not have any clue on what is causing the issue.
3. Eventually, after further troubleshooting, we could figure out that the issue happens when invoking SSLContext.setCurvesList(ctx, OpenSsl.NAMED_GROUPS) in ReferenceCountedOpenSslContext.java, with default settings.
4. This is because of the named curve x25519 in the DEFAULT_NAMED_GROUPS list. This curve is being rejected by OpenSSL in FIPS mode. (Looks like correct behavior by OpenSSL, as X25519 does not seem to be in the list of its FIPS approved algorithms as of today)
5. We were able to get the application working after overriding the named curves configuration with Java system property -Djdk.tls.namedGroups with value secp256r1,secp384r1,secp521r1.

Regards,
Rama

[normanmaurer]

@ramtech123 thanks for the info... I think we can improve stuff a bit here to make things easier in the future. Let me look into it soonish

[normanmaurer]

@ramtech123 sorry it took me longer then expected but this should fix it and help also to better debug things in the future netty/netty#14434

[normanmaurer]

And netty/netty#14435