[BUG] OpenSearch 2.3.0 Docker staging image crashing with error from Security Plugin

[qreshi]

What is the bug?
The following exception can be seen when OpenSearch is starting up using the 2.3.0 staging docker image:

[2022-09-06T17:11:33,008][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [94c20e8f2193] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.3.0.jar:2.3.0]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.3.0.jar:2.3.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:419) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:346) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:419) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:346) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
	... 6 more
Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/openssl/PEMParser
	at io.netty.handler.ssl.BouncyCastlePemReader.newParser(BouncyCastlePemReader.java:203) ~[?:?]
	at io.netty.handler.ssl.BouncyCastlePemReader.getPrivateKey(BouncyCastlePemReader.java:134) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1126) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:898) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:402) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:176) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:262) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:419) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:346) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
	... 6 more
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMParser
	at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:587) ~[?:?]
	at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]
	at io.netty.handler.ssl.BouncyCastlePemReader.newParser(BouncyCastlePemReader.java:203) ~[?:?]
	at io.netty.handler.ssl.BouncyCastlePemReader.getPrivateKey(BouncyCastlePemReader.java:134) ~[?:?]
	at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1126) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:384) ~[?:?]
	at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:120) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.buildSSLServerContext(DefaultSecurityKeyStore.java:898) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:402) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:176) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:262) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:419) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.node.Node.<init>(Node.java:346) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0]
	... 6 more
uncaught exception in thread [main]

What is the expected behavior?
No crashing when OpenSearch starts up.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
This issue wasn't happening earlier last week and at the time of this issue being created only the PA and Security plugins are added to the 2.3.0 manifest.

[cwperks]

@qreshi I had a change merged into netty which was included in the 4.1.80.Final release. See netty/netty#12670.

There were 2 bugs associated with this PR that were identified and fixes have been merged and targeted for 4.1.81.Final (unreleased) version of netty.

• Fix regression in SslContext private key loading netty/netty#12743
• Fix bug in SslContext private key reading fall-back path netty/netty#12746

We can revert core back to netty:4.1.79.Final until a bugfix version is released or include the necessary dependencies in the security plugin for the BouncyCastlePemReader.

[peternied]

[Security Team Triage] We should revert back to the last functional version of netty, netty:4.1.79.Final and then try again in a future release when these issues are no longer present.