Incompatible BouncyCastle FIPS changes in bookkeeper 4.14

[jiazhai]

Describe the bug
In bookkeeper # 2631, the default BouncyCastle was changed from non-fips into fips version. But the default version of BouncyCastle in Pulsar is the non-fips one(aimed to make it compatible with the old version of Pulsar).

Bouncy Castle provides both FIPS and non-FIPS version, but in a JVM, it can not include both of the 2 versions(non-Fips and Fips), and we have to exclude the current version before including the other. This make the backward compatible a little hard, and that's why Pulsar has to involve individual module for Bouncy Castle.

Pulsar excluded the dependencies of bookkeeper-server's BouncyCastle in Pulsar's pom file, and Pulsar only includes the non-fips version, but the bookkeeper-server still wants to use the hard-coded fips version in bookkeeper # 2631

And if we want to start BookKeeper with TLS enabled through Pulsar's binary, it will meet the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at org.apache.bookkeeper.common.util.ReflectionUtils.forName(ReflectionUtils.java:49)
	at org.apache.bookkeeper.tls.SecurityProviderFactoryFactory.getSecurityProviderFactory(SecurityProviderFactoryFactory.java:39)
	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:129)
	at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
	at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
	at org.apache.bookkeeper.server.Main.main(Main.java:208)
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 9 more

To Reproduce
Steps to reproduce the behavior:

1. using pulsar 2.8.0,
2. start bookkeeper through bin/pulsar bookie, with TLS enabled.

Expected behavior
By using bin/pulsar bookie, BookKeeper server should able to start success.
BK should not hard-coded fips version for Bouncy Castle, it should have an option to use non-fips version to make it compatible.

Additional context
We may need to provide the fix in the BookKeeper side first, and then change the dependency of bookkeeper version in Pulsar.

[jiazhai]

we should also add a test in pulsar, which enable tls for bookkeeper and use bc-non-fips, to cover this error.

14:06:44.561 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.NoClassDefFoundError: org/bouncycastle/jcajce/provider/BouncyCastleFipsProvider
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.bookkeeper.common.util.ReflectionUtils.forName(ReflectionUtils.java:49) ~[org.apache.bookkeeper-bookkeeper-common-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.tls.SecurityProviderFactoryFactory.getSecurityProviderFactory(SecurityProviderFactoryFactory.java:39) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.<init>(DefaultPerChannelBookieClientPool.java:67) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.BookieClientImpl.lookupClient(BookieClientImpl.java:211) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.proto.BookieClientImpl.addEntry(BookieClientImpl.java:318) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.sendWriteRequest(PendingAddOp.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.client.PendingAddOp.safeRun(PendingAddOp.java:278) ~[org.apache.bookkeeper-bookkeeper-server-4.14.1.jar:4.14.1]
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.1.jar:4.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
... 14 more
14:06:46.566 [pulsar-web-40-3]

[codelipenghui]

Reopen the issue since we need to add integration tests at the Pulsar side to avoid regression.

[eolivelli]

@codelipenghui can you please open a new ticket ?
this ticket is bound to 2.8.1 release
it is awkward to see this re-opened, as 2.8.1 has been released.

I believe it will be clearer from a project management and release management perspective to open a new issue

[codelipenghui]

The issue had no activity for 30 days, mark with Stale label.

[tisonkun]

@codelipenghui @hangc0276 is this ticket still relevant?

It's weird when I filter release/blocker and see this issue targets to 2.8.1 and stale for a long time.

At least we should correct the metadata as @eolivelli mentioned #10937 (comment)

[tisonkun]

Closed as completed. I don't think it's still relevant. For integration tests, a new issue is better.