KAFKA-17078: Add SecurityManagerCompatibility shim #16522

gharris1727 · 2024-07-03T21:37:58Z

This is a shim to allow the use of both modern (JRE 18+) and legacy APIs (deprecated in 17, degraded in 23, removal unknown) related to JEP 411: Deprecate the Security Manager for Removal.

This shim implements the interim strategy outlined in KIP-1006: Remove SecurityManager Support where legacy APIs are used as long as they are available and functional. Once the legacy APIs are removed or degraded, the modern APIs are used instead.

The shim is structured as a general interface with four 'strategy' implementations for testability. These implementations allow for mocking out the classloading infrastructure to simulate situations which no current JRE implements, namely removal and further degradation of functionality.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

Signed-off-by: Greg Harris <greg.harris@aiven.io>

…ty-manager-shim

C0urante

Thanks @gharris1727! The shim strategy here looks excellent overall.

I've gone through everything except the new unit test for the security manager compatibility class. Most comments are trivial or requests for more documentation; I do think I found one possible bug, though. Let me know when this is ready for another round and I can take a look at the tests.

clients/src/main/java/org/apache/kafka/common/internals/SecurityManagerCompatibility.java

C0urante · 2024-08-05T15:37:08Z

clients/src/main/java/org/apache/kafka/common/internals/SecurityManagerCompatibility.java

+
+    <T> T doPrivileged(PrivilegedAction<T> action);
+
+    Subject current();


Could we maybe call this currentSubject? The name is a little ambiguous and doesn't immediately point readers towards the JDK API that it's a shim for.

No, the JRE-18+ API is called Subject#current and I think we should have parity with that, so that in the future SecurityManagerCompatibility.get().current() can be replaced with Subject.current().

I dunno, it's a simple IDE find+replace to update this in the future, and Subject::current has meaning to it that SecurityManagerCompatibility.get().current() does not. I don't think it's worth the hit in readability for the (very long) interim period where we use this class instead of the JDK APIs that will eventually replace it. But I won't block on it.

Signed-off-by: Greg Harris <greg.harris@aiven.io>

...apache/kafka/common/security/oauthbearer/internals/OAuthBearerSaslClientCallbackHandler.java

Signed-off-by: Greg Harris <greg.harris@aiven.io>

C0urante

LGTM, thanks Greg!

showuon

LGTM!

showuon · 2024-08-27T07:18:51Z

clients/src/main/java/org/apache/kafka/common/internals/SecurityManagerCompatibility.java

+     * AccessControlContext will be ignored while the action is performed.
+     *
+     * @param <T> the type of the value returned by the PrivilegedAction's
+     *                  {@code run} method.


nit: indent

showuon · 2024-09-25T03:27:09Z

@gharris1727 , I think we can merge this change now. Let me know if you have objection.

danishnawab · 2024-09-26T15:22:40Z

Is there a plan on how/when this change will be integrated?
This issue currently prevents us from upgrading to JDK 23 (which is GA for more than a week now).

Using kafka-clients:3.7.1 with JDK 23 we get the following exception:

java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)

mumrah · 2024-09-26T15:28:44Z

@gharris1727 can you merge in trunk to run this with the new CI?

…ty-manager-shim

gharris1727 · 2024-09-26T15:47:47Z

Hey @danishnawab thanks for reminding us.

Since this may only be included in future releases (3.7.2, 3.8.1, 3.9.1, 4.0.0), existing releases you can use the workaround of setting the system property java.security.manager to allow.

danishnawab · 2024-09-26T16:23:43Z

Thanks for the tip @gharris1727.
Ideally I would like to avoid this flag as it has the potential to paper over the future incompatibility in other libraries.

Any idea a future release with this fix might be out? I can then evaluate if we should wait a bit longer for that.

gharris1727 · 2024-09-26T17:44:29Z

Hey @danishnawab I think the earliest we could release this would be 1-4 months from today. Officially Kafka only supports up to Java 21, but we can look to add Java 23 compatibility support in Kafka 4.0, in case this is not the only compatibility issue.

gharris1727 · 2024-09-26T18:06:25Z

@showuon I see some real test failures in the CI, this is not ready to merge just yet. I'll start working through the failures now.

Signed-off-by: Greg Harris <greg.harris@aiven.io>

jsancio · 2024-09-27T12:38:35Z

clients/src/main/java/org/apache/kafka/common/internals/ReflectiveStrategy.java

+/**
+ * A strategy which uses reflection to access methods without requiring them at compile-time.
+ */
+abstract class ReflectiveStrategy implements SecurityManagerCompatibility {


This doesn't need to to be an abstract class. It doesn't have any fields. This can be achieve with static methods, no? And it avoids all of the complexity of inheritance and abstract classes.

Also this class doesn't really implement SecurityManagerCompatibility. It doesn't implement a single method of that interface.

I was using this class to store common methods that were used by both the Legacy and Modern implementations, but not by Composite or Unsupported.

I converted it to a package-local util class instead.

Signed-off-by: Greg Harris <greg.harris@aiven.io>

gharris1727 · 2024-09-27T17:05:01Z

Test failures are resolved, I'll merge this to trunk tomorrow. I plan to backport this to 3.8 and 3.7 immediately, and then 3.9 after the code freeze is finished.

Edit: there were some concerns raised about backporting, i'll only merge this to trunk for now.

danishnawab · 2024-11-18T12:43:35Z

@gharris1727 it seems these changes couldn't make it to 3.9.0 release [1].
What is the new release plan here?

[1] https://downloads.apache.org/kafka/3.9.0/RELEASE_NOTES.html

gharris1727 · 2024-11-18T17:55:48Z

@danishnawab This and the other improvements for Java 23 compatibility are coming in 4.0.0: https://issues.apache.org/jira/browse/KAFKA-17638

danishnawab · 2024-11-19T09:54:00Z

Thanks @gharris1727.
Is there an estimated release schedule for 4.0.0? Even a ballpark estimate would help plan things.

gharris1727 · 2024-11-19T16:09:27Z

@danishnawab You can find the tenative release schedule for 4.0 in the release plan: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+4.0.0

wendigo · 2024-11-20T11:33:30Z

Is this going to be backported to 3.9? Without this change, users on JDK 24 are forced to upgrade to 4.0 due to a lack of alternatives (or patch it on their end). This is something that came up during our pre-24 tests in https://github.com/trinodb/trino

gharris1727 · 2024-11-20T17:29:07Z

@wendigo I've raised this on the mailing list for discussion: https://lists.apache.org/thread/vl43q9wqq4xs67xx61f0t0850y2b037o

Please join the discussion and advocate for this. I think it makes sense to disconnect the Java version from the major version upgrade, but this wasn't ready in time for the 3.9.0 release.

wendigo · 2024-11-20T17:51:53Z

@gharris1727 done :)

vsevel · 2024-12-16T12:29:23Z

hello. is this PR resolving KIP-1006?
we can't migrate to jdk 23 because of the security manager removal, as explained in this quarkus issue.
what do we need to run a kafka client with jdk 23? when will it be available?

wendigo · 2024-12-16T12:34:39Z

According to the discussion in the mailing list, there is no will to backport this to 3.9. This will be available in 4.0 which also drops support for Zookeeper based coordination which is a no-go for a lot of projects so I guess that Kafka won't support JDK23+ in the meantime

vsevel · 2024-12-16T16:51:12Z

will a kafka client in version 4 be able to talk to a kafka 3 broker?

gharris1727 · 2024-12-16T17:08:58Z

will a kafka client in version 4 be able to talk to a kafka 3 broker?

Thanks for the question @vsevel
Yes, a Kafka 4.x client will be able to work with Kafka brokers >= 2.1.x.
And similarly, a Kafka 4.x broker will be able to work with Kafka clients >= 2.1.x.

If you decide to upgrade to the Kafka 4.0 clients to gain Java 23+ support, applications will no longer be able to contact brokers running Kafka 0.x, 1.x, or 2.0.x

You can see the planned phase-out of protocol versions here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0

gharris1727 added 3 commits July 3, 2024 14:36

KAFKA-17078: Add SecurityManagerCompatibility shim

92a31c7

Signed-off-by: Greg Harris <greg.harris@aiven.io>

fixup: spotless

ad1922c

Signed-off-by: Greg Harris <greg.harris@aiven.io>

Merge remote-tracking branch 'upstream/trunk' into kafka-17078-securi…

a42da9d

…ty-manager-shim

C0urante reviewed Aug 5, 2024

View reviewed changes

gharris1727 added 4 commits August 5, 2024 09:29

fixup: deduplicate invoke calls

f1c112f

Signed-off-by: Greg Harris <greg.harris@aiven.io>

fixup: Move SecurityManagerCompatibility inner classes out, add javadocs

c6fc3ed

Signed-off-by: Greg Harris <greg.harris@aiven.io>

fixup: spotless

94a4dd5

Signed-off-by: Greg Harris <greg.harris@aiven.io>

fixup: RemoteLogManager anonymous->lambda was inferring the wrong type

6d92ecb

Signed-off-by: Greg Harris <greg.harris@aiven.io>

OmniaGM reviewed Aug 7, 2024

View reviewed changes

...apache/kafka/common/security/oauthbearer/internals/OAuthBearerSaslClientCallbackHandler.java Show resolved Hide resolved

fixup: Fix handling of checked exceptions from callAs/doAs

205054f

Signed-off-by: Greg Harris <greg.harris@aiven.io>

C0urante approved these changes Aug 19, 2024

View reviewed changes

showuon self-assigned this Aug 20, 2024

showuon approved these changes Aug 27, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/trunk' into kafka-17078-securi…

93aedeb

…ty-manager-shim

github-actions bot added core Kafka Broker connect clients labels Sep 26, 2024

Fix code typo and javadoc indenting

292a443

Signed-off-by: Greg Harris <greg.harris@aiven.io>

jsancio reviewed Sep 27, 2024

View reviewed changes

ReflectiveStrategy abstract -> util class

9159e67

Signed-off-by: Greg Harris <greg.harris@aiven.io>

gharris1727 merged commit 818ee8a into apache:trunk Sep 30, 2024
9 checks passed

wendigo mentioned this pull request Nov 20, 2024

Use and require Java 24 trinodb/trino#23498

Open

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-17078: Add SecurityManagerCompatibility shim #16522

KAFKA-17078: Add SecurityManagerCompatibility shim #16522

gharris1727 commented Jul 3, 2024

C0urante left a comment

C0urante Aug 5, 2024

gharris1727 Aug 5, 2024

C0urante Aug 6, 2024

C0urante left a comment

showuon left a comment

showuon Aug 27, 2024

showuon commented Sep 25, 2024

danishnawab commented Sep 26, 2024

mumrah commented Sep 26, 2024

gharris1727 commented Sep 26, 2024

danishnawab commented Sep 26, 2024

gharris1727 commented Sep 26, 2024

gharris1727 commented Sep 26, 2024

jsancio Sep 27, 2024

gharris1727 Sep 27, 2024

gharris1727 commented Sep 27, 2024 •

edited

Loading

danishnawab commented Nov 18, 2024

gharris1727 commented Nov 18, 2024

danishnawab commented Nov 19, 2024

gharris1727 commented Nov 19, 2024

wendigo commented Nov 20, 2024

gharris1727 commented Nov 20, 2024

wendigo commented Nov 20, 2024

vsevel commented Dec 16, 2024

wendigo commented Dec 16, 2024

vsevel commented Dec 16, 2024

gharris1727 commented Dec 16, 2024


		<T> T doPrivileged(PrivilegedAction<T> action);

		Subject current();

KAFKA-17078: Add SecurityManagerCompatibility shim #16522

KAFKA-17078: Add SecurityManagerCompatibility shim #16522

Conversation

gharris1727 commented Jul 3, 2024

Committer Checklist (excluded from commit message)

C0urante left a comment

Choose a reason for hiding this comment

C0urante Aug 5, 2024

Choose a reason for hiding this comment

gharris1727 Aug 5, 2024

Choose a reason for hiding this comment

C0urante Aug 6, 2024

Choose a reason for hiding this comment

C0urante left a comment

Choose a reason for hiding this comment

showuon left a comment

Choose a reason for hiding this comment

showuon Aug 27, 2024

Choose a reason for hiding this comment

showuon commented Sep 25, 2024

danishnawab commented Sep 26, 2024

mumrah commented Sep 26, 2024

gharris1727 commented Sep 26, 2024

danishnawab commented Sep 26, 2024

gharris1727 commented Sep 26, 2024

gharris1727 commented Sep 26, 2024

jsancio Sep 27, 2024

Choose a reason for hiding this comment

gharris1727 Sep 27, 2024

Choose a reason for hiding this comment

gharris1727 commented Sep 27, 2024 • edited Loading

danishnawab commented Nov 18, 2024

gharris1727 commented Nov 18, 2024

danishnawab commented Nov 19, 2024

gharris1727 commented Nov 19, 2024

wendigo commented Nov 20, 2024

gharris1727 commented Nov 20, 2024

wendigo commented Nov 20, 2024

vsevel commented Dec 16, 2024

wendigo commented Dec 16, 2024

vsevel commented Dec 16, 2024

gharris1727 commented Dec 16, 2024

gharris1727 commented Sep 27, 2024 •

edited

Loading