[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

jerboaa · 2024-03-22T10:27:20Z

Describe the bug

In our Mandrel native integration tests we see new failures since the JDK 23+15 ea build in quarkus-integration-test-kafka-oauth-keycloak-999-SNAPSHOT-runner tests:

2024-03-22 02:18:58,664 WARN  [org.apa.kaf.com.net.Selector] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Unexpected error from localhost/127.0.0.1 (channelId=-1); closing connection: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base@23-beta/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)
	at java.base@23-beta/jdk.internal.vm.ScopedValueContainer.callWithoutScope(ScopedValueContainer.java:162)
	at java.base@23-beta/jdk.internal.vm.ScopedValueContainer.call(ScopedValueContainer.java:147)
	at java.base@23-beta/java.lang.ScopedValue$Carrier.runWith(ScopedValue.java:74)
	at java.base@23-beta/java.lang.ScopedValue$Carrier.call(ScopedValue.java:419)
	at java.base@23-beta/java.lang.ScopedValue.callWhere(ScopedValue.java:588)
	at java.base@23-beta/javax.security.auth.Subject.callAs(Subject.java:439)
	at java.base@23-beta/javax.security.auth.Subject.doAs(Subject.java:614)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:535)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:434)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:333)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:274)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:349)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:252)
	at java.base@23-beta/java.lang.Thread.runWith(Thread.java:1588)
	at java.base@23-beta/java.lang.Thread.run(Thread.java:1575)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:836)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:812)

See: https://github.com/graalvm/mandrel/actions/runs/8384081617/job/22961614228#step:12:968

I'm pretty sure, though, this isn't a native-only issue but can happen in JVM mode on JDK 23+15 as well.

Expected behavior

No java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed is being thrown.

Actual behavior

java.lang.UnsupportedOperationException is being thrown.

How to Reproduce?

Build Integration Tests - Kafka OAUTH with Keycloak native test and run test with a 23-beta+15-ea, vendor version: Mandrel-24.1.0-dev13054560 mandrel build. For example from here.

Additional information

This seems to be caused by https://bugs.openjdk.org/browse/JDK-8296244 freshly part of JDK 23+15. See the CSR on how to possibly fix it: https://bugs.openjdk.org/browse/JDK-8327134

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2024-03-22T10:27:24Z

/cc @Karm (mandrel), @alesj (kafka), @cescoffier (kafka), @galderz (mandrel), @ozangunalp (kafka), @pedroigor (bearer-token), @sberyozkin (bearer-token,security), @zakkak (mandrel)

cescoffier · 2024-03-22T12:08:30Z

Unfortunately there is not much we can do. It's an issue to report in the Apache Kafka Client.

jerboaa · 2024-03-22T12:55:33Z

OK. Feel free to bring this to the Apache Kafka Client team.

gsmet · 2024-03-25T16:29:41Z

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

jerboaa · 2024-03-25T16:43:40Z

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

Yes the security manager is going away. The reason this change was done was supporting this move. Please see the CSR of this change which should have all the info needed: https://bugs.openjdk.org/browse/JDK-8327134

It appears a possible work-around is to add -Djava.security.manager=allow. But this comes with the following caveat:

If for some reason, user cannot modify their code in time, they can add
`-Djava.security.manager=allow` on the command line as a workaround.
This allows the old implementation to be used even if a Security Manager is never set.
Please note that in a future version when the Security Manager is completely
removed this workaround would fail.

Hope that helps.

jerboaa · 2024-04-04T15:47:55Z

@cescoffier Is there an upstream kafka client issue for this? Who would we need to talk to to get it created? I'm hoping to get this on the radar for kafka client folks.

cescoffier · 2024-04-04T15:50:59Z

Yes, it's an issue for Apache Kafka (the upstream project).

jerboaa · 2024-04-04T16:15:11Z

Understood. If possible, could somebody create an upstream issue? That tracker seems invite only and I'm not keen on creating one just for a single bug. Could somebody with an (existing) account there file one? Thanks!

https://issues.apache.org/jira/projects/KAFKA/issues

cescoffier · 2024-04-05T05:30:50Z

@ozangunalp do you have an account?

@jerboaa let me ping Mickael Maison (working in the AMQ Streams team)

jerboaa · 2024-04-05T08:27:32Z

Thanks! Please let me know once somebody has created an issue.

cescoffier · 2024-04-08T07:23:38Z

@jerboaa There is an existing ticket: https://issues.apache.org/jira/browse/KAFKA-15862.

However, it's not planned before the Q3.

jerboaa · 2024-04-08T08:49:51Z

@cescoffier OK, thanks. Whether or not that is a problem for quarkus is something you know better. GraalVM for JDK 23 will get out in September 2024, which is when a fix/known issue for this needs to be in place.

cescoffier · 2024-04-08T18:30:56Z

Yeah, the timing might be problematic. Basically, if it's not fixed before September 2024, it would not be possible to run Quarkus apps using Kafka authentication in native.

Note that it's not only Quarkus, it's anything compiling Java to native using GraalVM.

jerboaa · 2024-04-08T18:50:43Z

Yeah, the timing might be problematic. Basically, if it's not fixed before September 2024, it would not be possible to run Quarkus apps using Kafka authentication in native.

That's my understanding. At least with GraalVM for JDK 23 (the older GraalVM for JDK 21 would still work).

geoand · 2024-06-06T17:17:05Z

Is this still an issue?

jerboaa · 2024-06-06T17:36:05Z

Yes. See graalvm/mandrel#742 (comment)

cescoffier · 2024-06-07T05:39:10Z

The Kafka team has an issue for it too. It should land 'just in time' (September-ish)

jerboaa · 2024-08-22T13:41:06Z

The Kafka team has an issue for it too. It should land 'just in time' (September-ish)

@cescoffier Do you know any update on this? JDK 23 is in RC 2 and there will be a Mandrel for JDK 23 release shortly after JDK 23 GA. Should we start planning on drafting a "known issue" or the like for this, or do you think there will be a fix for it in time still.

cescoffier · 2024-08-22T14:14:11Z

@jerboaa no news so far - there is a keep and a jira (https://issues.apache.org/jira/browse/KAFKA-15862), but nothing in motion.

jerboaa · 2024-08-22T14:21:22Z

Thanks. Sounds like it's going to be a known issue for the Mandrel 24.1 for JDK 23 release.

jerboaa · 2024-09-06T10:43:59Z

@cescoffier Is there a way to advise quarkus users about the issue with the kafka-oauth-keycloak extension when they run on JDK 23+? Could we tell them that this extension isn't working on JDK 23 perhaps pointing here?

cescoffier · 2024-09-06T10:49:46Z

Except in the documentation, i don't think we can do much.

@ozangunalp do you think we can intercept the exception and log a more meaningful message?

ozangunalp · 2024-09-06T11:23:20Z

Last time I checked I couldn't reproduce this on JVM mode with 23.ea.29-open (by simply running the kafka-oauth-keycloak test)

@jerboaa is there anywhere I can get an early access release Mandrel for JDK 23?

cc @cescoffier

zakkak · 2024-09-06T12:46:39Z

@jerboaa is there anywhere I can get an early access release Mandrel for JDK 23?

@ozangunalp you can get the latest build from https://github.com/graalvm/mandrel/actions/runs/10722382634/artifacts/1896697043

Built in this CI run https://github.com/graalvm/mandrel/actions/runs/10722382634

jerboaa · 2024-09-06T12:47:25Z

Last time I checked I couldn't reproduce this on JVM mode with 23.ea.29-open (by simply running the kafka-oauth-keycloak test)

You probably need to exercise this code path:

2024-03-22 02:18:58,664 WARN  [org.apa.kaf.com.net.Selector] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Unexpected error from localhost/127.0.0.1 (channelId=-1); closing connection: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base@23-beta/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
	at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
	at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)

Not sure if the test does that in all cases.

ozangunalp · 2024-09-06T14:37:17Z

That's not about the test code path. I had a -Djava.security.manager=allow on that test.
I can certainly reproduce it on jdk 23.

jerboaa · 2024-09-06T14:44:06Z

I had a -Djava.security.manager=allow on that test.

That would explain it yes.

The test is known to fail for quite some time generating a lot of CI noise. Let's skip it till quarkusio#39634 is resolved.

cescoffier · 2024-11-06T06:42:32Z

The root cause should be fixed in Kafka 4. Until that, we would need the flag (as shown by Ozan)

jerboaa · 2024-11-06T09:08:40Z

The root cause should be fixed in Kafka 4. Until that, we would need the flag (as shown by Ozan)

Note that -Djava.security.manager=allow will unconditional throw an exception once JEP 486 integrates for JDK 24. Is there a plan to move to Kafka 4 in time for JDK 25, say?

cescoffier · 2024-11-06T13:38:37Z

Hard to say, it will be tricky.
Kafka 4 is not released yet, hopefully this year. But even with that we would need to do some integration work to continue supporting both Kafka 3 and 4 for a bit.

drakgoku · 2024-11-10T16:52:04Z

This happens because from JDK 17 onwards, the Security Manager is deprecated and in JDK 23 there are significant changes in how security is handled.

It seems that all of us who are on JDK 23 or updated projects have been stuck because of Kafka and have posted it in the wrong place. It's not Quarkus' fault but Kafka's. But since this problem doesn't exist on the Internet, then we come to post it here.
So this post should be redirected to Kafka or Java or even spring.

This is going to be forgotten. You have to link to Java, Kafka and Spring boot.

This mainly has to be linked to Kafka/Java to be fixed.

zakkak · 2024-11-15T07:34:58Z

So this post should be redirected to Kafka or Java or even spring.
...
This mainly has to be linked to Kafka/Java to be fixed.

@drakgoku this is already tracked in Kafka as https://issues.apache.org/jira/browse/KAFKA-15862 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support

As mentioned in spring-projects/spring-kafka#3619 (comment) it would be best if people affected by it could somehow reach the Kafka team.

I don't have an account on their issue tracker and I can't really see the comments but it looks like the issue was last updated on 30/Sep/24, and according to @cescoffier it will be fixed in version 4.0.0

Update: You can also try reaching out the Kafka team through their mailing lists https://kafka.apache.org/contact.html

anshupitlia · 2024-11-16T09:36:51Z

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

Yes the security manager is going away. The reason this change was done was supporting this move. Please see the CSR of this change which should have all the info needed: https://bugs.openjdk.org/browse/JDK-8327134

It appears a possible work-around is to add -Djava.security.manager=allow. But this comes with the following caveat:
If for some reason, user cannot modify their code in time, they can add
`-Djava.security.manager=allow` on the command line as a workaround.
This allows the old implementation to be used even if a Security Manager is never set.
Please note that in a future version when the Security Manager is completely
removed this workaround would fail.
Hope that helps.

Tried this config, still not working. :(

…s not have security manager which is needed by Kafka. More details here - quarkusio/quarkus#39634.

The test is known to fail for quite some time generating a lot of CI noise. Let's skip it till quarkusio#39634 is resolved.

vsevel · 2024-12-16T08:09:51Z

we have been hit by this issue as well trying to upgrade to 23 to solve a graalvm issue fixed in 21, as suggested in a jobrunr issue discussed in this zulip thread.

jerboaa added the kind/bug Something isn't working label Mar 22, 2024

quarkus-bot bot added area/kafka area/native-image area/oidc area/security labels Mar 22, 2024

jerboaa mentioned this issue Mar 22, 2024

[CI] Quarkus main fails with Java 23 mandrel build of graal/master on Linux graalvm/mandrel#652

Closed

sberyozkin removed the area/oidc label Mar 22, 2024

jerboaa mentioned this issue Jun 4, 2024

[CI] Quarkus main fails with Java 23 mandrel build of mandrel/24.1 on Linux graalvm/mandrel#742

Closed

jerboaa mentioned this issue Jun 26, 2024

[24.1] Backport JDK 23+26 upgrade changes to 24.1 release tree graalvm/mandrel#751

Closed

This was referenced Jul 4, 2024

[24.1] Backport: Updated JFR substitutions from [GR-54877] graalvm/mandrel#764

Merged

[CI] Quarkus main fails with Java 24 mandrel build of graal/master on Linux graalvm/mandrel#755

Closed

zakkak added a commit to zakkak/quarkus that referenced this issue Sep 26, 2024

Skip kafka-oauth-keycloak test with GraalVM > 24.0

28f4b4b

The test is known to fail for quite some time generating a lot of CI noise. Let's skip it till quarkusio#39634 is resolved.

zakkak added a commit to zakkak/quarkus that referenced this issue Sep 26, 2024

Skip kafka-oauth-keycloak test with GraalVM >= 24.1

b20e1fa

The test is known to fail for quite some time generating a lot of CI noise. Let's skip it till quarkusio#39634 is resolved.

zakkak added a commit to zakkak/quarkus that referenced this issue Sep 26, 2024

Skip kafka-oauth-keycloak test with GraalVM >= 24.1

f06e771

The test is known to fail for quite some time generating a lot of CI noise. Let's skip it till quarkusio#39634 is resolved.

zakkak mentioned this issue Sep 26, 2024

Skip kafka-oauth-keycloak test with GraalVM >= 24.1 #43543

Merged

ozangunalp mentioned this issue Oct 30, 2024

Kafka Oauth extension #18998

Closed

drakgoku mentioned this issue Nov 10, 2024

Kafka and Spring Boot JDK 23 - Caused by: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed spring-projects/spring-kafka#3619

Closed

anshupitlia added a commit to anshupitlia/product-information-system that referenced this issue Nov 16, 2024

Add bare minimum Kafka producer. Downgrade Java version, since 23 doe…

2c3a8db

…s not have security manager which is needed by Kafka. More details here - quarkusio/quarkus#39634.

bschuhmann pushed a commit to bschuhmann/quarkus that referenced this issue Nov 16, 2024

Skip kafka-oauth-keycloak test with GraalVM >= 24.1

5daf426

The test is known to fail for quite some time generating a lot of CI noise. Let's skip it till quarkusio#39634 is resolved.

vsevel mentioned this issue Dec 16, 2024

KAFKA-17078: Add SecurityManagerCompatibility shim apache/kafka#16522

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

jerboaa commented Mar 22, 2024 •

edited

Loading

quarkus-bot bot commented Mar 22, 2024

cescoffier commented Mar 22, 2024

jerboaa commented Mar 22, 2024

gsmet commented Mar 25, 2024

jerboaa commented Mar 25, 2024

jerboaa commented Apr 4, 2024 •

edited

Loading

cescoffier commented Apr 4, 2024

jerboaa commented Apr 4, 2024

cescoffier commented Apr 5, 2024

jerboaa commented Apr 5, 2024

cescoffier commented Apr 8, 2024

jerboaa commented Apr 8, 2024

cescoffier commented Apr 8, 2024

jerboaa commented Apr 8, 2024

geoand commented Jun 6, 2024

jerboaa commented Jun 6, 2024

cescoffier commented Jun 7, 2024

jerboaa commented Aug 22, 2024

cescoffier commented Aug 22, 2024

jerboaa commented Aug 22, 2024

jerboaa commented Sep 6, 2024

cescoffier commented Sep 6, 2024

ozangunalp commented Sep 6, 2024 •

edited

Loading

zakkak commented Sep 6, 2024

jerboaa commented Sep 6, 2024

ozangunalp commented Sep 6, 2024 •

edited

Loading

jerboaa commented Sep 6, 2024

cescoffier commented Nov 6, 2024

jerboaa commented Nov 6, 2024

cescoffier commented Nov 6, 2024

drakgoku commented Nov 10, 2024 •

edited

Loading

zakkak commented Nov 15, 2024 •

edited

Loading

anshupitlia commented Nov 16, 2024

vsevel commented Dec 16, 2024

[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

Comments

jerboaa commented Mar 22, 2024 • edited Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Additional information

quarkus-bot bot commented Mar 22, 2024

cescoffier commented Mar 22, 2024

jerboaa commented Mar 22, 2024

gsmet commented Mar 25, 2024

jerboaa commented Mar 25, 2024

jerboaa commented Apr 4, 2024 • edited Loading

cescoffier commented Apr 4, 2024

jerboaa commented Apr 4, 2024

cescoffier commented Apr 5, 2024

jerboaa commented Apr 5, 2024

cescoffier commented Apr 8, 2024

jerboaa commented Apr 8, 2024

cescoffier commented Apr 8, 2024

jerboaa commented Apr 8, 2024

geoand commented Jun 6, 2024

jerboaa commented Jun 6, 2024

cescoffier commented Jun 7, 2024

jerboaa commented Aug 22, 2024

cescoffier commented Aug 22, 2024

jerboaa commented Aug 22, 2024

jerboaa commented Sep 6, 2024

cescoffier commented Sep 6, 2024

ozangunalp commented Sep 6, 2024 • edited Loading

zakkak commented Sep 6, 2024

jerboaa commented Sep 6, 2024

ozangunalp commented Sep 6, 2024 • edited Loading

jerboaa commented Sep 6, 2024

cescoffier commented Nov 6, 2024

jerboaa commented Nov 6, 2024

cescoffier commented Nov 6, 2024

drakgoku commented Nov 10, 2024 • edited Loading

zakkak commented Nov 15, 2024 • edited Loading

anshupitlia commented Nov 16, 2024

vsevel commented Dec 16, 2024

jerboaa commented Mar 22, 2024 •

edited

Loading

jerboaa commented Apr 4, 2024 •

edited

Loading

ozangunalp commented Sep 6, 2024 •

edited

Loading

ozangunalp commented Sep 6, 2024 •

edited

Loading

drakgoku commented Nov 10, 2024 •

edited

Loading

zakkak commented Nov 15, 2024 •

edited

Loading