Support for JDK17 #885

dkroehan · 2022-02-18T12:35:19Z

Describe the bug
It seems that Kryo is currently not compatible with JDK17. When calling the default constructor it fails with:

java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @61322f9d
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
	at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
	at com.esotericsoftware.kryo.unsafe.UnsafeUtil.<clinit>(UnsafeUtil.java:100)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at com.esotericsoftware.kryo.util.Util.<clinit>(Util.java:47)
	at com.esotericsoftware.kryo.serializers.OptionalSerializers.addDefaultSerializers(OptionalSerializers.java:38)
	at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:230)
	at com.esotericsoftware.kryo.Kryo.<init>(Kryo.java:168)

To Reproduce
Kryo kryo = new Kryo();

Environment:

JDK Version: OpenJDK 17.0.2 (Corretto-17.0.2.8.1)
Kryo Version: 5.3.0

Additional context
In my case I am also registering some standard classes.
It seems to work for classes like:

java.util.Locale
java.time.Instant
java.util.HashMap

But it fails for java.util.regex.Pattern due to:

java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.util.regex.Pattern.pattern accessible: module java.base does not "opens java.util.regex" to unnamed module @185a6e9

What would be the proper way to fix this?

The text was updated successfully, but these errors were encountered:

theigl · 2022-02-18T13:07:35Z

Your specific case can be solved by using a custom RegexSerializer that doesn't rely on reflection.

Alternatively, you can add JVM arguments for opening up the modules in question. See this comment.

I think it makes sense to include the RegexSerializer and UUIDSerializer directly in Kryo, so those two classes can be serialized on JDK 17 without opening any modules.

theigl · 2022-02-18T13:24:47Z

I cannot reproduce your first issue. I just downloaded and ran my tests on Corretto to make sure it isn't a JVM specific problem.

dkroehan · 2022-02-21T07:31:44Z

Thank you for the quick response.

I did set a breakpoint at AccessibleObject#checkCanSetAccessible at line 354 where an InaccessibleObjectException is created for debugging purposes.
This is where I found: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @61322f9d

This is caused by the Util static initializer block at: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/util/Util.java#L47
This calls UnsafeUtil with the static initializer at: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/unsafe/UnsafeUtil.java#L99

The InaccessibleObjectException is caught in line 101, which causes the directByteBufferConstructor to be null. I didn't pay attention that the Exception is actually not bubbling up.

Now I'm wondering if this is expected behaviour or a failure with java 17.

dkroehan · 2022-02-21T08:21:58Z

I tried the RegexSerializer using:

kryo.register(Pattern.class, new RegexSerializer());

It seems to work.

FYI: UnmodifiableCollectionsSerializer from kryo-serializers also lacks compatibility. There is already an open issue, which I just updated:
#magro/kryo-serializers#131 (comment)

theigl · 2022-02-21T09:57:14Z

Now I'm wondering if this is expected behaviour or a failure with java 17.

This is expected behavior and has no effect unless you want to use UnsafeByteBuffers for Input/Output. If you want to use those, you have to open the module up for reflection.

FYI: UnmodifiableCollectionsSerializer from kryo-serializers also lacks compatibility. There is already an open issue, which I just updated:

AFAIK it is impossible to write a non-reflective version of a serializer for UnmodifiableCollection that covers all potential cases. So these serializers will probably never make it into Kryo and you will have to add add-opens arguments if you need to serialize them.

theigl · 2022-04-27T08:29:22Z

Kryo is currently missing built-in, safe serializers for the following JDK classes:

java.net.URI
java.util.UUID
java.util.regex.Pattern
java.util.concurrent.atomic.AtomicBoolean
java.util.concurrent.atomic.AtomicInteger
java.util.concurrent.atomic.AtomicLong
java.util.concurrent.atomic.AtomicReference

These serializers are trivial to implement, but cannot be registered in Kryo 5 as default serializers because it would break backwards compatibility. I will add them to DefaultSerializers so users can register them, but they will only be registered by default in Kryo 6.

jdorleans · 2022-06-17T06:29:11Z

I'm trying to upgrade from Java 11 to 17 running Kryo 5 with Unsafe I/O, but am getting a weird NoClassDefFoundError for UnsafeUtil while writing data with writeClassAndObject:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.esotericsoftware.kryo.kryo5.unsafe.UnsafeUtil
	at com.esotericsoftware.kryo.kryo5.unsafe.UnsafeOutput.writeByte(UnsafeOutput.java:93)
	at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeName(DefaultClassResolver.java:123)
	at com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:114)
	at com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613)
	at com.esotericsoftware.kryo.kryo5.Kryo.writeClassAndObject(Kryo.java:708)

I also tested moving away from Unsafe* to use Input/Output and it works fine. Now I'm wondering how do I get Unsafe to work again.

theigl · 2022-06-23T09:01:30Z

@jdorleans: On JDK 17, you currently need to open up sun.nio.ch like this:

--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

Otherwise initialization of UnsafeUtil fails because it cannot run DirectBuffer.class.getMethod("cleaner"). We should allow usage of unsafe without requiring access to DirectBuffer. I'll investigate how other libraries solve this. For now, please add the --add-opens statement.

graves501 · 2022-07-07T11:07:40Z

I guess I'm just doing something wrong, but I'm also currently trying the latest kryo version (5.3.0) in combination with Java 17.
I'm trying to clone an object that contains a list.

final var kryo = new Kryo();
kryo.register(ClassWithList.class);
ClassWithList copy = kryo.copy(classWithList);

I get this error messsage:

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: java.util.ImmutableCollections$List12
Note: To register this class use: kryo.register(java.util.ImmutableCollections.List12.class);

Registering this class doesn't work, since List12 is not publicly accessible.

theigl · 2022-07-10T11:10:30Z

@graves501: You can either set registrationRequired(false) or manually register the classes using one of the following:

kryo.register(List.of(1).getClass());
kryo.register(List.of(1,2,3).getClass());

kryo.register(Class.forName("java.util.ImmutableCollections$List12"));
kryo.register(Class.forName("java.util.ImmutableCollections$ListN"));

graves501 · 2022-07-13T09:59:55Z

Thanks, that did the job!

theigl · 2022-12-20T15:51:48Z

Quick update:

I added the following serializers to Kryo. These serializers are not registered by default (because of backwards compatibility), but can be registered as needed:

java.net.URI
java.util.UUID
java.util.regex.Pattern
java.util.concurrent.atomic.AtomicBoolean
java.util.concurrent.atomic.AtomicInteger
java.util.concurrent.atomic.AtomicLong
java.util.concurrent.atomic.AtomicReference

See Add additional safe serializers for commonly used JDK classes #930

It should no longer be necessary to open up sun.nio.ch. Access to DirectBuffer is delayed until it is first used.

See Delay access to DirectBuffers until first use #932
I added a helper method to register serializers for ImmutableCollections:

ImmutableCollectionsSerializers.registerSerializers(kryo);

See #923 Add helper to register serializers for java.util.ImmutableCollections #933

It would be great if anyone could confirm the second change using the current snapshots.

leonchen83 · 2023-01-03T03:39:17Z

@theigl

Exception in thread "main" java.lang.reflect.InaccessibleObjectException: Unable to make field private transient java.lang.Object java.lang.Throwable.backtrace accessible: module java.base does not "opens java.lang" to unnamed module @2eafffde
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.esotericsoftware.kryo.serializers.CachedFields.addField(CachedFields.java:123)
	at com.esotericsoftware.kryo.serializers.CachedFields.rebuild(CachedFields.java:99)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:82)
	at com.esotericsoftware.kryo.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:124)
	at com.esotericsoftware.kryo.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:108)
	at com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:469)
	at com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:454)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:89)
	at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:581)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:112)
	at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:613)
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:708)
	at cn.nextop.test.Main.writeClassAndObject(Main.java:59)
	at cn.nextop.test.Main.main(Main.java:37)

version: kryo 5.4.0

reproduce code

import java.util.concurrent.ExecutionException;

import org.objenesis.strategy.StdInstantiatorStrategy;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.unsafe.UnsafeInput;
import com.esotericsoftware.kryo.unsafe.UnsafeOutput;
import com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy;

public class Main {
	public static void main(String[] args) {
		Main main = new Main();
		
		Kryo kryo = main.init();
		ExecutionException rpc = new ExecutionException(null, new Throwable());
		byte[] bytes = main.writeClassAndObject(kryo, rpc);
		ExecutionException value = (ExecutionException) main.readClassAndObject(kryo, bytes);
	}
	
	public Kryo init() {
		Kryo kryo = new Kryo();
		kryo.setReferences(true);
		kryo.setRegistrationRequired(false);
		kryo.setWarnUnregisteredClasses(false);
		
		DefaultInstantiatorStrategy init = (DefaultInstantiatorStrategy) kryo.getInstantiatorStrategy();
		init.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
		return kryo;
	}
	
	public Object readClassAndObject(Kryo kryo, byte[] bytes ) {
		UnsafeInput input = new UnsafeInput();
		input.setBuffer(bytes); Object r = kryo.readClassAndObject(input); return r;
	}
	
	public byte[] writeClassAndObject (Kryo kryo, Object value) {
		UnsafeOutput output = new UnsafeOutput(1024, 32 * 1024 * 1024);
		kryo.writeClassAndObject(output, value); return output.toBytes();
	}
}

We need to serialize ExecutionException because we need to implement an RPC service by kryo. so could we add ExecutionExceptionSerializer to kryo 5.4.0?

theigl · 2023-01-03T10:50:16Z

@leonchen83: I would not consider serializing exceptions a typical use-case for Kryo, so I'm against adding exception serializers to Kryo itself.

You have two options:

Write your own serializer
Use JavaSerializer for serializing exceptions

I'd go for the second option. Exceptions should occur very rarely, so speed of serialization and payload size usually doesn't matter that much.

Redisson for instance is using this approach.

amodolo · 2023-05-30T13:14:51Z

Hi @theigl,
also ClosureSerializer is not compatible with JDK 17. Is there any workarround to solve this

Version 5.5.0
JDK: temurin 17.0.7

Here is the code used to reproduce the exception

public static void main(String[] args) {
    Kryo kryo = new Kryo();
    kryo.register(ClosureSerializer.Closure.class, new ClosureSerializer());
}

here is the exception

00:00  WARN: Unable to obtain SerializedLambda#readResolve via reflection. Falling back on resolving lambdas via capturing class.
java.lang.reflect.InaccessibleObjectException: Unable to make private java.lang.Object java.lang.invoke.SerializedLambda.readResolve() throws java.io.ObjectStreamException accessible: module java.base does not "opens java.lang.invoke" to unnamed module @46238e3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
	at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)
	at com.esotericsoftware.kryo.serializers.ClosureSerializer.<init>(ClosureSerializer.java:59)
	at dev.test.serialization.kryo.Main.main(Main.java:9)
00:00  WARN: Unable to obtain SerializedLambda#capturingClass via reflection. Falling back to resolving capturing class via Class.forName.
java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.lang.Class java.lang.invoke.SerializedLambda.capturingClass accessible: module java.base does not "opens java.lang.invoke" to unnamed module @46238e3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.esotericsoftware.kryo.serializers.ClosureSerializer.<init>(ClosureSerializer.java:69)
	at dev.test.serialization.kryo.Main.main(Main.java:9)

I've already try to set this argument to the JVM

--add-opens=java.base/java.lang.invoke=ALL-UNNAMED

but it solves nothing

theigl · 2023-05-30T13:27:15Z

@amodolo: I am using ClosureSerializer on OpenJDK 17 for quite a while now. It works fine. Are you sure you are passing the JVM argument correctly? I'm passing the following argument:

--add-opens java.base/java.lang.invoke=ALL-UNNAMED

Maybe the additional = causes problems in your case?

amodolo · 2023-05-30T13:54:09Z

You are right. there is an additional =. even without it, the problem is still there. Here is my configuration.

theigl · 2023-05-30T14:08:38Z

I just ran ClosureSerializerTest in Kryo with the following settings:

It works fine.

The command run by IntelliJ looks like this:

/Users/XXX/Library/Java/JavaVirtualMachines/temurin-17.0.7/Contents/Home/bin/java --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --enable-preview ......

How does it look in your case?

amodolo · 2023-05-30T14:19:52Z

Oh i see what's going wrong.
i passed the --add-opens to the program arguments instead of VM options. Your configuration solves the problem.
Thank you

endlesstian · 2023-10-18T05:45:26Z

Kryo is currently missing built-in, safe serializers for the following JDK classes:

java.net.URI

java.util.UUID

java.util.regex.Pattern

java.util.concurrent.atomic.AtomicBoolean

java.util.concurrent.atomic.AtomicInteger

java.util.concurrent.atomic.AtomicLong

java.util.concurrent.atomic.AtomicReference

These serializers are trivial to implement, but cannot be registered in Kryo 5 as default serializers because it would break backwards compatibility. I will add them to DefaultSerializers so users can register them, but they will only be registered by default in Kryo 6.

Hi, when will Kryo 6 release? Thanks.
BRs,
Tian

theigl · 2023-10-18T15:01:49Z

@endlesstian: You can already register all these serializers yourself. They are available for manual registration since Kryo 5.4.0.

As for Kryo 6, I currently do not have a release date. I will have more spare time in December and I'll see if I can create a release candidate by the end of the year.

vyalyh-oleg · 2023-12-23T12:56:11Z

Can we expect, that the rkyo library will support modular system ?
--add-opens isn't a problem, but I would like to open the specific package not to all UNNAMED modules, but e.g. to a specific one (kryo).
I think it could solve some problems.

theigl · 2023-12-23T13:01:11Z

@vyalyh-oleg: Kryo currently only adds an automatic module name. So far, there is no module definition. What is required for add-opens to be restricted to Kryo?

dkroehan added bug needs repro labels Feb 18, 2022

theigl added question and removed bug needs repro labels Feb 18, 2022

theigl mentioned this issue Feb 18, 2022

Serializing UUID not working with java 17 (but same works on java 11) #859

Closed

theigl mentioned this issue Apr 27, 2022

An illegal reflective access operation has occurred on application start #895

Closed

chrishpowell mentioned this issue May 14, 2022

sjm fails with Graal (Java17) bbottema/simple-java-mail#394

Closed

herodotus-ecosystem mentioned this issue May 27, 2022

环境是jdk17，可以使用吗？ alibaba/jetcache#635

Open

theigl added the help wanted label Jul 21, 2022

regadas mentioned this issue Sep 7, 2022

Support running on Java 17 twitter/chill#692

Open

EdwardvanRaak mentioned this issue Sep 29, 2022

Allow Kryo serializer registration openEHR/archie#441

Closed

nschlimm mentioned this issue Oct 2, 2022

Upgrade project to support java 17 projectbarbel/barbelhisto-core#13

Open

This was referenced Dec 16, 2022

Add additional safe serializers for commonly used JDK classes #930

Merged

Delay access to DirectBuffers until first use #932

Merged

theigl pinned this issue Dec 20, 2022

JamesTKhan mentioned this issue May 10, 2023

Build issues on newer JDK's (workaround available) JamesTKhan/Mundus#170

Closed

nikhil-sec mentioned this issue Nov 23, 2023

java.util.concurrent* objects are not compatible with JDK-17 redisson/redisson#5452

Closed

jgomer2001 mentioned this issue Nov 29, 2023

feat(agama): use a mixed strategy for serialization JanssenProject/jans#6834

Closed

ad1happy2go mentioned this issue May 9, 2024

[SUPPORT] Hudi fails ACID verification test apache/hudi#11170

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for JDK17 #885

Support for JDK17 #885

dkroehan commented Feb 18, 2022

theigl commented Feb 18, 2022

theigl commented Feb 18, 2022

dkroehan commented Feb 21, 2022

dkroehan commented Feb 21, 2022 •

edited

Loading

theigl commented Feb 21, 2022

theigl commented Apr 27, 2022

jdorleans commented Jun 17, 2022 •

edited

Loading

theigl commented Jun 23, 2022

graves501 commented Jul 7, 2022 •

edited

Loading

theigl commented Jul 10, 2022

graves501 commented Jul 13, 2022

theigl commented Dec 20, 2022 •

edited

Loading

leonchen83 commented Jan 3, 2023 •

edited

Loading

theigl commented Jan 3, 2023

amodolo commented May 30, 2023

theigl commented May 30, 2023

amodolo commented May 30, 2023

theigl commented May 30, 2023

amodolo commented May 30, 2023

endlesstian commented Oct 18, 2023

theigl commented Oct 18, 2023

vyalyh-oleg commented Dec 23, 2023

theigl commented Dec 23, 2023

Support for JDK17 #885

Support for JDK17 #885

Comments

dkroehan commented Feb 18, 2022

theigl commented Feb 18, 2022

theigl commented Feb 18, 2022

dkroehan commented Feb 21, 2022

dkroehan commented Feb 21, 2022 • edited Loading

theigl commented Feb 21, 2022

theigl commented Apr 27, 2022

jdorleans commented Jun 17, 2022 • edited Loading

theigl commented Jun 23, 2022

graves501 commented Jul 7, 2022 • edited Loading

theigl commented Jul 10, 2022

graves501 commented Jul 13, 2022

theigl commented Dec 20, 2022 • edited Loading

leonchen83 commented Jan 3, 2023 • edited Loading

theigl commented Jan 3, 2023

amodolo commented May 30, 2023

theigl commented May 30, 2023

amodolo commented May 30, 2023

theigl commented May 30, 2023

amodolo commented May 30, 2023

endlesstian commented Oct 18, 2023

theigl commented Oct 18, 2023

vyalyh-oleg commented Dec 23, 2023

theigl commented Dec 23, 2023

dkroehan commented Feb 21, 2022 •

edited

Loading

jdorleans commented Jun 17, 2022 •

edited

Loading

graves501 commented Jul 7, 2022 •

edited

Loading

theigl commented Dec 20, 2022 •

edited

Loading

leonchen83 commented Jan 3, 2023 •

edited

Loading