UnreadableSessionDataException with HazelcastSessionDataStore in Jetty 9.4.15 and newer

[eicki]

With Jetty 9.4.12 session persistence using hazelcast worked, it is broken in 9.4.15 probably caused by the changes done with 871f73c#diff-57280dc468d8ce6adc575652556a6423

If a node goes down the session cannot be read on the other node giving the following exception:

18.04.2019 10:43:44.677 WARN  [session] 
org.eclipse.jetty.server.session.UnreadableSessionDataException: Unreadable session node081m4ws7sira94t0oig5uhtjc14 for node0__0.0.0.0
	at org.eclipse.jetty.hazelcast.session.HazelcastSessionDataStore.doLoad(HazelcastSessionDataStore.java:69)
	at org.eclipse.jetty.server.session.AbstractSessionDataStore$1.run(AbstractSessionDataStore.java:101)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1418)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1456)
	at org.eclipse.jetty.server.session.SessionContext.run(SessionContext.java:96)
	at org.eclipse.jetty.server.session.AbstractSessionDataStore.load(AbstractSessionDataStore.java:110)
	at org.eclipse.jetty.server.session.AbstractSessionCache.loadSession(AbstractSessionCache.java:464)
	at org.eclipse.jetty.server.session.AbstractSessionCache.get(AbstractSessionCache.java:365)
	at org.eclipse.jetty.server.session.SessionHandler.getSession(SessionHandler.java:1000)
	at org.eclipse.jetty.server.session.SessionHandler.getHttpSession(SessionHandler.java:610)
	at org.eclipse.jetty.server.session.SessionHandler.checkRequestedSessionId(SessionHandler.java:1748)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1650)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.IOException: Not ClassLoadingObjectInputStream
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:63)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:197)
	at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1168)
	at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:117)
	at org.eclipse.jetty.hazelcast.session.HazelcastSessionDataStore.doLoad(HazelcastSessionDataStore.java:64)
	... 31 more
Caused by: java.io.IOException: Not ClassLoadingObjectInputStream
	at org.eclipse.jetty.server.session.SessionData.deserializeAttributes(SessionData.java:117)
	at org.eclipse.jetty.server.session.SessionData.readObject(SessionData.java:441)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:79)
	at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:72)
	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:191)
	... 34 more

[eicki]

I also checked 9.4.16, it is broken there as well.

[joakime]

Seems like a duplicate of #3175

[janbartel]

@eicki are you sure you have configured your Hazelcast instances on all your nodes to use the org.eclipse.jetty.hazelcast.session.SessionDataSerializer? I can see from your stacktrace that the default java serializer is being used:

at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:79)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:72)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:191)
... 34 more

[eicki]

Yes that is probably the issue, I'll try it out. Thank You.

[janbartel]

I'm closing this issue on the assumption that the correct serialization class wasn't configured. If that's not the cause, please reopen this issue and supply all relevant info.

[ghost]

@janbartel comment wasn't clear enough for our case. See my comment on #3175 if you use the custom client configuration xml.