RESTEasy Reactive REST client causes SocketException minutes after AWS lambda has finished

[gefloh]

Hello,

currently I am migrating several components from RESTEasy Classic to Reactive (just switching the dependency at first, not changing the way REST is used). In one project which is an AWS lambda and doesn't have own REST resources but uses the reactive REST client, I am seeing a strange behavior which causes me some headache. After roughly 6 minutes and after the lambda has long finished running, Quarkus seems to continue using the former connection which causes then a SocketException. Enabling detailed logs for the REST client shows:

2023-02-02T14:59:33.362+01:00 Duration: x ms	Billed Duration: x ms	Memory Size: x MB	Max Memory Used: x MB	Init Duration: x ms	
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 - R:<host>/<ip>:443] READ COMPLETE
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 - R:<host>/<ip>:443] EXCEPTION: java.net.SocketException: Connection reset
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 - R:<host>/<ip>:443] CLOSE
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 ! R:<host>/<ip>:443] USER_EVENT: io.netty.channel.socket.ChannelInputShutdownReadComplete@2bb1f694
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 ! R:<host>/<ip>:443] USER_EVENT: SslCloseCompletionEvent(java.nio.channels.ClosedChannelException)
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 ! R:<host>/<ip>:443] INACTIVE
2023-02-02T15:05:20.705+01:00 [id: 0x19ba1ce9, L:/<ip>:48654 ! R:<host>/<ip>:443] UNREGISTERED

This seems to happen on the I/O thread:

"threadName": "vert.x-eventloop-thread-1"

Does anyone know what is causing this? What can I do to avoid this?

Thanks,
Gerhard

[quarkus-bot[bot]]

/cc @FroMage (resteasy-reactive), @Sgitario (rest-client,resteasy-reactive), @cescoffier (rest-client), @geoand (rest-client,resteasy-reactive), @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda), @stuartwdouglas (resteasy-reactive)

[geoand]

@galderz @patriot1burke do you have any idea why the shutdown handlers would not work properly for AWS Lambda in native mode?

[galderz]

@geoand Well, the implementation is completely different due to substrate VM so maybe there's a bug in there. Nothing rings a bell right now. @gefloh can you provide a reproducer that we can run? Or if someone understands how AWS does shutdown, something can be emulated?

[gefloh]

@galderz Well, I probably can create a small Lambda for reproducing that. The thing is: I couldn't simulate this problem locally yet, which means, you would have to upload it to AWS to run it. I am not sure if that helps then.

[stuartwdouglas]

So the exception is basically on the client side, when AWS is shutting down the client lambda container?

I would guess there is some difference in how the close is being reported, e.g. maybe in JVM mode the socket just returns -1 while in native it throws an exception.

[gefloh]

Yes, although in the meantime I tend to give up on this and go with the workaround of disabling the keep alive completely. I tried to create a reproducer but found that with a basic Lambda everything is working perfectly. Then I added different dependencies that are used by our Lambdas and the error was there, so it seems one of the dependencies causes this behavior.
Still, searching for the problem is extremely time consuming because:

• Native compilation is needed, which takes times
• I have to run the Lambda and wait for the shutdown through AWS which takes several minutes
• AND the issue does not occur on every run, sometimes the Lambda has to be run several times, so probably it's also an issue with the async nature of Vert.x.

Currently it looks more reasonable to use the workaround and try later again with the keep alive when "something" (tm) has changed in the dependencies.

[gefloh]

Workaround to avoid the exception (see discussion above):

final HttpClientOptions options = new HttpClientOptions();
options.setKeepAlive(false);

Thanks @cescoffier for the hint with keep-alive, thanks @geoand.

[geoand]

Interesting enough, we don't see to have a way to turn off Vertx's keep-alive for the Reactive REST Client... @TomasHofman would you like to add that?

[TomasHofman]

I will take a look, but it will probably take me a week or two before I have time. So if anybody wants to do it sooner that's fine ;).

[geoand]

No rush 🙂

[gefloh]

I could also have a look and add this. It doesn't sound too complicated. ;-)

[TomasHofman]

Sure, go for it.