SQS long polling throws exceptions on shutdown

[zmaks]

Type: Bug

Component: SQS

Describe the bug

My app receives SQS messages via long polling using default SqsTemplate using the receive() method.
The problem is that when I shut down the app, there are bunch of errors in the logs.

There 3 errors like this since it makes retries:

java.util.concurrent.CompletionException: io.awspring.cloud.sqs.operations.MessagingOperationFailedException: Message receive operation failed for endpoint https://sqs.eu-west-1.amazonaws.com/000000000000/foo-bar-queue
	at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[na:na]
...
Caused by: io.awspring.cloud.sqs.operations.MessagingOperationFailedException: Message receive operation failed for endpoint https://sqs.eu-west-1.amazonaws.com/...
	at io.awspring.cloud.sqs.operations.AbstractMessagingTemplate.lambda$receiveManyAsync$5(AbstractMessagingTemplate.java:165) ~[spring-cloud-aws-sqs-3.0.1.jar:3.0.1]
...
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: executor not accepting a task
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[sdk-core-2.20.63.jar:na]
...
Caused by: java.lang.IllegalStateException: executor not accepting a task
	at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:61) ~[netty-resolver-4.1.89.Final.jar:4.1.89.Final]

And one more at the end:

2023-10-11T10:45:34.763+02:00 ERROR 87176 --- [tyEventLoop-2-0] i.n.u.c.D.rejectedExecution              : Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	...
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.channel.pool.SimpleChannelPool.notifyConnect(SimpleChannelPool.java:218) ~[netty-transport-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.channel.pool.SimpleChannelPool.access$000(SimpleChannelPool.java:43) ~[netty-transport-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.channel.pool.SimpleChannelPool$2.operationComplete(SimpleChannelPool.java:184) ~[netty-transport-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.channel.pool.SimpleChannelPool$2.operationComplete(SimpleChannelPool.java:181) ~[netty-transport-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89) ~[netty-transport-4.1.89.Final.jar:4.1.89.Final]
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:197) ~[netty-transport-4.1.89.Final.jar:4.1.89.Final]

How to handle these shutdown-related errors properly?

Spring Boot version: 3.0.4
AWS Spring version: 3.0.2

Sample
Here is the code that does the polling:

@Component
class SqsMessageReader(
    @Autowired private val sqsTemplate: SqsTemplate,
    @Value("\${custom.cloud.aws.sqs.queueUrl}") private val queue: String
) {

    fun read(): MyMessage? {
        return sqsTemplate.receive(queue, MyMessage::class.java).getOrNull()?.payload
    }
}

[tomazfernandes]

Hi @zmaks,

Since you're manually receiving messages using the SqsTemplate, it's up to you to make sure it's not stuck polling when you shutdown your application.

How to do that really depends on how you structure your app - for example, you can collect the CompletableFuture instances from polling async and join() them before allowing the app to complete shutdown.

Makes sense?

[zmaks]

Hi @tomazfernandes,

Thank you for the answer. On the other hand, the library provides an API: sqsTemplate.receive(queue, type) method which is sync and doesn't return a CompletableFuture or Future, so I can't control how it makes the request for this specific method. It does just does long polling with 10 sec timeout which throws exceptions if the app is shutdown. Probably, it would be nice to have an ability to setup SqsTemplate and provide some exception handler for such cases.

But I like your suggestion, I'll switch to async version of this method. Thanks!

[tomazfernandes]

Probably, it would be nice to have an ability to setup SqsTemplate and provide some exception handler for such cases.

We could add a structure for that, but since we propagate the error up to the caller, I'm not sure what actual value this would bring to the user. If it's just about the logs, we might as well have a flag to disable it.

Otherwise the behavior is pretty much the same as the AWS clients - if you use the blocking method and polling fails due to shutdown or anything else, it'll throw an exception just the same.

If you have any ideas let me know.

Thanks.