-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add graceful streaming pull shutdown #292
Conversation
This internal flag has been hardcoded to True for several years and we do not utilize the "False" case anywhere, nor do we mention it in the docs or expose it to the end users.
The leaser thread should not terminate implicitly when the streaming pull manager's consumer thread becomes inactive, as there might still be messages being processed inside the scheduler and we want to keep extending these messages' ACK deadlines until the scheduler has been shut down. NOTE: The manager's streaming pull RPC does not need to be active, since all mod-ACK requests are sent using a separate unary request instead of over the stream.
The library only supports Python 3.6+, thus we don't need the conditional anymore.
For consistency with the leaser, the heartbeater should not terminate implicitly when the manager stops the background stream, but should instead wait for the manager to stop it explicitly.
After the streaming pull manager shuts down the consumer thread and becomes "inactive", there might still be requests waiting in the queue to be dispatched, thus the dispatcher should not implicitly enter the no-op mode of operation.
It appears that ACK requests sent after the shutdown somehow get lost, even though the dispatcher processes them and sends them through the streaming pull manager's ( Update: This was because all messages were published in the same batch, meaning that all of them had to be ACK-ed for the ACKs to persist. It's the backend behavior that will be changed, but not in the immediate future. |
This trickled down to |
This reverts commit 00874fe.
Closes #276.
This PR implements an optional blocking streaming pull shutdown. "Blocking" means blocking on
streaming_future.cancel()
until all currently executing user callbacks complete, as well as dispatching any message ACK or NACK requests these callbacks produce.The PR also makes sure that any received messages sitting in internal queues - but not yet dispatched to the callbacks - are automatically NACK-ed on shutdown to speed up their re-delivery (the backend is not left waiting for their ACK deadlines to expire).
PR checklist: