fix(pubsub): include request overhead when computing publish batch size overflow #9911
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The maximum allowed size for a PublishRequest on the backend is lower than a mere sum of the byte sizes of individual messages. This commit adjusts the batch size overflow calculation to account for this overhead. It also caps the effective maximum BatchSetting.max_size value to 10_000_000 bytes (the limit on the backend). (credit also to GitHub @relud for outlining the main idea first in the issue description)
googlebot
added
the
cla: yes
software-dov
reviewed
Dec 3, 2019
software-dov
reviewed
Dec 3, 2019
software-dov
reviewed
Dec 4, 2019
|
@software-dov Thanks for the quick review, addressed the last outstanding comment. |
This was referenced Jan 29, 2020
[CHANGE ME] Re-generated container to pick up changes in the API or client library generator.
#10256
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #7108.
This PR fixes the logic that computes the publish batch size overflow, taking the total request message size overhead into account. The improved logic prevents the server-side errors simular to the following:
How to test
(see also the system test in this PR)
BatchSettings.max_bytessubstantially larger than 10_000_000, andBatchSettings.max_latencyto one second (so that the publish autocommit does not kick in too soon).Actual result (before the fix):
The backend responds with a "400 InvalidArgument" error.
Expected result (after the fix):
All messages are successfully published (the code splits them into multiple publish batches).
PR checklist