Don't raise exception when each message size is smaller than chunk_limit_size #3560

kenhys · 2021-11-22T07:16:50Z

Follow up #3553

In the previous versions, even though each record size is smaller
than chunk limit size, but whole message size exceeds
chunk limit size, BufferChunkOverflowError is raised unexpectedly.

For example, if chunk limit size is 1_280_000, process 3 event
stream (every 1_000_00 bytes), it throws an exception like this:

Fluent::Plugin::Buffer::BufferChunkOverflowError(<a 1000025 bytes
record (nth: 1) is larger than buffer chunk limit size, a 1000025
bytes record (nth: 2) is larger than buffer chunk limit size>)

Now changed not to raise exception if it's record size is smaller
enough than chunk limit size.

The idea is based on that adding byte size is smaller than chunk
limit size, chunk should be unstaged and pushed into queue,
If not, it should be skipped like #3553.

Signed-off-by: Kentaro Hayashi hayashi@clear-code.com

Which issue(s) this PR fixes:

Fixes #1849

What this PR does / why we need it:

Docs Changes:

Release Note:

kenhys · 2021-11-24T00:55:58Z

updates about error message is split into #3562

kenhys · 2021-11-24T02:06:14Z

This PR relies on adding_bytes, it might not be implemented 3rd party buffer plugin.

kenhys · 2021-11-24T04:16:32Z

rebased with the recent master.

kenhys · 2021-11-26T03:29:54Z

This PR will not be merged for v1.14.3.

cosmo0920 · 2021-11-26T03:31:16Z

This PR will not be merged for v1.14.3.

This PR needs to be more testing required?

ashie · 2021-11-26T03:38:38Z

Yes, we need further checking for this.
So we skip v1.14.3 to merge this.

lib/fluent/plugin/buffer.rb

kenhys · 2021-12-17T06:15:54Z

I've followed suggested approach (as before_bytesize is too general, so it was changed to represent what it is)

ashie · 2021-12-20T01:58:04Z

I'm still checking this, especially the difference of original_bytesize and committed_bytesize...

lib/fluent/plugin/buffer.rb

Follow up fluent#3553 In the previous versions, even though each record size is smaller than chunk limit size, but whole message size exceeds chunk limit size, BufferChunkOverflowError is raised unexpectedly. For example, if chunk limit size is 1_280_000, when processing 3 event stream (every 1_000_000 bytes), it throws an exception like this: Fluent::Plugin::Buffer::BufferChunkOverflowError(<a 1000025 bytes record (nth: 1) is larger than buffer chunk limit size, a 1000025 bytes record (nth: 2) is larger than buffer chunk limit size>) Now changed not to raise exception if it's record size is smaller enough than chunk limit size. Thus each message is stored into separated chunks. The idea is based on that adding byte size is smaller than chunk limit size, chunk should be unstaged and pushed into queue, If not, it should be skipped like fluent#3553. Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>

Checking original_bytesize isn't needed to detect a big record than chunk_limit_size. Signed-off-by: Takuro Ashie <ashie@clear-code.com>

ashie · 2021-12-23T08:57:59Z

Waiting CI...

daipom · 2024-03-19T02:02:02Z

lib/fluent/plugin/buffer.rb

+                      synchronize { @stage.delete(modified_metadata) }
+                      staged_chunk_used = false
+                      chunk.unstaged!
+                      break


Why does this unstage the current chunk?

kenhys force-pushed the fix-lower-chunk-overflow branch from 1d9716e to 080b198 Compare November 24, 2021 02:05

kenhys force-pushed the fix-lower-chunk-overflow branch 2 times, most recently from ce07f9f to 3f91ab4 Compare November 24, 2021 04:15

kenhys force-pushed the fix-lower-chunk-overflow branch from 3f91ab4 to b340d4e Compare November 24, 2021 04:19

kenhys marked this pull request as ready for review November 24, 2021 07:05

kenhys added this to the v1.15 milestone Dec 3, 2021

kenhys force-pushed the fix-lower-chunk-overflow branch from b340d4e to 0c57f88 Compare December 7, 2021 03:24

ashie reviewed Dec 16, 2021

View reviewed changes

lib/fluent/plugin/buffer.rb Outdated Show resolved Hide resolved

kenhys force-pushed the fix-lower-chunk-overflow branch from 0c57f88 to cca4f44 Compare December 17, 2021 06:13

ashie reviewed Dec 22, 2021

View reviewed changes

lib/fluent/plugin/buffer.rb Show resolved Hide resolved

kenhys and others added 2 commits December 23, 2021 16:14

buffer: Make clear the code for checking BufferChunkOverflow

5d93f36

Checking original_bytesize isn't needed to detect a big record than chunk_limit_size. Signed-off-by: Takuro Ashie <ashie@clear-code.com>

ashie force-pushed the fix-lower-chunk-overflow branch from cca4f44 to 5d93f36 Compare December 23, 2021 07:16

ashie approved these changes Dec 23, 2021

View reviewed changes

ashie merged commit 8229ed9 into fluent:master Dec 24, 2021

kenhys modified the milestones: v1.15, v1.14.4 Dec 24, 2021

ashie mentioned this pull request Jan 5, 2022

v1.14.4 #3589

Merged

kenhys deleted the fix-lower-chunk-overflow branch August 3, 2023 08:43

daipom reviewed Mar 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't raise exception when each message size is smaller than chunk_limit_size #3560

Don't raise exception when each message size is smaller than chunk_limit_size #3560

kenhys commented Nov 22, 2021

kenhys commented Nov 24, 2021

kenhys commented Nov 24, 2021

kenhys commented Nov 24, 2021

kenhys commented Nov 26, 2021

cosmo0920 commented Nov 26, 2021

ashie commented Nov 26, 2021 •

edited

Loading

kenhys commented Dec 17, 2021

ashie commented Dec 20, 2021

ashie commented Dec 23, 2021

daipom Mar 19, 2024

Don't raise exception when each message size is smaller than chunk_limit_size #3560

Don't raise exception when each message size is smaller than chunk_limit_size #3560

Conversation

kenhys commented Nov 22, 2021

kenhys commented Nov 24, 2021

kenhys commented Nov 24, 2021

kenhys commented Nov 24, 2021

kenhys commented Nov 26, 2021

cosmo0920 commented Nov 26, 2021

ashie commented Nov 26, 2021 • edited Loading

kenhys commented Dec 17, 2021

ashie commented Dec 20, 2021

ashie commented Dec 23, 2021

daipom Mar 19, 2024

Choose a reason for hiding this comment

ashie commented Nov 26, 2021 •

edited

Loading