Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v23.3.x] CORE-1722: compression: Use preallocated decompression buffers for lz4 #17825

Merged
merged 6 commits into from
Apr 15, 2024

Conversation

abhijat
Copy link
Contributor

@abhijat abhijat commented Apr 12, 2024

Manual backport with several changes to use appropriate header paths etc.

FIXES #17815

Backport of #17385

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x

Release Notes

  • none

abhijat added 6 commits April 12, 2024 11:28
A configuration setting is added to disable lz4 buffer preallocation.

(cherry picked from commit 3ec7f4e)
An object which holds a pair of preallocated buffers is added. The
buffers are char arrays allocated at startup. State of allocation is
maintained along with the buffers, to ensure correctness when using
these buffers with malloc and free calls. The c api is introduced in the
next commit.

(cherry picked from commit cf44e60)
The custom allocator wraps the managed buffers into a c struct which is
used by lz4 to route malloc and free calls.

If the feature is disabled the struct is constructed such that calls are
directly sent to malloc and free.

(cherry picked from commit 9116875)
An optional block size parameter is added so that the compression op can
set large block size, this enables testing the newly added preallocated
buffers for large block size.

(cherry picked from commit 5b3d7a3)
@abhijat
Copy link
Contributor Author

abhijat commented Apr 12, 2024

CI failure #12897

@piyushredpanda piyushredpanda added this to the v23.2.28 milestone Apr 12, 2024
@piyushredpanda piyushredpanda requested a review from Lazin April 15, 2024 12:08
@abhijat abhijat merged commit da7c1f2 into redpanda-data:v23.3.x Apr 15, 2024
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants