Zstd tuning

[s0l0ist]

First, great work with the latest release. Using SEAL with the default zstd compression level dramatically increases performance; however I've noticed much higher levels of memory consumption during serialization.

I can choose the zlib override at runtime to avoid the memory issues, but of course the performance suffers.

Are there any plans for exposing zstd options at runtime?

[kimlaine]

That would certainly be possible. It's really easy now to add support for other compr_mode_types. For example, one could be zstd_mem_opt (memory optimized), and another could be zstd_perf_opt. Alternatively, maybe one particular setting is just much better (insignificant reduction in performance and significant improvement in memory cost) and we could only use that. I'm not familiar at all with tuning Zstandard, but if someone else is maybe they could recommend specific settings. In any case, I'll add this to the to-do list.

[kimlaine]

This is not actually about Zstandard; I looked into this more after another issue was reported. Turns out there was a bug in SEAL that caused a huge number of memory allocations of increasing size when using ZSTD compression.

We'll be releasing v3.6.1 shortly to fix this issue.

[s0l0ist]

Yes, that is exactly the same issue. Thanks @kimlaine!

[kimlaine]

3.6.1 is now available. Please let us know if there are further issues.