mb makehashes crash (because of zsync feature?)

[lrupp]

From poeml#176

Activating zsync in the Mirrorbrain configuration and run the following to get the zsync "file" done:
mb makehashes /var/www/download.kiwix.org/ -v --force -t /usr/share/mirrorbrain
crashes.

Moving the chunk size to 4MB works partly, but mb makehashes still crashes with a file of 200GB.

[andrii-suse]

The error looks like pure memory allocation problem: I experimented with 100G file : the zsums column needs to store 236M - it is passed to postgres as hex, so around 500M.
With 200G file it will try to pass ~1G hex string, so can definitely meet the error from the ticket.

Proper solution may be trying to update zsums in batches instead of one go and probably pass binary data directly to postgres instead of hex. Using much bigger block size is an option as well - current 4K is way too small for 200G files.

[darix]

maybe we should just introduce a size limit for such things.

[mlschroe]

I hope that you're aware that libzypp does not use the zsync column? Why would you want to enable it for opensuse?

[andrii-suse]

Why would you want to enable it for opensuse?

I don't. opensuse branch is where all changes go in at the moment

[darix]

master branch stays in the state of peter's original code.