Problem: production rocksdb configuration is not optimal

[yihuang]

Solution:

• update related dependencies to allow custom rocksdb options.
• using rocksdb v7, closes Problem: rocksdb compaction could cause write rate to drop to mimimum #759
• tune rocksdb options.

During the versiondb migration experiments, I found the default rocksdb settings are multiple times slower than the one I experimented, because I've tuned options and did a full compaction on the db before, but after I keep the node running with the normal binary for a while, suddenly my script's performance drop significantly, and the sst files are all rewritten by compaction process.
I suspect the main difference is sst file sizes, and probably filter and compression types as well, so I plan to tune the rocksdb options for default binary first, then try the versiondb migration script again, to see if it can get back to the old speed.

Compare Compression Options

• Tool: sst_dump --file=sample.sst --command=recompress ...
• Block size: 32k

Compression Type	Size	Time
Snappy	228861267	1.71s
LZ4	228341872	1.85s
ZSTD	217266835	3.93
ZSTD level 11	214701135	13.7s
ZSTD level 11 with 11k dictionary	186643202	20.6s
ZSTD level 11 with 110k dictionary	177305609	22.4s

Basically I picked relatively slower speed and higher compression ratio for the bottommost level, I think our node do have the capacity to occupy one core to do the heavier compaction gradually.

👮🏻👮🏻👮🏻 !!!! REFERENCE THE PROBLEM YOUR ARE SOLVING IN THE PR TITLE AND DESCRIBE YOUR SOLUTION HERE !!!! DO NOT FORGET !!!! 👮🏻👮🏻👮🏻

PR Checklist:

• Have you read the CONTRIBUTING.md?
• Does your PR follow the C4 patch requirements?
• Have you rebased your work on top of the latest master?
• Have you checked your code compiles? (make)
• Have you included tests for any non-trivial functionality?
• Have you checked your code passes the unit tests? (make test)
• Have you checked your code formatting is correct? (go fmt)
• Have you checked your basic code style is fine? (golangci-lint run)
• If you added any dependencies, have you checked they do not contain any known vulnerabilities? (go list -json -m all | nancy sleuth)
• If your changes affect the client infrastructure, have you run the integration test?
• If your changes affect public APIs, does your PR follow the C4 evolution of public contracts?
• If your code changes public APIs, have you incremented the crate version numbers and documented your changes in the CHANGELOG.md?
• If you are contributing for the first time, please read the agreement in CONTRIBUTING.md now and add a comment to this pull request stating that your PR is in accordance with the Developer's Certificate of Origin.

Thank you for your code, it's appreciated! :)

[mmsqe]

still get could not determine kind of name for C.rocksdb_lru_cache_options_set_num_shard_bits build error in grocksdb

[yihuang]

still get could not determine kind of name for C.rocksdb_lru_cache_options_set_num_shard_bits build error in grocksdb

try recent version of rocksdb, or using nix, the nix script use v7.8.3.

[yihuang]

@JayT106 I'll merge now to unblock some other fixes, feel free to further discuss the options picked here.

[JayT106]

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000
Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

[yihuang]

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000 Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

can you try SetCacheIndexAndFilterBlocks(true) to control ram usage during genesis export?
I think the reason open file limit affect ram usage is because the index/filter caches, CacheIndexAndFilterBlocks will manage those caches together with block cache.

[JayT106]

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000 Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

can you try SetCacheIndexAndFilterBlocks(true) to control ram usage during genesis export? I think the reason open file limit affect ram usage is because the index/filter caches, CacheIndexAndFilterBlocks will manage those caches together with block cache.

tried cronosd1.0.2-3-gca5e564 on the dev machine has 64GB RAM Recent datasets cronosmainnet_25-1-rocksdb-pruned.20230122.2040.tar.lz4, OOM. Will try the same code with limited SetMaxOpenFiles

[yihuang]

file descriptors should not be a problem, if ram usage is a concern we should enable SetCacheIndexAndFilterBlocks, and with bigger sst file size (SetTargetFileSizeMultiplier) and block size (32k), the total amount of filter and index should be reduced.

I remembered the value SetMaxOpenFiles was affecting the RAM usage a lot when exporting the genesis file. >64GB using when the value is 4096 and <32GB using when it is 1000 Maybe we need to evaluate it against the new settings. Before the genesis export feature is settled in the SDK, it will always be an issue.

can you try SetCacheIndexAndFilterBlocks(true) to control ram usage during genesis export? I think the reason open file limit affect ram usage is because the index/filter caches, CacheIndexAndFilterBlocks will manage those caches together with block cache.

tried cronosd1.0.2-3-gca5e564 on the dev machine has 64GB RAM Recent datasets cronosmainnet_25-1-rocksdb-pruned.20230122.2040.tar.lz4, OOM. Will try the same code with limited SetMaxOpenFiles

Can you try SetCacheIndexAndFilterBlocks(true)?

Ah ,right. Was thinking your changes already enabled this option. will do it.

[yihuang]

Is genesis export OOM or node syncing OOM? @borischeuk-crypto has tried on one archive node, didn't see much change in runtime statistics.

Tried SetCacheIndexAndFilterBlocks(true), the genesis export OOM.