Improve get_block_by_seqno / lt / unix time speed #666
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is not for production, but for getting feedback.
The main problem is that the methods (
get_file_desc_by_seqno
/get_file_desc_by_unix_time
/get_file_desc_lt
) combine multithreaded queries into 1 thread, so they also go through the map every time, without using indexes. These methods degrade performance.This is my improvement for this problem, currently there is only test file, indexes and async+indexes examples with
get_block_by_seqno
, but if all 'ok', it can be used in all 3 methods.For test, I've requested blocks by seqno from MC and WC with seqno from 3 600 000 to 3 610 000.
Current realization:
It can be improved by precalculating min_seqno in each file to skip a lot of
td::map
inArchiveManager::load_package
, also there we can add calculation of minimal lt and unix time. If we add this min index and add skipping we got improvements:As you see masterchain requests of 10k blocks works x2 faster, while workchain request of 10k blocks speeds up from 27.9 sec to 8.4 sec. But, now we can add async worker to calculate
file_desc
in threads, it'll improve situation:8.9 sec -> 2.8 sec
27.9 sec -> 4.1 sec
This impact on all get block requests including liteserver, validator, indexers, etc.
I must admit that I'm not excellent at C++ and could have made a mistake, but our dton.io indexer now index old blocks much faster :)