feat: add blockReadConcurrency option to exporter #361
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By default we attempt to load all sibilings in a given layer of a DAG at once to allow slow/async loading routines extra time to fetch data before it is needed. Some blockstores (e.g. CAR files) require the exporter to only request the next sequential CID in a DAG. Add a `blockReadConcurrency` option (named similarly to the importer's `blockWriteConcurrency` option) to control this behaviour. Fixes #359
|
cc @guanzo |
SgtPooki
approved these changes
Sep 20, 2023
rvagg
approved these changes
Sep 21, 2023
|
Does the concurrency option also need to be set in these 2 places? |
…-concurrency-exporter-option
Co-authored-by: Rod Vagg <rod@vagg.org>
|
@guanzo yes, good spot - this has been updated and tests for this added. |
achingbrain
deleted the
feat/add-block-read-concurrency-exporter-option
branch
|
🎉 This PR is included in version 13.4.0 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 11.1.4 🎉 The release is available on npm package (@latest dist-tag) Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By default we attempt to load all sibilings in a given layer of a DAG at once to allow slow/async loading routines extra time to fetch data before it is needed.
Some blockstores (e.g. CAR files) require the exporter to only request the next sequential CID in a DAG.
Add a
blockReadConcurrencyoption (named similarly to the importer'sblockWriteConcurrencyoption) to control this behaviour.Fixes #359