perf: exporter concurrent blockstore get

[alanshaw]

Here's a super quick hack I did to prove the usefulness of #236

In this PR we just concurrently fetch ALL the blocks from the blockstore if the requested byte range is the entirety of the file.

This is likely not what you want to merge, but I just wanted to see something similar working in practice.

In the 1st video, bitswap is having to send a message with a single block each time. We get ~40 blocks in ~30s:

before.mov

In the 2nd video there's a delay before receiving blocks but then we get 172 in rapid succession after ~20s. I think the delay before receiving blocks in the second video is the server side fetching the blocks from it's blockstore (clearly not very perfromantly!):

after.mov

[alanshaw]

P.S. I was not able to npm install in this repo after a fresh clone or run any tests...

[alanshaw]

I have a slightly better version of this in the fork here: https://github.com/web3-storage/fast-unixfs-exporter

Notable changes:

1. Adds blockReadConcurrency option (defaults to Infinity)
2. Wraps blockstore.get with p-queue when concurrency is less than Infinity
3. Requests all blocks for a given node from the blockstore in parallel, yielding blocks in the expected order as they arrive.

[achingbrain]

In the 2nd video there's a delay before receiving blocks

If the code behind the video is the code in this PR, the delay might be due to the use of Promise.all here - it'll wait to load all 174 blocks before processing any of them - looks like you fixed that up in your fork though 👍

I've tidied up the idea here in #249 - it doesn't have a concurrency parameter right now - trivial to add but it's also limited by the number of DAGLinks a given DAGNode can have so maybe it's not necessary?

---

Something I wanted to explore with this sort of optimisation is to switch it around to use getMany (though I think the return type needs to change to AsyncIterable<Pair<Key, Value>> first) - that would simplify the code here and shift the concurrency control to the blockstore, so for example if your blockstore is talking to a rate-limited API you might want to limit overall concurrency there, or not at all if it's in-memory.

[achingbrain]

Closing in favour of #249 - thanks for opening this!