Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support range requests #48

Merged
merged 1 commit into from
Jan 18, 2024
Merged

Support range requests #48

merged 1 commit into from
Jan 18, 2024

Conversation

hannahhoward
Copy link
Contributor

@hannahhoward hannahhoward commented Jan 9, 2024

Goals

Support range requests in the client. For now, only supports one HTTP range (FWIW, the HTTP range spec allows multiple ranges to be specified -- I don't think that gets used for video and audio)

Implementation

So it turns out ipfs-unixfs-exporter already has range support of a kind -- when you call node.content() you can pass ExporterOptions object with and offset and length, and it will properly return the right data, loading the correct blocks. (I think)

What is needed then is just to do the right parsing -- http ranges are pretty weird -- they support negatives and also missing end values.

This adds a range property to the FetchOptions that has a rangeStart and rangeEnd parameter, which should correspond to the integer values in the range parameter (i.e. 'bytes=0-1000' becomes {rangeStart:0, rangeEnd: 1000})

There are two things to do with this parameter

  1. pass it to the L1's as entity bytes -- so you get the right blocks in the response.
  2. Convert to offset/length params on unixfs-exporter to do ranged verification

Next steps

I've made a similar change to actually read the range header in the service worker (PR incoming).

This should be a complete solution from a raw implementation standpoint, but it needs further testing. There are unit tests of the changes, but little integration testing.

In particular, the UnixFS exporter range requests are tested against a single block file. We need to make a test that verifies it works with a multi-block file.

We also probably need a round trip integration test that verifies we can make a request (fetchContent or fetchContentWithFallback) with the range option set, and the expected URL will go to the backend, and given the proper car response, it verifies.

The good news is I believe, at least tentatively, that we can make this all work without getting into unixfs-exporter code.

- pass entity-bytes param to the backend
- utilize offset/length params on unixfs-exporter to do ranged
  verification
@guanzo guanzo force-pushed the feat/js-client-range-request-spike branch from 000a266 to 025fa94 Compare January 18, 2024 23:49
@guanzo guanzo merged commit ab8c510 into main Jan 18, 2024
1 check passed
@guanzo guanzo deleted the feat/js-client-range-request-spike branch January 18, 2024 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants