filecoin-api storefront reads Blob bytes to derive Piece CID from anywhere #1349
Assignees
Comments
10 tasks
4 tasks
8 tasks
vasco-santos
changed the title
vasco-santos
changed the title
vasco-santos
added a commit
that referenced
this issue
Apr 26, 2024
With the aim to facilitate #1349 storefront `dataStore` is renamed to `contentStore` (note alignment of name with the `content` property in Filecoin Pipeline capabilities/message namings). My goal is to make `contentStore.stream()` perform HTTP Request to read thing from Roundabout (if `filecoin/offer` comes with `content` CARCID code, it will try to get a CAR, if `content` comes as RAW, it will try to get RAW). The main point in this change is to not require `contentStore` to have a `put` method, like `dataStore` was requiring just for testing. It extracts that requirement only to run tests, so that we can implement a `contentStore` as something with just a `stream` function that behind the scenes can perform a HTTP Request to Roundabout instead of being an abstraction on top of a S3 bucket like today. Note that this is not strictly needed, we could just use this as is and in `dataStore` implementation and make `put` https://github.com/w3s-project/w3infra/blob/main/filecoin/store/data.js#L39 `throw new Error('not implemented')` and just use an extended `DataStore` class for w3infra testing. But I feel this way is nicer, and also like more new naming as it aligns with everything including content name defined in w3index https://github.com/w3s-project/specs/blob/main/w3-index.md#sharded-dag-index-example BREAKING CHANGE: dataStore in storefront renamed to contentStore
|
FYI I am currently working on this one |
vasco-santos
added a commit
to storacha/w3infra
that referenced
this issue
Apr 30, 2024
) Today Storefront relies directly on a bucket (in S3) to read bytes to derive PieceCID for validation. With blob protocol, we are moving to write to R2 only 🙌🏼 therefore, we aim to make filecoin Storefront able to read from anywhere, but also not increase the infra costs with this change. This PR changes Storefront to rely on Roundabout to read Blob bytes to derive Piece CID from anywhere instead of `contentStore` being directly hooked to a bucket. See [usage](https://github.com/w3s-project/w3up/blob/main/packages/filecoin-api/src/storefront/events.js#L43) Relying on Roundabout introduces a layer of indirection, but in current implementation where this validation runs in AWS and content will likely be read from CF, makes it cost optimal to read from Roundabout. Closes storacha/w3up#1349
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Context
filecoin-api, more specifically Storefront relies on adatastoreto read bytes of some content previously offered, in order to validate that derived PieceCID, matches the offered PieceCID.Today this is assumed to be
carpark-prod-0bucket in S3, where this content is assumed to be stored. We are moving toward getting rid of this assumption.There are two potential solutions:
Relying on Roundabout introduces a layer of indirection, but in current implementation where this validation runs in AWS and content will likely be read from CF, makes it cost optimal to read from Roundabout.
Task
Storefront relies on Roundabout to read Blob bytes to derive Piece CID from anywhere.
See #1332 (comment)
The text was updated successfully, but these errors were encountered: