Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: blob implementation #1342

Merged
merged 27 commits into from
Apr 12, 2024
Merged

feat: blob implementation #1342

merged 27 commits into from
Apr 12, 2024

Conversation

vasco-santos
Copy link
Contributor

@vasco-santos vasco-santos commented Mar 25, 2024 •
edited
Loading

Adds implementation of blob/*, web3.storage/* and ucan/conclude handlers and capabilities.

Also upgrades to latest ucanto all over the place

@vasco-santos vasco-santos force-pushed the feat/blob-implementation branch 2 times, most recently from d3c09d2 to 7660794 Compare March 25, 2024 12:19
@vasco-santos vasco-santos mentioned this pull request Mar 25, 2024
Closed
@vasco-santos vasco-santos mentioned this pull request Mar 25, 2024
Open
@vasco-santos vasco-santos force-pushed the feat/blob-implementation branch 3 times, most recently from 472a097 to 9624c8a Compare March 27, 2024 12:08
@vasco-santos vasco-santos mentioned this pull request Mar 27, 2024
Closed
10 tasks
@vasco-santos vasco-santos force-pushed the feat/blob-implementation branch 10 times, most recently from 6178240 to a78c3ea Compare April 3, 2024 14:25
@vasco-santos vasco-santos force-pushed the feat/blob-implementation branch 3 times, most recently from a36b64d to 5c6b000 Compare April 4, 2024 19:13
Gozala
Gozala reviewed Apr 5, 2024
View reviewed changes
Copy link
Contributor

@Gozala Gozala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an amazing progress, I'm blown away. I did provide some feedback here and there only thing I'd call out is that we really should capture dependencies across tasks and we'd have to define those in capability schemas also otherwise they will be ignored during encode.

Other than that I was also thinking when writing spec that it would be a good idea to have separate internal invocations when space usage is increased / decreased so that billing could more simply tap into that without having to make guesses if allocate resulted in more space use or not. Relatedly there had been various other cases where we want to do batch operations that billing would have trouble with, however it could be also addressed if we had simple internal ops that increase / decreate usage. This is definitely something we need @alanshaw feedback on

packages/capabilities/src/blob.js Outdated Show resolved Hide resolved
packages/capabilities/src/blob.js Outdated Show resolved Hide resolved
packages/capabilities/src/types.ts Outdated Show resolved Hide resolved
packages/capabilities/src/types.ts Outdated Show resolved Hide resolved
packages/capabilities/src/types.ts Outdated Show resolved Hide resolved
packages/upload-api/src/blob/allocate.js Outdated
Comment on lines 84 to 99
if (hasBlob.ok) {
return {
ok: { size: blob.size },
}
}

return {
ok: {
size: blob.size,
address: {
url: createUploadUrl.ok.url.toString(),
headers: createUploadUrl.ok.headers,
},
},
}
})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would expect allocation to produce same result regardless of whether we have a blob or not, it's just if we do have a blob already we would succeed put without awaiting on user to perform upload and consequently succeed blob/add as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am following the spec, which makes address optional for this reason? https://github.com/web3-storage/specs/blob/feat/http-put/w3-blob.md#allocate-blob-receipt-schema

In the meantime, latest implementation of blob dd should actually do what you say

packages/upload-api/src/blob/lib.js Outdated Show resolved Hide resolved
packages/upload-api/src/blob/lib.js Outdated Show resolved Hide resolved
packages/upload-api/src/blob/list.js Outdated Show resolved Hide resolved
packages/upload-api/src/types/blob.ts Show resolved Hide resolved
@vasco-santos vasco-santos force-pushed the feat/blob-implementation branch 3 times, most recently from 6f1b921 to 0668957 Compare April 5, 2024 16:08
@vasco-santos vasco-santos marked this pull request as ready for review April 5, 2024 16:51
@vasco-santos
chore: add location claim
8931124
@vasco-santos
chore: address review
9fac96c
@vasco-santos
chore: address review
730b601
@vasco-santos
chore: add await error
92500c6
@vasco-santos
chore: update based on storacha/specs#117
c88f634
@vasco-santos
chore: address review
5a49c72
@vasco-santos
fix: derive blob provider from last 32 bytes from content
6579dde
@vasco-santos
fix: try to fix tests
7618768
@vasco-santos
chore: address review
4de3b96
@vasco-santos
chore: clean up
e58e864
@vasco-santos
fix: adopt fully ucan await pipeline
5adcb0d
@vasco-santos
fix: add expiration to allocate receipt and wire up space for accept
6f2bf8f
@vasco-santos
chore: re-enable attw
dc5e821
@vasco-santos
chore: rename exp to ttl and make optional for blob accept
0cc774a
@vasco-santos
fix: address review comments for capabilities and types
a40e9f2
@vasco-santos
fix: address review comments for upload api
e5e60d5
@vasco-santos
refactor: make schedule functions the same as invocation creation
f32d7ab
@vasco-santos
test: simplify and add tests
b91dd04
@vasco-santos
test: add all storage tests
8b0d4ab
@vasco-santos
chore: address last review comments
4cf99fa
@vasco-santos
fix: remove unused dep
15d732c
@vasco-santos vasco-santos merged commit 00735a8 into main Apr 12, 2024
14 checks passed
@vasco-santos vasco-santos deleted the feat/blob-implementation branch April 12, 2024 08:53
This was referenced Apr 12, 2024
Merged
Merged
vasco-santos pushed a commit that referenced this pull request Apr 12, 2024
@it-dag-house
chore(main): release capabilities 13.3.0 (#1366)
d6fbc4a 
🤖 I have created a release *beep* *boop*
---


##
[13.3.0](capabilities-v13.2.1...capabilities-v13.3.0)
(2024-04-12)


### Features

* blob, web3.storage and ucan conclude capabilities together with api
handlers ([#1342](#1342))
([00735a8](00735a8))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
vasco-santos added a commit that referenced this pull request Apr 12, 2024
@it-dag-house @vasco-santos
chore(main): release upload-api 9.1.0 (#1367)
8721d93 
🤖 I have created a release *beep* *boop*
---


##
[9.1.0](upload-api-v9.0.1...upload-api-v9.1.0)
(2024-04-12)


### Features

* blob, web3.storage and ucan conclude capabilities together with api
handlers ([#1342](#1342))
([00735a8](00735a8))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: Vasco Santos <santos.vasco10@gmail.com>
@reidlw reidlw mentioned this pull request Apr 19, 2024
Open
alanshaw pushed a commit to storacha/upload-service that referenced this pull request Nov 5, 2024
@github-actions
chore(main): release capabilities 1.0.0 (#16)
1c0902d 
🤖 I have created a release *beep* *boop*
---


## 1.0.0 (2024-11-05)


### ⚠ BREAKING CHANGES

* add `index/add` handler
([storacha#1421](https://github.com/storacha/upload-service/issues/1421))
* restrict store API to CARs
([storacha#1415](https://github.com/storacha/upload-service/issues/1415))
* **capabilities:** `BlobMultihash` type in `@web3-storage/capabilities`
renamed to `Multihash`.
* allocations storage interface now requires remove to be implemented
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))
* coupon
([storacha#1136](https://github.com/storacha/upload-service/issues/1136))
* see latest specs
https://github.com/web3-storage/specs/blob/cbdb706f18567900c5c24d7fb16ccbaf93d0d023/w3-filecoin.md
* filecoin client to use new capabilities
* filecoin capabilities

### refactor

* filecoin api services events and tests
([storacha#974](https://github.com/storacha/upload-service/issues/974))
([953537b](953537b))
* filecoin capabilities
([c0b97bf](c0b97bf))
* filecoin client to use new capabilities
([b0d9c3f](b0d9c3f))


### Features

* add "plan/create-admin-session" capability
([storacha#1411](https://github.com/storacha/upload-service/issues/1411))
([50eeeb5](50eeeb5))
* add `index/add` handler
([storacha#1421](https://github.com/storacha/upload-service/issues/1421))
([cbe9524](cbe9524))
* add `initialize` method to `PlansStorage`
([storacha#1278](https://github.com/storacha/upload-service/issues/1278))
([6792126](6792126))
* add `store/get` and `upload/get` capabilities
([storacha#942](https://github.com/storacha/upload-service/issues/942))
([40c79eb](40c79eb))
* add `subscription/list` capability
([storacha#1088](https://github.com/storacha/upload-service/issues/1088))
([471d7e5](471d7e5))
* add a function to verify and return Abilities.
([storacha#1252](https://github.com/storacha/upload-service/issues/1252))
([2f026a2](2f026a2))
* add blob list and remove
([storacha#1385](https://github.com/storacha/upload-service/issues/1385))
([2f69946](2f69946))
* add blob protocol to upload-client
([storacha#1425](https://github.com/storacha/upload-service/issues/1425))
([49aef56](49aef56))
* add blob/get
([storacha#1484](https://github.com/storacha/upload-service/issues/1484))
([328039d](328039d))
* add revocation to access-client and w3up-client
([storacha#975](https://github.com/storacha/upload-service/issues/975))
([6c877aa](6c877aa))
* add usage/report capability
([storacha#1079](https://github.com/storacha/upload-service/issues/1079))
([6418b4b](6418b4b))
* blob, web3.storage and ucan conclude capabilities together with api
handlers
([storacha#1342](https://github.com/storacha/upload-service/issues/1342))
([00735a8](00735a8))
* **capabilities:** add `index/add` capability
([storacha#1410](https://github.com/storacha/upload-service/issues/1410))
([1b71b89](1b71b89))
* change `plan/update` to `plan/set` and use existing `PlansStorage#set`
to implement an invocation handler
([storacha#1258](https://github.com/storacha/upload-service/issues/1258))
([1ccbfe9](1ccbfe9))
* coupon
([storacha#1136](https://github.com/storacha/upload-service/issues/1136))
([1b94f2d](1b94f2d))
* filecoin info
([storacha#1091](https://github.com/storacha/upload-service/issues/1091))
([adb2442](adb2442))
* Generate Space proofs on the fly, on `access/claim`
([storacha#1555](https://github.com/storacha/upload-service/issues/1555))
([9e2b1d4](9e2b1d4))
* implement `plan/get` capability
([storacha#1005](https://github.com/storacha/upload-service/issues/1005))
([f0456d2](f0456d2))
* introduce capability for changing billing plan
([storacha#1253](https://github.com/storacha/upload-service/issues/1253))
([d33b3a9](d33b3a9))
* move aggregate information out of deals in filecoin/info
([storacha#1192](https://github.com/storacha/upload-service/issues/1192))
([18dc590](18dc590))
* publish index claim
([storacha#1487](https://github.com/storacha/upload-service/issues/1487))
([237b0c6](237b0c6))
* restrict store API to CARs
([storacha#1415](https://github.com/storacha/upload-service/issues/1415))
([e53aa87](e53aa87))
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))
([5d52e44](5d52e44))
* router ([#11](#11))
([c810735](c810735))
* upgrade ucanto/transport to 9.1.0 in all packages to get more verbose
errors from HTTP transport on non-ok response
([storacha#1312](https://github.com/storacha/upload-service/issues/1312))
([d6978d7](d6978d7))
* usage/record capability definition
([storacha#1562](https://github.com/storacha/upload-service/issues/1562))
([98c8a87](98c8a87))
* wip router
([ffcd9c7](ffcd9c7))


### Fixes

* add missing ContentNotFound definition for filecoin offer failure
([c0b97bf](c0b97bf))
* add missing filecoin submit success and failure types
([c0b97bf](c0b97bf))
* capabilities should export blob caps
([storacha#1376](https://github.com/storacha/upload-service/issues/1376))
([460729e](460729e))
* client tests
([b0d9c3f](b0d9c3f))
* fix arethetypesworking errors in all packages
([storacha#1004](https://github.com/storacha/upload-service/issues/1004))
([2e2936a](2e2936a))
* issue where typedoc docs would only show full docs for w3up-client
([storacha#1141](https://github.com/storacha/upload-service/issues/1141))
([0b8d3f3](0b8d3f3))
* migrate repo
([storacha#1389](https://github.com/storacha/upload-service/issues/1389))
([475a287](475a287))
* one more tweak to the `PlanStorage` interface
([storacha#1280](https://github.com/storacha/upload-service/issues/1280))
([5a44565](5a44565))
* package metadata
([storacha#1161](https://github.com/storacha/upload-service/issues/1161))
([b8a1cc2](b8a1cc2))
* put access.session back
([storacha#1100](https://github.com/storacha/upload-service/issues/1100))
([10a1a4b](10a1a4b))
* rename blob and index client capabilities
([storacha#1478](https://github.com/storacha/upload-service/issues/1478))
([17e3a31](17e3a31))
* repo URLs
([storacha#1550](https://github.com/storacha/upload-service/issues/1550))
([e02ddf3](e02ddf3))
* tests
([b179910](b179910))
* trigger capabilities release
([storacha#1399](https://github.com/storacha/upload-service/issues/1399))
([7d9ab35](7d9ab35))
* type errors
([c0b97bf](c0b97bf))
* update data-segment dep
([228ff79](228ff79))
* upgrade @ucanto/validator with bugfix
([storacha#1151](https://github.com/storacha/upload-service/issues/1151))
([d4e961b](d4e961b))
* upgrade ucanto core
([storacha#1127](https://github.com/storacha/upload-service/issues/1127))
([5ce4d22](5ce4d22))
* upgrade ucanto in filecoin api
([c95fb54](c95fb54))
* upgrade ucanto libs and format filecoin api
([storacha#1359](https://github.com/storacha/upload-service/issues/1359))
([87ca098](87ca098))
* upload API test fixes
([6b0d72d](6b0d72d))


### Other Changes

* Add `pnpm dev` to watch-build all packages
([storacha#1533](https://github.com/storacha/upload-service/issues/1533))
([07970ef](07970ef))
* **main:** release capabilities 10.1.0
([storacha#979](https://github.com/storacha/upload-service/issues/979))
([bdd3970](bdd3970))
* **main:** release capabilities 10.2.0
([storacha#983](https://github.com/storacha/upload-service/issues/983))
([a906488](a906488))
* **main:** release capabilities 11.0.0
([storacha#994](https://github.com/storacha/upload-service/issues/994))
([76b0489](76b0489))
* **main:** release capabilities 11.0.1
([storacha#1008](https://github.com/storacha/upload-service/issues/1008))
([37cdc5a](37cdc5a))
* **main:** release capabilities 11.1.0
([storacha#1026](https://github.com/storacha/upload-service/issues/1026))
([7fdb541](7fdb541))
* **main:** release capabilities 11.2.0
([storacha#1084](https://github.com/storacha/upload-service/issues/1084))
([0e7b6dc](0e7b6dc))
* **main:** release capabilities 11.3.0
([storacha#1098](https://github.com/storacha/upload-service/issues/1098))
([7d671bd](7d671bd))
* **main:** release capabilities 11.3.1
([storacha#1101](https://github.com/storacha/upload-service/issues/1101))
([20b5b35](20b5b35))
* **main:** release capabilities 11.4.0
([storacha#1105](https://github.com/storacha/upload-service/issues/1105))
([1b6210f](1b6210f))
* **main:** release capabilities 11.4.1
([storacha#1131](https://github.com/storacha/upload-service/issues/1131))
([a5b7154](a5b7154))
* **main:** release capabilities 12.0.0
([storacha#1137](https://github.com/storacha/upload-service/issues/1137))
([bb23e9f](bb23e9f))
* **main:** release capabilities 12.0.1
([storacha#1147](https://github.com/storacha/upload-service/issues/1147))
([d1a9c78](d1a9c78))
* **main:** release capabilities 12.0.2
([storacha#1152](https://github.com/storacha/upload-service/issues/1152))
([b9d7ff5](b9d7ff5))
* **main:** release capabilities 12.0.3
([storacha#1163](https://github.com/storacha/upload-service/issues/1163))
([ec5c385](ec5c385))
* **main:** release capabilities 12.1.0
([storacha#1195](https://github.com/storacha/upload-service/issues/1195))
([a21c1a5](a21c1a5))
* **main:** release capabilities 13.0.0
([storacha#1230](https://github.com/storacha/upload-service/issues/1230))
([3d5b3ef](3d5b3ef))
* **main:** release capabilities 13.1.0
([storacha#1257](https://github.com/storacha/upload-service/issues/1257))
([85adc9a](85adc9a))
* **main:** release capabilities 13.1.1
([storacha#1283](https://github.com/storacha/upload-service/issues/1283))
([31c38e9](31c38e9))
* **main:** release capabilities 13.2.0
([storacha#1315](https://github.com/storacha/upload-service/issues/1315))
([0505458](0505458))
* **main:** release capabilities 13.2.1
([storacha#1362](https://github.com/storacha/upload-service/issues/1362))
([26b5751](26b5751))
* **main:** release capabilities 13.3.0
([storacha#1366](https://github.com/storacha/upload-service/issues/1366))
([d6fbc4a](d6fbc4a))
* **main:** release capabilities 13.3.1
([storacha#1377](https://github.com/storacha/upload-service/issues/1377))
([149f592](149f592))
* **main:** release capabilities 14.0.0
([storacha#1386](https://github.com/storacha/upload-service/issues/1386))
([69bfc08](69bfc08))
* **main:** release capabilities 14.0.1
([storacha#1395](https://github.com/storacha/upload-service/issues/1395))
([a76c970](a76c970))
* **main:** release capabilities 14.0.2
([storacha#1400](https://github.com/storacha/upload-service/issues/1400))
([7b46852](7b46852))
* **main:** release capabilities 15.0.0
([storacha#1412](https://github.com/storacha/upload-service/issues/1412))
([ec90b81](ec90b81))
* **main:** release capabilities 16.0.0
([storacha#1419](https://github.com/storacha/upload-service/issues/1419))
([50e3934](50e3934))
* **main:** release capabilities 17.0.0
([storacha#1428](https://github.com/storacha/upload-service/issues/1428))
([6ee21fd](6ee21fd))
* **main:** release capabilities 17.1.0
([storacha#1447](https://github.com/storacha/upload-service/issues/1447))
([8ee1fd9](8ee1fd9))
* **main:** release capabilities 17.1.1
([storacha#1483](https://github.com/storacha/upload-service/issues/1483))
([5394ed5](5394ed5))
* **main:** release capabilities 17.2.0
([storacha#1494](https://github.com/storacha/upload-service/issues/1494))
([99876a5](99876a5))
* **main:** release capabilities 17.3.0
([storacha#1503](https://github.com/storacha/upload-service/issues/1503))
([891c2a5](891c2a5))
* **main:** release capabilities 17.4.0
([storacha#1559](https://github.com/storacha/upload-service/issues/1559))
([feaea7a](feaea7a))
* no longer depends on hd-scripts, packages use/configure eslint
directly, fixes warnings from npm lint script
([storacha#1058](https://github.com/storacha/upload-service/issues/1058))
([ebdb99b](ebdb99b))
* package renames
([0f797ed](0f797ed))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alanshaw alanshaw alanshaw left review comments

@Gozala Gozala Gozala approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vasco-santos @Gozala @alanshaw