Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: router #11

Merged
merged 12 commits into from
Nov 5, 2024
Merged

feat: router #11

merged 12 commits into from
Nov 5, 2024

Conversation

alanshaw
Copy link
Member

@alanshaw alanshaw commented Nov 1, 2024

This PR adds a router interface:

/**
 * The routing service is responsible for selecting storage nodes to allocate
 * blobs with.
 */
export interface RoutingService {
  /**
   * Selects a candidate for blob allocation from the current list of available
   * storage nodes.
   */
  selectStorageProvider(
    digest: MultihashDigest,
    size: number
  ): Promise<Result<Principal, CandidateUnavailable | Failure>>
  /**
   * Returns information required to make an invocation to the requested storage
   * node.
   */
  configureInvocation<C extends BlobAllocate | BlobAccept>(
    provider: Principal,
    capability: C,
    options?: Omit<UCANOptions, 'audience'>
  ): Promise<Result<Configuration<C>, ProofUnavailable | Failure>>
}

Tests implement a storage provider node and setup 2 nodes that are used in tests that the routing service picks randomly from. However the router will be sticky when asked to store the same blob multiple times. This allows existing tests for allocation and accept responses to pass. Note: we need to review if the semantics for allocate and accept responses really makes sense in a context where there are multiple storage providers (TLDR; the allocation size does not).

@alanshaw alanshaw marked this pull request as ready for review November 4, 2024 16:59
@alanshaw
Copy link
Member Author

alanshaw commented Nov 5, 2024

I'm merging this as I need to start work on upload-service-infra done before Devcon. However please feel free to leave review and I will address the feedback. Apologies this is a monster PR that touches many files.

@alanshaw alanshaw merged commit c810735 into main Nov 5, 2024
15 checks passed
@alanshaw alanshaw deleted the feat/router branch November 5, 2024 07:19
alanshaw pushed a commit that referenced this pull request Nov 5, 2024
🤖 I have created a release *beep* *boop*
---


## 1.0.0 (2024-11-05)


### ⚠ BREAKING CHANGES

* add `index/add` handler
([storacha#1421](https://github.com/storacha/upload-service/issues/1421))
* restrict store API to CARs
([storacha#1415](https://github.com/storacha/upload-service/issues/1415))
* **capabilities:** `BlobMultihash` type in `@web3-storage/capabilities`
renamed to `Multihash`.
* allocations storage interface now requires remove to be implemented
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))
* coupon
([storacha#1136](https://github.com/storacha/upload-service/issues/1136))
* see latest specs
https://github.com/web3-storage/specs/blob/cbdb706f18567900c5c24d7fb16ccbaf93d0d023/w3-filecoin.md
* filecoin client to use new capabilities
* filecoin capabilities

### refactor

* filecoin api services events and tests
([storacha#974](https://github.com/storacha/upload-service/issues/974))
([953537b](953537b))
* filecoin capabilities
([c0b97bf](c0b97bf))
* filecoin client to use new capabilities
([b0d9c3f](b0d9c3f))


### Features

* add "plan/create-admin-session" capability
([storacha#1411](https://github.com/storacha/upload-service/issues/1411))
([50eeeb5](50eeeb5))
* add `index/add` handler
([storacha#1421](https://github.com/storacha/upload-service/issues/1421))
([cbe9524](cbe9524))
* add `initialize` method to `PlansStorage`
([storacha#1278](https://github.com/storacha/upload-service/issues/1278))
([6792126](6792126))
* add `store/get` and `upload/get` capabilities
([storacha#942](https://github.com/storacha/upload-service/issues/942))
([40c79eb](40c79eb))
* add `subscription/list` capability
([storacha#1088](https://github.com/storacha/upload-service/issues/1088))
([471d7e5](471d7e5))
* add a function to verify and return Abilities.
([storacha#1252](https://github.com/storacha/upload-service/issues/1252))
([2f026a2](2f026a2))
* add blob list and remove
([storacha#1385](https://github.com/storacha/upload-service/issues/1385))
([2f69946](2f69946))
* add blob protocol to upload-client
([storacha#1425](https://github.com/storacha/upload-service/issues/1425))
([49aef56](49aef56))
* add blob/get
([storacha#1484](https://github.com/storacha/upload-service/issues/1484))
([328039d](328039d))
* add revocation to access-client and w3up-client
([storacha#975](https://github.com/storacha/upload-service/issues/975))
([6c877aa](6c877aa))
* add usage/report capability
([storacha#1079](https://github.com/storacha/upload-service/issues/1079))
([6418b4b](6418b4b))
* blob, web3.storage and ucan conclude capabilities together with api
handlers
([storacha#1342](https://github.com/storacha/upload-service/issues/1342))
([00735a8](00735a8))
* **capabilities:** add `index/add` capability
([storacha#1410](https://github.com/storacha/upload-service/issues/1410))
([1b71b89](1b71b89))
* change `plan/update` to `plan/set` and use existing `PlansStorage#set`
to implement an invocation handler
([storacha#1258](https://github.com/storacha/upload-service/issues/1258))
([1ccbfe9](1ccbfe9))
* coupon
([storacha#1136](https://github.com/storacha/upload-service/issues/1136))
([1b94f2d](1b94f2d))
* filecoin info
([storacha#1091](https://github.com/storacha/upload-service/issues/1091))
([adb2442](adb2442))
* Generate Space proofs on the fly, on `access/claim`
([storacha#1555](https://github.com/storacha/upload-service/issues/1555))
([9e2b1d4](9e2b1d4))
* implement `plan/get` capability
([storacha#1005](https://github.com/storacha/upload-service/issues/1005))
([f0456d2](f0456d2))
* introduce capability for changing billing plan
([storacha#1253](https://github.com/storacha/upload-service/issues/1253))
([d33b3a9](d33b3a9))
* move aggregate information out of deals in filecoin/info
([storacha#1192](https://github.com/storacha/upload-service/issues/1192))
([18dc590](18dc590))
* publish index claim
([storacha#1487](https://github.com/storacha/upload-service/issues/1487))
([237b0c6](237b0c6))
* restrict store API to CARs
([storacha#1415](https://github.com/storacha/upload-service/issues/1415))
([e53aa87](e53aa87))
* return allocated bytes in `store/add` receipt
([storacha#1213](https://github.com/storacha/upload-service/issues/1213))
([5d52e44](5d52e44))
* router ([#11](#11))
([c810735](c810735))
* upgrade ucanto/transport to 9.1.0 in all packages to get more verbose
errors from HTTP transport on non-ok response
([storacha#1312](https://github.com/storacha/upload-service/issues/1312))
([d6978d7](d6978d7))
* usage/record capability definition
([storacha#1562](https://github.com/storacha/upload-service/issues/1562))
([98c8a87](98c8a87))
* wip router
([ffcd9c7](ffcd9c7))


### Fixes

* add missing ContentNotFound definition for filecoin offer failure
([c0b97bf](c0b97bf))
* add missing filecoin submit success and failure types
([c0b97bf](c0b97bf))
* capabilities should export blob caps
([storacha#1376](https://github.com/storacha/upload-service/issues/1376))
([460729e](460729e))
* client tests
([b0d9c3f](b0d9c3f))
* fix arethetypesworking errors in all packages
([storacha#1004](https://github.com/storacha/upload-service/issues/1004))
([2e2936a](2e2936a))
* issue where typedoc docs would only show full docs for w3up-client
([storacha#1141](https://github.com/storacha/upload-service/issues/1141))
([0b8d3f3](0b8d3f3))
* migrate repo
([storacha#1389](https://github.com/storacha/upload-service/issues/1389))
([475a287](475a287))
* one more tweak to the `PlanStorage` interface
([storacha#1280](https://github.com/storacha/upload-service/issues/1280))
([5a44565](5a44565))
* package metadata
([storacha#1161](https://github.com/storacha/upload-service/issues/1161))
([b8a1cc2](b8a1cc2))
* put access.session back
([storacha#1100](https://github.com/storacha/upload-service/issues/1100))
([10a1a4b](10a1a4b))
* rename blob and index client capabilities
([storacha#1478](https://github.com/storacha/upload-service/issues/1478))
([17e3a31](17e3a31))
* repo URLs
([storacha#1550](https://github.com/storacha/upload-service/issues/1550))
([e02ddf3](e02ddf3))
* tests
([b179910](b179910))
* trigger capabilities release
([storacha#1399](https://github.com/storacha/upload-service/issues/1399))
([7d9ab35](7d9ab35))
* type errors
([c0b97bf](c0b97bf))
* update data-segment dep
([228ff79](228ff79))
* upgrade @ucanto/validator with bugfix
([storacha#1151](https://github.com/storacha/upload-service/issues/1151))
([d4e961b](d4e961b))
* upgrade ucanto core
([storacha#1127](https://github.com/storacha/upload-service/issues/1127))
([5ce4d22](5ce4d22))
* upgrade ucanto in filecoin api
([c95fb54](c95fb54))
* upgrade ucanto libs and format filecoin api
([storacha#1359](https://github.com/storacha/upload-service/issues/1359))
([87ca098](87ca098))
* upload API test fixes
([6b0d72d](6b0d72d))


### Other Changes

* Add `pnpm dev` to watch-build all packages
([storacha#1533](https://github.com/storacha/upload-service/issues/1533))
([07970ef](07970ef))
* **main:** release capabilities 10.1.0
([storacha#979](https://github.com/storacha/upload-service/issues/979))
([bdd3970](bdd3970))
* **main:** release capabilities 10.2.0
([storacha#983](https://github.com/storacha/upload-service/issues/983))
([a906488](a906488))
* **main:** release capabilities 11.0.0
([storacha#994](https://github.com/storacha/upload-service/issues/994))
([76b0489](76b0489))
* **main:** release capabilities 11.0.1
([storacha#1008](https://github.com/storacha/upload-service/issues/1008))
([37cdc5a](37cdc5a))
* **main:** release capabilities 11.1.0
([storacha#1026](https://github.com/storacha/upload-service/issues/1026))
([7fdb541](7fdb541))
* **main:** release capabilities 11.2.0
([storacha#1084](https://github.com/storacha/upload-service/issues/1084))
([0e7b6dc](0e7b6dc))
* **main:** release capabilities 11.3.0
([storacha#1098](https://github.com/storacha/upload-service/issues/1098))
([7d671bd](7d671bd))
* **main:** release capabilities 11.3.1
([storacha#1101](https://github.com/storacha/upload-service/issues/1101))
([20b5b35](20b5b35))
* **main:** release capabilities 11.4.0
([storacha#1105](https://github.com/storacha/upload-service/issues/1105))
([1b6210f](1b6210f))
* **main:** release capabilities 11.4.1
([storacha#1131](https://github.com/storacha/upload-service/issues/1131))
([a5b7154](a5b7154))
* **main:** release capabilities 12.0.0
([storacha#1137](https://github.com/storacha/upload-service/issues/1137))
([bb23e9f](bb23e9f))
* **main:** release capabilities 12.0.1
([storacha#1147](https://github.com/storacha/upload-service/issues/1147))
([d1a9c78](d1a9c78))
* **main:** release capabilities 12.0.2
([storacha#1152](https://github.com/storacha/upload-service/issues/1152))
([b9d7ff5](b9d7ff5))
* **main:** release capabilities 12.0.3
([storacha#1163](https://github.com/storacha/upload-service/issues/1163))
([ec5c385](ec5c385))
* **main:** release capabilities 12.1.0
([storacha#1195](https://github.com/storacha/upload-service/issues/1195))
([a21c1a5](a21c1a5))
* **main:** release capabilities 13.0.0
([storacha#1230](https://github.com/storacha/upload-service/issues/1230))
([3d5b3ef](3d5b3ef))
* **main:** release capabilities 13.1.0
([storacha#1257](https://github.com/storacha/upload-service/issues/1257))
([85adc9a](85adc9a))
* **main:** release capabilities 13.1.1
([storacha#1283](https://github.com/storacha/upload-service/issues/1283))
([31c38e9](31c38e9))
* **main:** release capabilities 13.2.0
([storacha#1315](https://github.com/storacha/upload-service/issues/1315))
([0505458](0505458))
* **main:** release capabilities 13.2.1
([storacha#1362](https://github.com/storacha/upload-service/issues/1362))
([26b5751](26b5751))
* **main:** release capabilities 13.3.0
([storacha#1366](https://github.com/storacha/upload-service/issues/1366))
([d6fbc4a](d6fbc4a))
* **main:** release capabilities 13.3.1
([storacha#1377](https://github.com/storacha/upload-service/issues/1377))
([149f592](149f592))
* **main:** release capabilities 14.0.0
([storacha#1386](https://github.com/storacha/upload-service/issues/1386))
([69bfc08](69bfc08))
* **main:** release capabilities 14.0.1
([storacha#1395](https://github.com/storacha/upload-service/issues/1395))
([a76c970](a76c970))
* **main:** release capabilities 14.0.2
([storacha#1400](https://github.com/storacha/upload-service/issues/1400))
([7b46852](7b46852))
* **main:** release capabilities 15.0.0
([storacha#1412](https://github.com/storacha/upload-service/issues/1412))
([ec90b81](ec90b81))
* **main:** release capabilities 16.0.0
([storacha#1419](https://github.com/storacha/upload-service/issues/1419))
([50e3934](50e3934))
* **main:** release capabilities 17.0.0
([storacha#1428](https://github.com/storacha/upload-service/issues/1428))
([6ee21fd](6ee21fd))
* **main:** release capabilities 17.1.0
([storacha#1447](https://github.com/storacha/upload-service/issues/1447))
([8ee1fd9](8ee1fd9))
* **main:** release capabilities 17.1.1
([storacha#1483](https://github.com/storacha/upload-service/issues/1483))
([5394ed5](5394ed5))
* **main:** release capabilities 17.2.0
([storacha#1494](https://github.com/storacha/upload-service/issues/1494))
([99876a5](99876a5))
* **main:** release capabilities 17.3.0
([storacha#1503](https://github.com/storacha/upload-service/issues/1503))
([891c2a5](891c2a5))
* **main:** release capabilities 17.4.0
([storacha#1559](https://github.com/storacha/upload-service/issues/1559))
([feaea7a](feaea7a))
* no longer depends on hd-scripts, packages use/configure eslint
directly, fixes warnings from npm lint script
([storacha#1058](https://github.com/storacha/upload-service/issues/1058))
([ebdb99b](ebdb99b))
* package renames
([0f797ed](0f797ed))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant