[Tracking issue] http delegated routing used in production

[BigLep]

Done Criteria

1. HTTP delegated routing is enabled on the ipfs.io gateway, hitting cid.contact in parallel to querying the DHT to determine providers for a CID.
2. ipfs.io gateway operators are able to assess delegated routing calls health with metrics.

Notes

This corresponds with "stage 1" in https://www.notion.so/pl-strflt/Indexer-IPFS-Reframe-Q3-Plan-77d0f3fa193b438784d2d85096e315e7

It doesn't matter if STI gives empty results or if the results are unreachable because they're to Filecoin Storage Providers that don't allow data transfer with Bitswap. Enabling retrieval over Bitswap for Filecoin SP's is a separate effort.

Reframe shipped in Kubo 0.14 as part of #8775. That said, it isn't used in production that we know of. The purpose of this tracking issue is to track followups that we have already identified as needing to be done or that come up as a result of attempting production deployment.

• Design desired comprehensive routing config #9188
• Expose Kubo metrics for reframe calls so a Kubo client has visibility into reframe activity (success/failure, latency).
• Convert to using the updated HTTP Delegated Routing API: Updated/new HTTP API Migration go-delegated-routing#63
• Enable on ipfs.io gateway (production usecase):
  • Old: https://github.com/protocol/bifrost-infra/issues/2120
  • New: https://github.com/protocol/bifrost-infra/issues/2183

---

Expanded scope for items observed while in production:

• Split provide messages (message size limits) go-delegated-routing#55

Other followup issues that should be done, but aren't directly tied to the done criteria.

• Add cacheable GET endpoint for findProviders go-delegated-routing#27
• Method proto.DelegatedRouting_Client.GetIPNS is never used. go-delegated-routing#30
• [Blog Post] Introducing Reframe ipfs-blog#446
• https://github.com/protocol/bifrost-infra/issues/2142

[BigLep]

2022-08-19 conversation:

Tasks involved in finishing this off:

1. Polish off the design

• Answer last remaining questions.
• We didn't estimate this, but it should be low/quick.

2. New routers (3 dev days)

• New routers
  • Sequential router
  • Parallel router
  • Router to filter by method
• May include refractoring
• Includes unit tests
• Code will live in https://github.com/libp2p/go-libp2p-routing-helpers
• Going to use existing routers if possible so we have less code to maintain but if this turns into a headache with Lotus, then may need new version.

3. Make this work in Kubo (5 dev days)

• Config parsing
• Generate Routing instances from config (simple approach)
• Using FX

4. Documentation (1 dev days)

• User-friendly config documentation
• Move design documentation around
• We can have a doc like "design decisions for routing routers".
• Idea: you could create kubo/docs/exploration-reposts dir, and put it there (ipld folks did that in past: https://github.com/ipld/specs/tree/master/design/history/exploration-reports, and then moved content to the proper website doics/specs )

5. Sharness smoke test (1 dev day)

• This will be adjusting the existing Sharness test we have.

6. Operational preparation (1 dev days)

• Need to answer how much time each of the routers are taking
• Edelweiss: define metrics we want, RPC metrics are already there

7. Testing on gateways (2 dev days)

Total estimate: 13 ideal dev days

Assuming a velocity of .7 ideal dev day to actual day, that puts us at 19 calendar days assuming just one person working on this.

Next steps here:

• @BigLep communicate worst-case calendar date in #reframe
• Clean up the existing issues
• Create new issues that PRs can be linked against

[BigLep]

@ajnavarro : I moved over the estimates from #9188 (comment) to here.

Can you help with the issue cleanup here? Specifically:

1. I think we need a new issue in go-libp2p-routing-helpers for the new routers.
2. I assume we'll update Routing: conform DHT to common Router interface #9079 to account for the design changes. This is "Make this work in Kubo" above right or are you going to create another issue?
3. I assume Routing: priority and mandatory Router Params #9083 and Delegated Routing: Allow to configure methods used by router.  #9157 can be closed in favor of the work happening in # 2 above, is that right?

[ajnavarro]

• New issue for parallel and sequential routers New Routing helpers: parallel and sequential. libp2p/go-libp2p-routing-helpers#56
• Updated issue Routing: conform DHT to common Router interface #9079
• What issue are you referring to? I think it is a typo.

[BigLep]

@ajnavarro : thanks. I updated "# 3" above. I believe the confusion stemmed from the fact that I wrote "#2" which got translated to issue "#2" in the repo.

[ajnavarro]

@BigLep yep, we can close #9083

[BigLep]

As communicated in Slack:

Total estimate is 13 ideal dev days
Assuming a velocity of .7 ideal dev day to actual day, that puts us at 19 calendar days assuming just one person working on this.
That feels high to me, but I think Bedrock should plan around that for now given we'll likely experience a jolt when Adin heads out, there is ongoing Kubo maintenance efforts that the team gets pulled into, and I don't think we'll be able to have an extra engineer on this beyond helping with code reviews.

[BigLep]

@ajnavarro : should #9157 combine into #9079 or will the config work be a separate PR. If it will be separate then I agree we should have two issues.

[ajnavarro]

@BigLep I think we can do them separately.

[ajnavarro]

• Config parser with tests: Done
• Create Router instances from validated config: Done
• Integrate with fx: In progress

[ajnavarro]

• First draft PR uploaded: feat: Delegated routing with custom configuration. #9274.
• Waiting for a new review on feat: Sequential and Parallel Routers libp2p/go-libp2p-routing-helpers#58.
• Now updating go-delegated-routing to the latest release. There were some changes in method signatures.

[lidel]

I was not sure where "productizing" https://routing.delegate.ipfs.io/reframe should live, so I filled https://github.com/protocol/bifrost-infra/issues/2142

[BigLep]

@guseggert : I added the important area to the done criteria we were missing last week of "ipfs.io gateway operators are able to assess reframe call health with metrics". Feel free to create a separate tracking issue for this work or do it here.

[BigLep]

I've expanded the done checklist to account for the migration from reframe to the updated HTTP delegated routing API: ipfs/go-delegated-routing#63 . I have updated the title as well.

[BigLep]

Pending verification that clientside metrics are flowing for ipfs.io gateway and cid.contact in https://github.com/protocol/bifrost-infra/issues/2183

[BigLep]

2023-04-25 maintainer conversation:
we're still trying to confirm that the ipfs.io gateway is calling cid.contact.
cid.contact is not seeing any requests from ipfs.io gateways per discussion in ipni/storetheindex#1597 (comment) and https://filecoinproject.slack.com/archives/C03RQFURZLM/p1682348755904599 .
@guseggert SSH'd to a box and confirmed it is using "autoclient"
@guseggert confirmed that when he uses the same version of Kubo with "autoclient" that cid.contact is being called
@guseggert is leading the investigation here

[willscott]

I suspect https://github.com/ipfs/boxo/blob/main/routing/http/client/transport.go#L24-L28 could be an issue:

• if the response body is not fully drained, the http client will not be able to re-use the connection. eventually you'll run out of the pool of clients and I think the logic then is to fail silently.
• The defaults will give a pretty limited number of default allowed "simultaneously inflight" to a single host - that likely needs to be tuned for gateways. https://cs.opensource.google/go/go/+/refs/tags/go1.20.3:src/net/http/transport.go;l=201-211

[guseggert]

if the response body is not fully drained, the http client will not be able to re-use the connection. eventually you'll run out of the pool of clients and I think the logic then is to fail silently.

IIUC the body is not fully drained in the case where the limit is reached, which is indeed a bug, but are there actually records in cid.contact with >1 MiB of providers?

[guseggert]

My other theory is that it is related to the accelerated DHT client. When I run locally with FullRT client, no reqs are made to cid.contact.

[willscott]

• there may be records with >1MiB of response

bitswap issues a query for 10 providers and the accelerated DHT returns 20 immediately, terminating the query before it even starts on cid.contact.

I don't think this explains things - we have cases of providers that only are publishing to cid.contact.
there wouldn't be 20 peers already in the DHT there, and we don't find them on gateways.

Just because you have some peers from FullRT you don't need to close the channel / cancel the query on the indexer side, right?

(in the same way that when indexers gets requests requesting a cascade to the DHT they don't cancel the dht cascade once we've returned responses from the index.)

[willscott]

~30% of queries to kubo on gateways end up timing out with no peers, so we should still see those at minimum reaching the indexers, right?

[guseggert]

Okay there's a plumbing bug in Kubo's rats nest of content routing plumbing that is causing this issue, when the experimental DHT client is turned on then the cid.contact router is not used. Working on a fix, could take me a few hours to untangle that mess.

[BigLep]

Thanks @guseggert for the update.

As you are working on this, friendly reminder that we plan to move "accelerated DHT" out of experimental in the next release. If there are followups that need to occur, fee free to drop them to #9703

In terms of releasing this, I assume we'll have it part of 0.20? (We'll talk during 2023-04-27 standup. Alternative is that Bifrost uses a custom branch until this come in a future release.)

A challenge here will be code review given team members being out. I think our best case is that @lidel take a look at it Thursday, 2023-04-27.

[willscott]

i would push to get an RC onto bifrost and validate things are working as expected before releasing

[guseggert]

Agreed we should one-box this on the gateways before releasing to make sure it works, otherwise the feedback loop is brutal

[BigLep]

2023-04-27 maintainer conversation: this is still the top priority of @guseggert that he is actively working on. We don't have a PR yet.

[guseggert]

Okay I have working code now and can confirm that the default HTTP routers are being invoked alongside the FullRT client, will open a PR shortly and the plan is for @Jorropo to take a look on Monday

[BigLep]

Thanks @guseggert . Could you please also give directions on what Bifrost would need to test it? I'd like to get their test/verification happening in parallel.

[BigLep]

2023-05-02 update: The fix has deployed to "one box" on ipfs.io gateway. Integration looks good. These are going to get rolled out to the whole fleet over the next day or two.

[BigLep]

I'm closing this issue because ipfs.io gateway <> cid.contact integration is rolled out (see https://github.com/protocol/bifrost-infra/issues/2183#issuecomment-1533342956) and confirmed to be working.