Add HTTP over libp2p and http blockstore #6
Conversation
src/get-helia.ts
Outdated
| @@ -44,6 +48,14 @@ export async function getHelia ({ usePersistentDatastore, libp2pConfigType }: Ge | |||
| // libp2p is the networking layer that underpins Helia | |||
| const libp2p = await getLibp2p({ datastore, type: libp2pConfigType }) | |||
|
|
|||
| // TODO we don't get the webtransport multiaddr after a provide. TODO debug. | |||
| const marcoServer = multiaddr('/ip4/34.221.29.193/udp/4001/quic-v1/webtransport/certhash/uEiC13IfwpbpLsAgaV3a-9JR9FDZPxPabgv-UqmAfQuHeVw/certhash/uEiAdK9M5b3NvBMINTaVbfLAjxsTMTY2x-pEQB6lPYQe-Tw/p2p/12D3KooWEKMXiNrBNi6LkNGdT7PxoGuTqFqAiQTosRHftk2vk4k7') | |||
There was a problem hiding this comment.
where's the code that runs this server? Some out of curiosity and some out of wanting to be able to debug if there are issues 😅
There was a problem hiding this comment.
src/http-blockstore.ts
Outdated
| // - Keep a list of HTTPS endpoints, not just libp2p peers. | ||
| // - Keep a list of previous peers we were connected to, so even if we lose a webtransport connection, we can spin it up later. | ||
|
|
||
| export class HttpBlockstore implements Blockstore { |
There was a problem hiding this comment.
IIUC this approach will cause problems when also enabling fetching over Bitswap (I think it might check the blockstore before doing a Bitswap request which would basically cause these lookups to be sequential).
An alternative way to do this would be (although @achingbrain might have more helia-friendly suggestions):
- Implement this as the Bitswap interface (i.e. want, unwant, cancel_wants)
- Make a third implementation of the Bitswap interface that basically asks both in parallel, but when one of them yields a block it cancels on the other one
Later we can basically do combined task prioritization with Bitswap whereby we hit peers (Bitswap or HTTP or HTTP over libp2p) based on how quickly they respond to us with good data.
src/http-blockstore.ts
Outdated
| // TODOS: | ||
| // - Validate the blocks we fetch | ||
| // - Use format=car to get the whole dag at once rather than per block (probably faster?) | ||
| // - Keep a list of HTTPS endpoints, not just libp2p peers. |
There was a problem hiding this comment.
I'd say we should (if we have time) just make up some record data and chuck it into IPNI for our records where the gateways are non-recursive (i.e. only based on data we have locally similar to https://github.com/ipfs/kubo/blob/c58aadb887a5ca88256ac8f3a8fbf60b0263c978/docs/config.md#gatewaynofetch).
Happy to help here, it shouldn't be too bad as I've done some hacking here before. Basically we'd use this format ipni/specs#6 to avoid dealing with the global code table (and associated bikeshedding) as much as possible. It's a small amount of parsing in the code here, and then we can hook up some hacky IPNI record publishing.
For recursive gateways (e.g. dweb.link, ipfs.io, cf-ipfs.com) we could either hard code them, get them from https://ipfs.github.io/public-gateway-checker/ or have an IPNI rendezvous record.
If there's no time, then hard coding it is 😄
src/http-blockstore.ts
Outdated
| import { fetchViaDuplex } from '@marcopolo_/libp2p-fetch' | ||
|
|
||
| // TODOS: | ||
| // - Validate the blocks we fetch |
There was a problem hiding this comment.
We need to do this and it should be fairly straightforward to do (just grab the multihash, lookup an implementation of the hash function, run it over the bytes and see if they match)
src/http-blockstore.ts
Outdated
|
|
||
| // TODOS: | ||
| // - Validate the blocks we fetch | ||
| // - Use format=car to get the whole dag at once rather than per block (probably faster?) |
There was a problem hiding this comment.
This is almost certainly a separate PR once we've got the rest of this done.
src/lib/connectAndGetFile.ts
Outdated
| @@ -86,7 +86,7 @@ export async function connectAndGetFile ({ channel, localMultiaddr, fileCid, hel | |||
| color: COLORS.active | |||
| } | |||
| }) | |||
| await helia.stop() | |||
| // await helia.stop() | |||
There was a problem hiding this comment.
This was client side stopping of the helia node when fetch was done. We chatted about it in slack a little bit.
However it's probably good to know that most of the stuff with channel.blah and postMessage is not used in the latest code on main. Its leftover from the more "debugging " version when we had the ui terminal output
|
Changed this so that we bitswap and http in parallel. It also supports normal HTTP endpoints. Depending on the graph this could be a large amount of small round trips :/ Uses this branch of js-ipfs-bitswap: https://github.com/ipfs/js-ipfs-bitswap/compare/marco/bitswap-and-http?expand=1 |
|
@aschmahmann do we want to revive this? |
Adds HTTP over libp2p support. Introduces an HTTP blockstore that asks peers over the IPFS http gateway if they have some block if the underlying blockstore doesn't have it.
Try it by requesting
/ipfs/QmPvnCPgTwHvBic3G9EsRBSssVp8UCtCs4jNWksTHTCDyx.A couple of TODOs left in the code.