-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: IPFS Gateway HTTP API #1
Conversation
This is the first proposal, being used as a process example as well as a real proposal. So we're figuring out the workflow as we go. I think review here is only about whether this is an appropriate proposal, and for discussion/collaboration on the proposal itself. Approval signals nothing about whether we should actually undertake the proposal at any point in time. We'll discuss that in separate issues. |
#### Dependencies/prerequisites | ||
|
||
- JS-IPLD libraries capable of core encodings and IPLD representation | ||
- IPLD selectors (for non-trivial DAG traversal) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just the golang / server side implementation, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JS implementation will be necessary for sub-DAG queries from the client, which would be needed for complex cases. So no, not just server side. But also we needn't block creating and exposing this API on that JS selector implementation (I gather it doesn't exist?). It will just be more useful once JS clients can express queries.
👍 👍 Designing an API that's usable by a lightweight client, while also being designed explicitly to avoid "N+1 query problems", seems very high-value to me. |
I'm +1 for a simple API that doesn't need a complicated client to access. However this proposal or the design of the outcome of this proposal should take into account the limitations of the existing HTTP API when used in anything that implements browser Fetch. TLDR: you cannot do full-duplex streaming, so design the API as simple request/response pairs, or only require streaming from the server to the client. More detail:
In js-IPFS we settled on using gRPC over websockets which addresses these points, though the API becomes harder to use without a client which runs counter to the spirit of this proposal. This may not be an issue, most people reach for some sort of client when dealing with an API in order to get something that respects the conventions and norms of their development platform and to not have to deal with the weird foibles of whatever the API developers thought was REST* - the people cURLing APIs are not (I believe) the majority of Web2-graduating-to-Web3 developers, not after the first few minutes anyway. Further reading:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The true scope of this proposal will depend on how capable Writable Gateway approach (added it as prior art below) turns out to be when it is not limited to files, but can operate on IPLD DAGs, control pinning.
But yes, I imagine it being simple request/response pairs with potential for leveraging https://streams.spec.whatwg.org when those features land in browsers, and perhaps, in the future support resumable uploads as well (prior art: https://tus.io, ipfs/js-ipfs#1540) to make DX/UX easier when working with huge imports.
We have distinct Gateway (/ipfs
, /ipns
) and API (/api/v0
) endpoints today, and only the most basic read operations can be executed via Gateway. The goal here would be to shift what is possible on the Gateway, so most popular operations work without touching advanced API.
Error reporting on /api/v0
is broken in browsers, as noted by @achingbrain in ipfs/js-ipfs#2519,
so for everything not handled by Gateway (eg. pubsub, advanced streaming), we need to replace HTTP-based /api/v0
with something like RPC over websockets anyway (js-ipfs already does this: ipfs/js-ipfs#3403).
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Thanks! I've noted the HTTP streaming limitations in the body. @lidel I have also declared you a co-author, thanks for fleshing some bits out. |
It's worse than that, errors and streaming are broken anywhere that uses Fetch - that's browsers, node-fetch, electron-fetch, React Native, etc. The only place it does work is via node's http module, but Fetch is the thing that users reach for when they need to make a network request on pretty much every platform JS runs on and it will be what they use if you're going for clientless here. |
I'd like to avoid supporting two APIs if possible unless we intend to deprecate/remove one of them. I'm concerned that either:
That's why I'm personally in-favor of the gRPC+websocket API. On the other hand, I have occasionally wanted a slightly more featureful gateway, mostly for integration with scripts where there is no convenient js-ipfs-http-client. But I'd still prioritize making the more general purpose API suck less first. |
l feel we have a soft consensus(?) that we need to deprecate Perhaps that should be a separate Proposal tho? |
Love it. One technical consideration I’d like to add: whenever we create an API or a protocol that uses selectors we should modularize the use of selectors so that, in the future, we could swap selectors out for another query mechanism and reuse the rest of the protocol or API. This should be easy, we just need to use a union and it shouldn’t add any extra burden to the implementation and will futureproof the work. |
Would love to see one more review approval from someone who works on the gateway. |
We're going this direction, but the work is covered in other proposals including in #100 . Closing as a result. |
This is a project proposal for the web3 dev team.