Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HTTP Gateway Specs #283

Merged
merged 26 commits into from
Jul 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
a28d1de
feat: initial HTTP gateway specs
lidel Jun 2, 2022
6e24eb0
gateway: add Content-Range
lidel Jun 3, 2022
2e4374f
gateway: registerProtocolHandler uri router
lidel Jun 3, 2022
101fa5e
CODEOWNERS: add lidel for ./http-gateways
lidel Jun 3, 2022
7be0611
gateway: resolving an advanced DNSLink chain
lidel Jun 8, 2022
6a0e2fc
gateway: only-if-cached HEAD behavior
lidel Jun 8, 2022
13f53a8
gateway: suggestions from reviewers
lidel Jun 8, 2022
a414411
gateway: include CIDv1 node in summary
lidel Jun 8, 2022
af0363e
gateway: reorder URI router section
lidel Jun 8, 2022
4156b43
gateway: add Denylists section
lidel Jun 8, 2022
5435910
gateway: switch only-if-cached miss to 412
lidel Jun 9, 2022
176133a
gateway: apply suggestions from review
lidel Jun 9, 2022
cad7046
gateway: apply suggestions from Cloudflare
lidel Jun 15, 2022
04111e6
gateway: add X-Content-Type-Options
lidel Jun 15, 2022
ce193e6
gateway: simplify dnslink summary
lidel Jun 23, 2022
ffe8af9
gateway: document 412 Precondition Failed
lidel Jun 23, 2022
7e65a76
gateway: link to ipld codecs explainer
lidel Jun 23, 2022
e3637e5
gateway: stub about handling traversal errors
lidel Jun 23, 2022
2c731d3
gateway: expand HTTP caching considerations
lidel Jun 23, 2022
6787fd8
gateway: editorial fixes
lidel Jun 23, 2022
25497a4
gateway: expand on Host header parsing
lidel Jun 23, 2022
4b57971
gateway: editorial fixes
lidel Jun 23, 2022
75543c3
gateway: X-Forwarded-Proto and X-Forwarded-Host
lidel Jun 24, 2022
849a7e3
gateway: editorial fixes
lidel Jun 24, 2022
e06249f
gateway: X-Trace-Id
lidel Jun 24, 2022
9fc9a9c
gateway: Generated HTML with directory index
lidel Jun 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Spec Stewards defined below are automatically requested for review when
# someone opens a pull request that modifies area of their interest.

http-gateways/ @lidel
12 changes: 5 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,12 @@ The specs contained in this repository are:
- [Protocol Architecture Overview](./ARCHITECTURE.md) - the top-level spec and the stack
- [Other IPFS Overviews](/overviews) - quick overviews of the various parts of IPFS
- **User Interface (aka Public APIs):**
- [Core API (aka using IPFS as a package/module)](./API_CORE.md)
- [JavaScript Interface](https://github.com/ipfs/interface-js-ipfs-core)
- [Golang Interface](https://github.com/ipfs/interface-go-ipfs-core)
- [CLI (the ipfs daemon API)](./API_CLI.md)
- [HTTP API](./API_HTTP.md)
- HTTP Gateway
- [HTTP Gateways](./http-gateways/README.md) - implementation agnostic interfaces for accessing content-addressed data over HTTP
- IPFS implementations may provide additional interfaces, for example:
- [HTTP RPC API exposed by go-ipfs](https://docs.ipfs.io/reference/http/api/)
- [Programmatic Core API for JavaScript](https://github.com/ipfs/js-ipfs/tree/master/docs/core-api#readme)
- **Data Formats:**
- [IPLD](https://github.com/ipld/spec) - InterPlanetary Linked Data.
- [IPLD](https://ipld.io/specs/) - InterPlanetary Linked Data.
- [Merkle DAG (Deprecated)](./MERKLE_DAG.md)
- Self Describing Formats ([multiformats](http://github.com/multiformats/multiformats)):
- [multihash](https://github.com/multiformats/multihash) - self-describing hash digest format.
Expand Down
100 changes: 100 additions & 0 deletions http-gateways/DNSLINK_GATEWAY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# DNSLink Gateway Specification

![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square)

**Authors**:

- Marcin Rataj ([@lidel](https://github.com/lidel))

----

**Abstract**

DNSLink Gateway is an extension of
[PATH_GATEWAY.md](./PATH_GATEWAY.md)
that enables hosting a specific content path under a specific DNS name.

This document describes the delta between [PATH_GATEWAY.md](./PATH_GATEWAY.md) and this gateway type.

In short:

- HTTP request includes a valid [DNSLink](https://dnslink.dev/) name in `Host` header
- gateway decides if DNSlink name is allowed
- gateway resolves DNSLink to an immutable content root identified by a CID
- HTTP response includes the data for the CID

# Table of Contents

- [DNSLink Gateway Specification](#dnslink-gateway-specification)
- [Table of Contents](#table-of-contents)
- [HTTP API](#http-api)
- [`GET /[{path}][?{params}]`](#get-pathparams)
- [`HEAD /[{path}][?{params}]`](#head-pathparams)
- [HTTP Request](#http-request)
- [Request headers](#request-headers)
- [`Host` (request header)](#host-request-header)
- [HTTP Response](#http-response)
- [Appendix: notes for implementers](#appendix-notes-for-implementers)
- [Leveraging DNS for content routing](#leveraging-dns-for-content-routing)

# HTTP API

## `GET /[{path}][?{params}]`

Downloads data at specified path under the content path for DNSLink name provided in `Host` header.

- `path` – optional path to a file or a directory under the content root sent in `Host` HTTP header
- Example: if `Host: example.com` then the content path to resolve is `/ipns/example.com/{path}`

## `HEAD /[{path}][?{params}]`

Same as GET, but does not return any payload.

# HTTP Request

Below MUST be implemented **in addition** to the [HTTP Request section from `PATH_GATEWAY.md`](./PATH_GATEWAY.md#http-request).

## Request headers

### `Host` (request header)

Defines the [DNSLink](https://docs.ipfs.io/concepts/glossary/#dnslink) name
to RECURSIVELY resolve into an immutable `/ipfs/{cid}/` prefix that should
be prepended to the `path` before the final IPFS content path resolution
is performed.

Implementations MUST ensure DNSLink resolution is safe and correct:
- each DNSLink may include an additional path segment, which MUST be preserved
- each DNSLink may point at other DNSLink, which means there MUST be a hard
recursion limit (e.g. 32) and HTTP 400 Bad Request error MUST be returned
when the limit is reached.

**Example: resolving an advanced DNSLink chain**

To illustrate, given DNSLink records:

- `_dnslink.a.example.com` TXT record: `dnslink=/ipns/b.example.net/path-b`
- `_dnslink.b.example.net` TXT record: `dnslink=/ipfs/bafy…qy3k/path-c`

HTTP client sends `GET /path-a` request with `Host: a.example.com` header
which recursively resolves all DNSLinks and produces the final immutable
content path:

1. `Host` header + `/path-a` → `/ipns/a.example.net/path-a`
2. Resolving DNSlink at `a.example.net` replaces `/ipns/a.example.net` with `/ipns/b.example.net/path-b`
3. Resolving DNSlink at `b.example.net` replaces `/ipns/b.example.net` with `/ipfs/bafy…qy3k/path-c`
4. The immutable content path is `/ipfs/bafy…qy3k/path-c/path-b/path-a`

# HTTP Response

Same as [HTTP Response section in `PATH_GATEWAY.md`](./PATH_GATEWAY.md#http-response).

# Appendix: notes for implementers
lidel marked this conversation as resolved.
Show resolved Hide resolved

## Leveraging DNS for content routing

- It is a good idea to publish
[DNSAddr](https://github.com/multiformats/multiaddr/blob/master/protocols/DNSADDR.md)
TXT records with known content providers for the data behind a DNSLink. IPFS
clients will be able to detect DNSAddr and preconnect to known content
providers, removing the need for expensive DHT lookup.
Loading