Gateway renders pretty 404 pages if available

[jphastings]

I noticed ipfs/ipfs#167 and thought this might be an elegant solution for allowing the gateway to show helpful information when requested files aren't found.

When a requested file isn't found ipfs-404.html is looked for in the same directory, rolling up through any of its parents, and displayed (without immutable cache headers) if present.

Run this patch locally and query the following for a demonstration:

• Pretty 404 page at /nope
• More specific pretty 404 page at /deeper/nope
• Fallback pretty 404 page at /nope/nope

[whyrusleeping]

Cool feature. cc @lgierth @diasdavid

[whyrusleeping]

As for tests, you probably want to add something in core/corehttp/gateway_test.go, some of the tests in there construct directories, in particular TestIPNSHostnameRedirect creates a directory with an index.html to test that working properly. I imagine a test for this 404 feature would be quite similar.

[jphastings]

Thanks for the advice @whyrusleeping, hopefully this covers the new cases. Shout if there's anything else I can do to help!

[whyrusleeping]

Thinking about this, what is the need for not setting a cache? Everything is content addressed, right? There should be no need not to cache.

@dignifiedquire @victorbjelkholm I think you guys might be interested in this PR

[Kubuxu]

Thinking about this, what is the need for not setting a cache? Everything is content addressed, right? There should be no need not to cache.

Choosing ETag for the cache is not simple in this case as the 404.html is taken for reverse recursive walk but I think using CID of the resolved 404.html should work but I am not 100% sure.

We would also have to resolve the 404.html when client caches it to confirm the ETag.

[jphastings]

@whyrusleeping I'm thinking of the case where I request a non-existant file from the gateway today; I don't want my browser's cache to be prevented from trying again, say, tomorrow when that DNSlink entry/peerID points to a new hash where that file now exists.

Right now this PR ensures the gateway sets the ETag of a requested but missing file to be the Cid of the pretty 404 page that was found (as @Kubuxu suggests). Smart browsers can then include it in the If-None-Match header of their next request to the gateway for this file which will avoid sending unnecessary data. I reckon this is all we need to do here to allow smart clients to request transfer of the bare minimum to show the most recent data.

Does this address your concern?

[daviddias]

Thinking about this, what is the need for not setting a cache? Everything is content addressed, right? There should be no need not to cache.

What if the gateway fails to find the file, timeouts and returns a 404 with cache-control: unlimited? An existing hash->file can be cached forever and the corollary is that a non-existing hash->file will someday become an existing hash->file.

[victorb]

If you load ipfs.io/ipfs/$HASH/$FILE and $FILE isn't found, a redirect to a 404 page under $HASH would always be a 404.

As both Etag and Cache-Control uses the URL, it has to be a redirect and if it redirects to $HASH/404.html, that redirect would always happen in the future.

Introducing IPNS would make things work differently though, and probably in that case it's best to dump the Cache-Control header. However, if IPFS isn't being used with IPNS, the 404 should be permanent.

What if the gateway fails to find the file, timeouts and returns a 404 with cache-control: unlimited?

In the case of ipfs.io/ipfs/$HASH/$FILE, don't we already see all the links of $HASH and if we cannot see the match for $FILE, we can safely show the 404 page?

[ivan386]

As both Etag and Cache-Control uses the URL, it has to be a redirect and if it redirects to $HASH/404.html, that redirect would always happen in the future.

No redirect for 404 page please. I need to know original url where 404 happen. Then i can redirect to right location by Javascript.

I think about .htaccess for ipfs.

[jphastings]

There's no redirecting happening at the moment. I believe @victorbjelkholm was talking about internally, ie. the data is being delivered when a missing file is requested.

In the case of ipfs.io/ipfs/$HASH/$FILE, don't we already see all the links of $HASH and if we cannot see the match for $FILE, we can safely show the 404 page?

I agree — in this case we know that $FILE definitely doesn't exist under the $HASH and we can safely deliver the data from an appropriate 404.html page from that URL provided we also set the cache headers to be non-permanent and the ETag to be of the data delivered (ie. the 404.html page) so that if $FILE becomes present (in the case where we're addressing via an IPNS link) the new data can be retrieved.

Further to this, in the case of ipfs.io/ipfs/$HASH/$FILE (ie. not IPNS), we could also deliver the data with an immutable cache header, as the contents of the $HASH will never change. If we like the idea of this I can change this conditional so that it allows setting the immutable header for /ipfs paths with 404s. In fact, we could set the immutable cache header for any /ipfs path, including for directories (which is not currently the case on HEAD). The only argument against this is that we are also declaring this business logic to be immutable (if we ever decided to move from 404.html to 404_error.html as the page too look for, what a customer sees would be dependent on which version of IPFS they first viewed that URL with).

Because of this I'm up for leaving the 404.html page without the immutable cache header, but leave the ETag in place, as it stands. It's extremely unlikely we'd ever need to change the 404.html to something else, but better to not force immutability on the business logic at this stage I feel!

[mitra42]

I think doing this at the Gateway is the wrong place, because if I've understood this correctly, its indicating to the caller (which might be an application, not a browser) that its retrieved the page. Shouldnt this be part of the logic in whatever is calling the Gateway (browser) to decide what to do when the page isn't found.

For example ... I can see situations where if the file can't be delivered by the Gateway, the app decides to go to somewhere else (such as an Archive) to try a different way to retrieve the same content.

[Stebalien]

@mitra42 The gateway will still return a 404 status so the application can decide what to do with it.

However, I believe this should be checking the client's Accept header. If the client doesn't want an HTML page, we shouldn't be returning an HTML 404 page.

[jphastings]

That's an excellent idea @Stebalien; only if the request header matches text/html should this functionality be present - I'll add that to this PR now!

@mitra42 while I agree with your point in pure terms, my understanding of the Gateway's purpose is that it should bring some of the benefits of IPFS to browsers which don't have native IPFS support, which leads me to two counter arguments:

• If a browser had native support for IPFS then this could be implemented in that plugin, but there isn't one of those (yet!) and the gateway is the bit of software that is the first step in that direction.
• If browsers were mostly used for displaying non-dynamically generated content then this is a proposal which could be put into effect there (so that it would benefit non-IPFS sourced sites too) but, because HTTP responses are most frequently generated dynamically, this is more easily implemented "on the [centralised] server" in the current world of HTTP (a concept which doesn't exist in IPFS land!) In fact the closest analogue to "on the server" in the world of an IPFS-HTTP bridge is the gateway.

I hope that makes sense!

[mitra42]

Agreed,

[jphastings]

It might be worth noting that if a person wishing to use this feature wants their pretty 404 page's assets to render correctly without specifying absolute paths (eg. if they want the 404s to work with accessing the file from an IPFS or IPNS URL like gateway.server/ipfs/HASH/missing_resource as well as from a DNS link like example.com/missing_resource) then some logic would need to be applied to give the browser enough information to know where to get assets from (as neither absolute nor relative paths will work on their own). I've found this snippet works well, though it will need to be tailored to how assets are set up:

<head>
  <!-- things that don't refernce assets -->
  <script>
    const ipfsPath = /^\/ip[fn]s\/(.+?)\//.exec(window.location.pathname);
    const basePath = (ipfsPath == null) ? '/' : ipfsPath[0];
    document.write('<base href="' + basePath + '"/>');
  </script>
  <!-- things that reference assets, eg. CSS and JS -->
</head>

[jphastings]

@whyrusleeping is there anything further you'd like me to alter on this PR?

[whyrusleeping]

This LGTM on a technical level. Now we need to get a signoff from the others on whether or not this is something we really want.

cc @diasdavid @dignifiedquire

[Kubuxu]

Apart from those 2 style quirks, SGTM.

[jphastings]

This should cover the changes you suggested @Stebalien — feedback is welcome! ☺️

[Stebalien]

Technically, if a 404 page is present, we should return 404 here. But I'm fine leaving that edge-case unfixed.

[Stebalien]

🥳

Thanks for your patience on this, this has been a long time coming.

[Lekensteyn]

If I am not mistaken, this change implies a lookup for ipfs-404.html on every attempt to obtain a directory listing, and subsequently breaks directory listings if the file is found. Is that intentional?

In core/corehttp/gateway_test.go, I think that the following tests are wrong:

{"/deeper/", "text/html", http.StatusNotFound, "Deep custom 404"},
{"/deeper", "text/html", http.StatusNotFound, "Deep custom 404"},

These and / should probably end up with a directory listing.

[jphastings]

That was intentional by me — the approach that web servers like NGinx take is index.html goes first, a 404 page goes second, then a directory listing if there's no 404 defined. I worked to echo that here.

[Lekensteyn]

That sounds rather unusual. By default directory indexes are disabled unless autoindex is changed. The error_page directive only affects the page shown on errors, it does not affect the routing logic (whether to display an index or not).

Did you have a custom config such as try_files $uri $uri/index.html =404; in mind?

With the current approach, there is no way to obtain a directory index if ipfs-404.html exists.

[jphastings]

That's a good point; I guess it comes down to whether IPFS wants directory listings to always be possible to retrieve on the HTTP gateway.

Given this is my first PR to IPFS I don't feel like I'm qualified to make such a judgement call (beyond what I'd personally like) — so maybe we can lean on @Stebalien?

[Lekensteyn]

If directory listings are not desirable, one could add an explicit index.html file to hide the contents. There is probably no point in using this custom 404 page feature for hiding files since people can just query the directory directly, that is where IPFS differs from nginx.

I am also new here and do not know if special files such as index.html and ipfs-404.html are documented anywhere, but that would be nice to have :-)

[Stebalien]

Hm. Yeah, I didn't fully think through the ramifications of this. I actually have at least one case where I want to support custom 404 pages and directory listings (dist.ipfs.io).

[DavidBurela]

I love that there is thinking around supporting 404 pages.
I've published my blog to IPFS/ENS and lacking 404 was something I wanted
e.g. http://blog.davidburela.eth/broken

But because it is a blog with pages in this folder format

/
  /2019
    /01
    /02
    /03
    ...
  /2020
    /01
    /02

it means I'd need to put a ipfs-404.html in every single subfolder. Maybe specifying it once at root of a merkle tree is enough?
<cid>/ipfs-404.html

[Stebalien]

Specifying it once at the root should be enough. The gateway will search all parent directories up the path for an ipfs-404.html file.