gateway: fix erroneous Cache-Control: immutable on dir listings

[ghost]

We set the Cache-Control: immutable header on responses from the gateway's /ipfs route. That's fine, but we also set it on directory listings such as https://ipfs.io/ipfs/QmSwzyZ4zfHD9j4JZdXuzbWH4gXuDfZ7CdfMzpfqrMVcrz/go-libp2p-blankhost/ which effectively prevents us from changing e.g. the CSS on the listings, or really anything about the listing.

We should only ever set Cache-Control: immutable for responses that produce exactly the hash in the URL. This is only the case for unixfs files, not for the auto-generated directory listings.

[recmo]

Immutability also affects ETags. In my PR I make them strongly validating, which is equivalent to saying byte-for-byte immutable. We can do the following:

1. Turn them into weakly validating for dirs, implying binary differences but semantic sameness. The standard allows the implementer to decide what a "semantic change" is. We can argue that stylistic changes are not semantic. But browsers would not refresh the result afaik.
2. Compute them on the fly by hashing the binary result.
3. Add a version number to the dir ETag that we increment each time the dir generation code or its assets are changed.

I kind of like 3 because it is the most accurate and easy to implement. But it doesn't work if the generation code is runtime modifiable and adds a bit of maintenance burden.

[Kubuxu]

IMO it would be better to do this:

diff --git a/core/corehttp/gateway_test.go b/core/corehttp/gateway_test.go
index 55c0b15ee..887f12ca5 100644
--- a/core/corehttp/gateway_test.go
+++ b/core/corehttp/gateway_test.go
@@ -18,13 +18,14 @@ import (
 	repo "github.com/ipfs/go-ipfs/repo"
 	config "github.com/ipfs/go-ipfs/repo/config"
 	testutil "github.com/ipfs/go-ipfs/thirdparty/testutil"
+	ufs "github.com/ipfs/go-ipfs/unixfs"
 
 	ci "gx/ipfs/QmPGxZ1DP2w45WcogpW1h43BvseXbfke9N91qotpoQcUeS/go-libp2p-crypto"
 	id "gx/ipfs/QmeWJwi61vii5g8zQUB9UGegfUbmhTKHgeDFP9XuSp5jZ4/go-libp2p/p2p/protocol/identify"
 )
 
 // `ipfs object new unixfs-dir`
-var emptyDir = "/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"
+var emptyDir = "/ipfs/" + ufs.EmptyDirNode().Cid().String()
 
 type mockNamesys map[string]path.Path
 

[ghost]

I agree in principle, but I'm against adding more dependencies on go-ipfs packages. The good thing about these hashes is that the underlying data won't change.

[whyrusleeping]

LGTM

[dignifiedquire]

Edge now ships with compat for this https://wpdev.uservoice.com/forums/257854-microsoft-edge-developer/suggestions/17168363-support-chrome-s-improved-reload-behavior-or-cache