Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gw): support UnixFS 1.5 on gateway responses as Last-Modified #659

Merged
merged 4 commits into from
Aug 28, 2024

Conversation

lidel
Copy link
Member

@lidel lidel commented Aug 19, 2024

This is bit more involved, and I don't want to block #653 on it, so filling this separate PR.

This PR builds on top of #653 and exposes mtime from dag-pb as Last-Modified header on deserialized /ipfs and /ipns responses (ones that are used for web hosting, and not trustless responses).

TODO

  • block backend
    • return Last-Modifiedon GET responses
    • return Last-Modifiedon HEAD responses
    • 304 Not Modified on matching If-Modified-Since (UnixFS 1.5 file only)
    • Last-Modified and If-Modified-Since tests with and without UnixFS 1.5

(Below moved to #664)

  • car backend
    • return Last-Modifiedon GET responses
    • return Last-Modifiedon HEAD responses
  • add gateway-conformance test and release it as v7
    • confirm conformance passes with all backends (remote car uses different implementation of Get and Head and ContentPathMetadata needs to be updated there as well)

@lidel lidel mentioned this pull request Aug 19, 2024
32 tasks
Copy link

codecov bot commented Aug 19, 2024

Codecov Report

Attention: Patch coverage is 49.27536% with 35 lines in your changes missing coverage. Please review.

Project coverage is 60.16%. Comparing base (3cd3857) to head (2e946d1).
Report is 1 commits behind head on main.

Files Patch % Lines
gateway/handler.go 42.59% 27 Missing and 4 partials ⚠️
gateway/backend_blocks.go 50.00% 4 Missing ⚠️

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #659      +/-   ##
==========================================
+ Coverage   60.13%   60.16%   +0.02%     
==========================================
  Files         241      241              
  Lines       30617    30681      +64     
==========================================
+ Hits        18412    18459      +47     
- Misses      10560    10570      +10     
- Partials     1645     1652       +7     
Files Coverage Δ
gateway/gateway.go 83.54% <ø> (ø)
gateway/handler_defaults.go 51.41% <100.00%> (+2.00%) ⬆️
gateway/backend_blocks.go 45.48% <50.00%> (+0.06%) ⬆️
gateway/handler.go 76.27% <42.59%> (-3.35%) ⬇️

... and 12 files with indirect coverage changes

Base automatically changed from feat/unixfs15 to main August 21, 2024 00:03
@lidel lidel force-pushed the feat/last-modified-unixfs-on-gw branch from f894d64 to 0617d60 Compare August 21, 2024 00:35
@lidel lidel force-pushed the feat/last-modified-unixfs-on-gw branch from 0617d60 to 1fcc68b Compare August 27, 2024 19:52
lidel added 2 commits August 28, 2024 00:36
bare minimum support for Last-Modified and If-Modified-Since
Copy link
Member Author

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed next steps here with @gammazero during triage today and we decided to ship basic gateway support for Last-Modified / If-Modified-Since for UnixFS 1.5 files with backend_blocks.go (used by Rainbow and Kubo).

The remaining work can be done at a later time, if there is real world need for support in CAR backend. I've filled #664 for that.

Merging this so we can include it in Kubo 0.30.0-rc2.

@@ -745,6 +772,50 @@ func (i *handler) handleIfNoneMatch(w http.ResponseWriter, r *http.Request, rq *
return false
}

func (i *handler) handleIfModifiedSince(w http.ResponseWriter, r *http.Request, rq *requestData) bool {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ this is based on what we already had in handleIfNoneMatch to support If-None-Match, which operates on Etags.

This one is executed only if If-Modified-Since is present in request, which usually is less popular than more deterministic If-None-Match.

Still, we add is here as it costs us nothing to avoid returning payload and returning HTTP 304 Not Modified instead.

@lidel lidel marked this pull request as ready for review August 27, 2024 23:19
@lidel lidel requested a review from a team as a code owner August 27, 2024 23:19
func TestHeadersUnixFSModeModTime(t *testing.T) {
t.Parallel()

ts, _, root := newTestServerAndNode(t, "unixfs-dir-with-mode-mtime.car")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ created with Kubo 0.30.0-rc1

@lidel
Copy link
Member Author

lidel commented Aug 28, 2024

Added more tests, CI in ipfs/kubo#10494 is green, merging.

@lidel lidel merged commit 08f200a into main Aug 28, 2024
18 checks passed
wenyue pushed a commit to wenyue/boxo that referenced this pull request Oct 17, 2024
…ed (ipfs#659)

bare minimum support for Last-Modified and If-Modified-Since
if mtime present in dag-pb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant