Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add indicator if CID is cached in local datastore #8880

Closed
wants to merge 1 commit into from
Closed

feat: add indicator if CID is cached in local datastore #8880

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion assets/dir-index-html/dir-index.html
View file
Open in desktop

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions assets/dir-index-html/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
"license": "MIT",
"scripts": {
"start": "cd test && go run .",
"build": "npm run build:clean && npm run build:remove-style-links && npm run build:minify-wrap-css && npm run build:combine-html-css && npm run build:remove-unused",
"build": "npm run build:clean && npm run build:remove-links && npm run build:minify-wrap && npm run build:combine-html-css-js && npm run build:remove-unused",
"build:clean": "rm dir-index.html",
"build:remove-style-links": "sed '/<link rel=\"stylesheet\"/d' ./src/dir-index.html > ./base-html.html",
"build:minify-wrap-css": "(echo \"<style>\" && cat ./src/icons.css ./src/style.css | tr -d \"\t\n\r\" && echo && echo \"</style>\") > ./minified-wrapped-style.html",
"build:combine-html-css": "sed '/<\\/title>/ r ./minified-wrapped-style.html' ./base-html.html > ./dir-index.html",
"build:remove-unused": "rm ./base-html.html && rm ./minified-wrapped-style.html"
"build:remove-links": "sed -e '/<script rel=\"script\"/d' -e '/<link rel=\"stylesheet\"/d' ./src/dir-index.html > ./base-html.html",
"build:minify-wrap": "(echo \"<style>\" && cat ./src/icons.css ./src/style.css | tr -d \"\t\n\r\" && echo && echo \"</style>\" && echo \"<script>\" && cat ./src/script.js | tr -d \"\t\n\n\" && echo && echo \"</script>\") > ./minified-wrapped.html",
"build:combine-html-css-js": "sed '/<\\/title>/ r ./minified-wrapped.html' ./base-html.html > ./dir-index.html",
"build:remove-unused": "rm ./base-html.html && rm ./minified-wrapped.html"
}
}
3 changes: 3 additions & 0 deletions assets/dir-index-html/src/dir-index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
<link rel="shortcut icon" href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlo89/56ZQ/8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACUjDu1lo89/6mhTP+zrVP/nplD/5+aRK8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHNiIS6Wjz3/ubFY/761W/+vp1D/urRZ/8vDZf/GvmH/nplD/1BNIm8AAAAAAAAAAAAAAAAAAAAAAAAAAJaPPf+knEj/vrVb/761W/++tVv/r6dQ/7q0Wf/Lw2X/y8Nl/8vDZf+tpk7/nplD/wAAAAAAAAAAAAAAAJaPPf+2rVX/vrVb/761W/++tVv/vrVb/6+nUP+6tFn/y8Nl/8vDZf/Lw2X/y8Nl/8G6Xv+emUP/AAAAAAAAAACWjz3/vrVb/761W/++tVv/vrVb/761W/+vp1D/urRZ/8vDZf/Lw2X/y8Nl/8vDZf/Lw2X/nplD/wAAAAAAAAAAlo89/761W/++tVv/vrVb/761W/++tVv/r6dQ/7q0Wf/Lw2X/y8Nl/8vDZf/Lw2X/y8Nl/56ZQ/8AAAAAAAAAAJaPPf++tVv/vrVb/761W/++tVv/vbRa/5aPPf+emUP/y8Nl/8vDZf/Lw2X/y8Nl/8vDZf+emUP/AAAAAAAAAACWjz3/vrVb/761W/++tVv/vrVb/5qTQP+inkb/op5G/6KdRv/Lw2X/y8Nl/8vDZf/Lw2X/nplD/wAAAAAAAAAAlo89/761W/++tVv/sqlS/56ZQ//LxWb/0Mlp/9DJaf/Kw2X/oJtE/7+3XP/Lw2X/y8Nl/56ZQ/8AAAAAAAAAAJaPPf+9tFr/mJE+/7GsUv/Rymr/0cpq/9HKav/Rymr/0cpq/9HKav+xrFL/nplD/8vDZf+emUP/AAAAAAAAAACWjz3/op5G/9HKav/Rymr/0cpq/9HKav/Rymr/0cpq/9HKav/Rymr/0cpq/9HKav+inkb/nplD/wAAAAAAAAAAAAAAAKKeRv+3slb/0cpq/9HKav/Rymr/0cpq/9HKav/Rymr/0cpq/9HKav+1sFX/op5G/wAAAAAAAAAAAAAAAAAAAAAAAAAAop5GUKKeRv/Nxmf/0cpq/9HKav/Rymr/0cpq/83GZ/+inkb/op5GSAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAop5G16KeRv/LxWb/y8Vm/6KeRv+inkaPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAop5G/6KeRtcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/n8AAPgfAADwDwAAwAMAAIABAACAAQAAgAEAAIABAACAAQAAgAEAAIABAACAAQAAwAMAAPAPAAD4HwAA/n8AAA==" />
<link rel="stylesheet" href="style.css"/>
<link rel="stylesheet" href="icons.css">
<script rel="script" src="script.js" async></script>
Jorropo marked this conversation as resolved.
Show resolved Hide resolved
<title>{{ .Path }}</title>
</head>
<body>
Expand Down Expand Up @@ -70,6 +71,7 @@
</td>
<td></td>
<td></td>
<td></td>
</tr>
{{ range .Listing }}
<tr>
Expand All @@ -79,6 +81,7 @@
<td>
<a href="{{ .Path | urlEscape }}">{{ .Name }}</a>
</td>
<td class="type-icon not-cached-locally"></td>
<td class="no-linebreak">
{{ if .Hash }}
<a class="ipfs-hash" translate="no" href={{ if $root.DNSLink }}"https://cid.ipfs.io/#{{ .Hash | urlEscape}}" target="_blank" rel="noreferrer noopener"{{ else }}"{{ $root.GatewayURL }}/ipfs/{{ .Hash | urlEscape}}?filename={{ .Name | urlEscape }}"{{ end }}>
Expand Down
8 changes: 8 additions & 0 deletions assets/dir-index-html/src/icons.css
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 22 additions & 0 deletions assets/dir-index-html/src/script.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
document.addEventListener('DOMContentLoaded', () => {
const listingsPath = [
{{ range .Listing }}
'{{.Path}}',
{{ end }}
];

const init = {
method: 'GET',
Copy link
Contributor

@Jorropo Jorropo Apr 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we need here is a HEAD request.
HEAD is just like a GET request, but it skips downloading the body of the file, it only sends the headers.

I'm not actually certain this would be work because the server will fetch all directly linked blocks to resolve types and sizes, so without traversal I think every file is gonna show as cached on the gateway because the first block is.
(anyway I would like you to try this pls, as downloading the complete files just to ensure the data is on the server is unacceptable)

headers: {'Accept': 'application/vnd.ipfs.cache.local'}
};

const notCachedHTML = '<div title="File not cached locally" class="ipfs-_attention">&nbsp;</div>';

listingsPath.forEach((element, index) => {
fetch(element, init)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK this request the full file, we dont want to download GiBs just because you openned the dir listing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Even with inexpensive HEAD application/vnd.ipfs.cache* check we don't want to send 10k requests when opening big directory like bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm
    • Even 1k QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm will be wasteful, if only a small subset is visible on the screen.

This should be refactored to fetch status only for visible items – see https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API

.then(response => {
if (response.status === 404)
document.getElementsByClassName('not-cached-locally').item(index).innerHTML = notCachedHTML;
});
});
});
4 changes: 4 additions & 0 deletions assets/dir-index-html/src/style.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,10 @@ td {
width:1.15em
}

.not-cached-locally {
vertical-align: bottom;
}

.no-linebreak {
white-space:nowrap
}
Expand Down