Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: improved Origin detection via img tag #117

Merged
merged 3 commits into from
Oct 21, 2020
Merged

feat: improved Origin detection via img tag #117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5bae793
feat: improved Origin detection via img tag
lidel Oct 20, 2020
2a80dfe
style: replace 🔒 with 💚
lidel Oct 21, 2020
ac2dbac
refactor: review feedback
lidel Oct 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 67 additions & 33 deletions app.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
/*
This program will check IPFS gateways status using 2 methods
This program will check IPFS gateways status using 3 methods
1) By asking for a script through a <script src=""> tag, which when loaded, it executes some code
2) By asking data through ajax requests to verify gateway's CORS configuration
2) By asking data through fetch requests to verify gateway's CORS configuration
3) By asking data through img requests to verify subdomain configuration
*/

const HASH_TO_TEST = 'bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m';
const SCRIPT_HASH = 'bafybeietzsezxbgeeyrmwicylb5tpvf7yutrm3bxrfaoulaituhbi7q6yi';
const IMG_HASH = 'bafybeibwzifw52ttrkqlikfzext5akxu7lz4xiwjgwzmqcpdzmp3n5vnbe'; // 1x1.png
const HASH_STRING = 'Hello from IPFS Gateway Checker';

const ipfs_http_client = window.IpfsHttpClient({
Expand All @@ -19,13 +21,13 @@ let checker = document.getElementById('checker');
checker.nodes = [];

checker.checkGateways = function(gateways) {
gateways.forEach((gateway) => {
let node = new Node(this.results, gateway, this.nodes.length);
this.nodes.push(node);
this.results.append(node.tag);
node.check();
});
};
for (const gateway of gateways) {
const node = new Node(this.results, gateway, this.nodes.length)
this.nodes.push(node)
this.results.append(node.tag)
setTimeout(() => node.check(), 100 * this.nodes.length);
}
}

checker.updateStats = function() {
this.stats.update();
Expand Down Expand Up @@ -78,6 +80,36 @@ let Status = function(parent, index) {
this.tag.textContent = '🕑';
};


function checkViaImgSrc (imgUrl) {
// we check if gateway is up by loading 1x1 px image:
// this is more robust check than loading js, as it won't be blocked
// by privacy protections present in modern browsers or in extensions such as Privacy Badger
const imgCheckTimeout = 15000
return new Promise((resolve, reject) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

load events + timers + promises is some sticky business...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed: entire app.js needs to be re-written from scratch in modern JS (painfully needs rate-limiting via execution queues, similar to ones we have in js-libp2p-delegated-*)

@jessicaschilling just like with cid.ipfs.io, we should rewrite this app (both backend and frontend) at some point (2021 Q1/Q2-ish?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidel There's a long tail of improvements to the gateway checker that never got completed here: #93

I'd suggest seeing what of that could be salvaged, or at least using that as the base issue for a rewrite. I'll add the notes from this PR to that issue.

const timeout = () => {
if (!timer) return false
clearTimeout(timer)
timer = null
return true
}
let timer = setTimeout(() => { if (timeout()) reject() }, imgCheckTimeout)
const img = new Image()
img.onerror = () => {
timeout()
reject()
}
img.onload = () => {
timeout()
resolve()
}
// now - ensures we don't read from browser cache
// filename - ensures correct content-type is returned / sniffed
// x-ipfs-companion-no-redirect - hint for our browser extension, makes sure we test remote server
img.src = `${imgUrl}?now=${Date.now()}&filename=1x1.png#x-ipfs-companion-no-redirect`
lidel marked this conversation as resolved.
Show resolved Hide resolved
})
}

Status.prototype.check = function() {
let gatewayAndScriptHash = this.parent.gateway.replace(":hash", SCRIPT_HASH);

Expand Down Expand Up @@ -134,23 +166,20 @@ let Cors = function(parent) {
};

Cors.prototype.check = function() {
const gatewayAndHash = this.parent.gateway.replace(':hash', HASH_TO_TEST);
const now = Date.now();
const testUrl = `${gatewayAndHash}?now=${now}#x-ipfs-companion-no-redirect`;
fetch(testUrl).then((res) => {
return res.text();
}).then((text) => {
const matched = (HASH_STRING === text.trim());
if (matched) {
this.parent.checked();
this.tag.textContent = '✅';
} else {
this.onerror();
}
}).catch((err) => {
this.onerror();
});
};
const gatewayAndHash = this.parent.gateway.replace(':hash', HASH_TO_TEST)
const now = Date.now()
const testUrl = `${gatewayAndHash}?now=${now}#x-ipfs-companion-no-redirect`
fetch(testUrl).then((res) => res.text()).then((text) => {
const matched = (HASH_STRING === text.trim())
if (matched) {
this.parent.checked()
this.tag.textContent = '✅'
this.parent.tag.classList.add('cors')
} else {
this.onerror()
}
}).catch((err) => this.onerror())
}

Cors.prototype.onerror = function() {
this.tag.textContent = '❌';
Expand All @@ -164,13 +193,18 @@ let Origin = function(parent) {
};

Origin.prototype.check = function() {
const cidInSubdomain = this.parent.gateway.startsWith('https://:hash.ipfs.');
if (cidInSubdomain) {
this.tag.textContent = '✅';
} else {
this.onerror();
}
};
// we are unable to check url after subdomain redirect because some gateways
// may not have proper CORS in place. instead, we manually construct subdomain
// URL and check if it loading known image works
const imgUrl = new URL(this.parent.gateway)
imgUrl.pathname = '/'
imgUrl.hostname = `${IMG_HASH}.ipfs.${imgUrl.hostname}`
checkViaImgSrc(imgUrl.toString()).then((res) => {
this.tag.textContent = '✅';
this.parent.tag.classList.add('origin')
this.parent.checked()
}).catch(() => this.onerror())
}

Origin.prototype.onerror = function() {
this.tag.textContent = '❌';
Expand Down
11 changes: 5 additions & 6 deletions gateways.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[
"https://ipfs.io/ipfs/:hash",
"https://:hash.ipfs.dweb.link",
"https://dweb.link/ipfs/:hash",
"https://gateway.ipfs.io/ipfs/:hash",
"https://ipfs.infura.io/ipfs/:hash",
"https://ninetailed.ninja/ipfs/:hash",
Expand All @@ -10,8 +10,8 @@
"https://hardbin.com/ipfs/:hash",
"https://gateway.blocksec.com/ipfs/:hash",
"https://cloudflare-ipfs.com/ipfs/:hash",
"https://:hash.ipfs.cf-ipfs.com",
"https://ipns.co/:hash",
"https://cf-ipfs.com/ipfs/:hash",
"https://ipns.co/ipfs/:hash",
"https://ipfs.mrh.io/ipfs/:hash",
"https://gateway.originprotocol.com/ipfs/:hash",
"https://gateway.pinata.cloud/ipfs/:hash",
Expand All @@ -29,12 +29,11 @@
"https://permaweb.io/ipfs/:hash",
"https://ipfs.stibarc.com/ipfs/:hash",
"https://ipfs.best-practice.se/ipfs/:hash",
"https://:hash.ipfs.2read.net",
"https://2read.net/ipfs/:hash",
"https://ipfs.2read.net/ipfs/:hash",
"https://storjipfs-gateway.com/ipfs/:hash",
"https://ipfs.runfission.com/ipfs/:hash",
"https://trusti.id/ipfs/:hash",
"https://:hash.ipfs.cosmos-ink.net",
Copy link
Member Author

@lidel lidel Oct 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reached out to @LinusCDE, and we are working on a fix, but for now its better to remove it to reduce confusion.
We will add it back when it supports proper subdomain isolation and interop with path-gateways

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Something came up, so it's good to remove it for now. I'll try to address it tomorrow.
Would a redirect from https://cosmos-ink.net/ipfs/<cid> and https://ipfs.cosmos-ink.net/ipfs/<cid> to https://<cid>.ipfscosmos-ink.net/ to the trick?

What about the IPNS? Same for that?

Copy link
Member Author

@lidel lidel Oct 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, redirects will work (ideally, redirect returned by go-ipfs, because that gives you CID conversion for free)

For ipns you could use https://<libp2p-key>.ipns.cosmos-ink.net/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidel I've now added a rule that should fix that.

if ($request_uri ~ ^/(?<prefix>ipfs|ipns)/(?<full_path>(?<cid_1>.*?)(?=/)(?<cid_1_path>/?.*)|(?<cid_2>.*))$) {
    # Either $cid_1 or $cid_2 will be empty. When $cid_1 is used, $cid_1_path is not empty as well
    return 301 "https://$cid_1$cid_2.$prefix.cosmos-ink.net$cid_1_path";
  }

This nginx rule is applied to ipfs.cosmos-ink.net, ipns.cosmos-ink.net, cosmos-ink.net.
Basicially for any path like https://(ipfs.)cosmos-ink.net/ipfs/<cid>/<otherstuff> I 301 it to https://<cid>.ipfs.cosmos-ink.net/<otherstuff>. Same goes for ipns.

I also added your PublicGateway suggestion from your second mail.

ideally, redirect returned by go-ipfs, because that gives you CID conversion for free

Running it on my Pi (cosmos-ink.net main domain) would be probably more trouble at this point, because I would need to rebuild the docker-image for armv7 or aarch64 support. But for any conversion that isn't https://(ipfs|ipns).cosmos-ink.net/(ipfs/ipns)/..., go-ipfs can feel free to convert that.

For the ipfs/ipns subdomains I also added two entries to prevent using that redirection rule when the cid is already in the subdomain (to allow https://<cid>.ipfs.cosmos-ink.net/ipfs/... in case ipfs web apps have a directory called "ipfs").

Could you check whether it now works as expected? I'll probably then create a PR to re-add the entry.

Copy link
Member Author

@lidel lidel Oct 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works fine for CIDv1, but redirect alone is not enough if someone passes CIDv0.

To illustrate, if you open https://cosmos-ink.net/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR it should redirect to https://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.cosmos-ink.net/
(go-ipfs takes care of CID conversion, without the cid conversion the Qm.. CIDv0 will be force-lowercased and fail).

ps. Do you mind commenting in ipfs/kubo#4931 regarding the need for docker image working out of the box on rpi?

"https://ipfs.overpi.com/ipfs/:hash",
"https://ipfs.lc/ipfs/:hash",
"https://ipfs.leiyun.org/ipfs/:hash",
Expand All @@ -52,7 +51,7 @@
"https://ipfs.fleek.co/ipfs/:hash",
"https://ipfs.jbb.one/ipfs/:hash",
"https://ipfs.yt/ipfs/:hash",
"https://:hash.ipfs.jacl.tech",
"https://jacl.tech/ipfs/:hash",
"https://hashnews.k1ic.com/ipfs/:hash",
"https://ipfs.vip/ipfs/:hash",
"https://ipfs.k1ic.com/ipfs/:hash"
Expand Down
4 changes: 4 additions & 0 deletions styles.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,7 @@ div.Node div.Took {
font-size: 80%;
font-style: italic;
}

div.Node.origin div.Link::after {
content: " 💚"
}