-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add goodbits #187
feat: add goodbits #187
Conversation
44d6a6c
to
3153de2
Compare
Deploying with Cloudflare Pages
|
c7d2f5b
to
0b91346
Compare
0b91346
to
1509c4b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's blocking you, this could be merged as is, but I'd consider doing batching as you read the good bits list as an iterable, and fixing some of the minor issues raised.
|
||
on: | ||
schedule: | ||
- cron: '13 0,5,10,15,20 * * *' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this "every 5 hours at 13 mins past" to avoid the rush of doing things on the hour?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Kept same timings as we did for denylist nftstorage/nft.storage#1761
We can revisit later if we need more/less frequency. No strong opinions for now.
strategy: | ||
matrix: | ||
env: ['staging', 'production'] | ||
timeout-minutes: 20 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this job sometimes get stuck? why 20? seems like it should take a few seconds with a list of 1 thing, and may take hours with a list of 1 billion things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no, this was kind of based on my actions template. But I will decrease it for 5 minutes for now
packages/cron/src/jobs/goodbits.js
Outdated
if (!res.ok) { | ||
throw new Error(`unexpected status fetching goodbits list: ${res.status}`) | ||
} | ||
const list = ndjsonParser(await res.text()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
y u no support your friendly neighbourhood software developers‽
- https://www.npmjs.com/package/iterable-ndjson for modern iterable flavour goodness from alan
- https://www.npmjs.com/package/ndjson the original streaming ndjson from contra & max
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't know about these packages :) Chose the one with better score in npms
, but I will change to Alan's one
|
||
const kvNamespaces = wranglerEnvConfig.kv_namespaces || [] | ||
const goodbitsListKv = kvNamespaces.find( | ||
(kv) => kv.binding === 'GOODBITSLIST' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: other uses of GOODBITS
in this PR use an underscore separator e.g. GOODBITS_SOURCES
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, I went this naming to be consistent with KV for DENYLIST
(no underscore there sadly)
@@ -24,7 +28,17 @@ export async function gatewayGet(request, env) { | |||
}, | |||
}) | |||
|
|||
return getTransformedResponseWithCustomHeaders(response) | |||
// Validation layer - CSP bypass | |||
const resourceCid = decodeURIComponent(response.headers.get('etag') || '') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we store the "this is a good bit" on the cached response rather than having to check it in the KV every time? Like once a cid is added to the list it's very unlikely to be removed... It'd be a little more awkward, but might end up being more similar your proposal in #51 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't gain much with it, or am I missing something?
On the proposal from #51 , we want to purge from cache what we don't want to serve anymore to avoid checking Denylist before serving cached content. Here, we would add temporarily to cache, but given cache is LRU it would eventually leave cache. So, we always need to check KV anyway.
Given this is an optimization, if you have some angle that I am missing, can you please open an issue for us to discuss there?
@@ -33,7 +47,7 @@ export async function gatewayGet(request, env) { | |||
* | |||
* @param {Response} response | |||
*/ | |||
function getTransformedResponseWithCustomHeaders(response) { | |||
function getTransformedResponseWithCspHeaders(response) { | |||
const clonedResponse = new Response(response.body, response) | |||
|
|||
clonedResponse.headers.set( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe this is the plan for a future PR, but I feel like the list of allowed connect-src
and maybe the CSP more generally should be driven by the tags in the goodbits list. Like "CID x
needs to connect to url y
etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We did not talk about this yet. I have that as a question mark on my head as well. We will figure out later as use cases appear :)
} | ||
|
||
// TODO: Remove once https://github.com/nftstorage/nftstorage.link/issues/51 is fixed | ||
const goodbitsEntry = await pRetry(() => datastore.get(cid), { retries: 5 }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we are always going to need retries at this scale. Even if they fix some root issue, we will still see the occasional random failure that can be fixed by asking again. Pesky entropy.
@@ -0,0 +1,18 @@ | |||
import { test, getMiniflare } from './utils/setup.js' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dunno man. I don't think us old developers are ready for filenames with spaces in them, even if the tooling is. 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
loool, I took some time to realize the filename had spaces instead of underscores. Not sure what I did there
t.true(csp.includes("form-action 'self' ; navigate-to 'self';")) | ||
}) | ||
|
||
test('Gets content with no csp header when goodbits csp bypass tag exists', async (t) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should take a moment to see if there is a much more permissive CSP header we can provide rather than dropping it completely...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we will think about that as other use cases rise in the goodbits and with tags. Until we spec a better solution, let's just keep the basics so that users with broken CIDs are not affected :)
Codecov Report
@@ Coverage Diff @@
## main #187 +/- ##
=======================================
Coverage 98.98% 98.98%
=======================================
Files 4 4
Lines 491 491
=======================================
Hits 486 486
Misses 5 5 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Addressed most of the review. Merging this in so that we can actually starting to use the goodbits content and unblock comms around this. We can revisit and specify a spec for goodbits tags when we have better idea on what use cases can exist |
Integrate goodbits into nftstorage.link.
Motivation
We introduced CSP #172 aiming to block phishing websites from operating. More than phishing websites, this measure brings to the table additional long term benefits:
ipfs://
browser handling will require all external sources to also rely on IPFS protocol (we can overcome these pains earlier on)There are already occurrences on the wild of NFTs relying on primitives that are now blocked by default via CSP. We are shipping https://github.com/nftstorage/goodbits as a place where we can manually add these NFTs, in order to bypass CSP for them after manual validation.
Implementation details
Similarly to what we do in reads pipeline for denylist, a new KV is created for optimal access from CF Workers on the gateway operation.
This PR includes:
Needs: