feat: built-in content blocking based on IPIP-383 #10161
Conversation
Fixes #8492. This introduces "nopfs" as a preloaded plugin into Kubo. It automatically make Kubo watch *.deny files found in: - /etc/ipfs/denylists - $XDG_CONFIG_HOME/ipfs/denylists - $IPFS_PATH/denylists (files need to be present before boot in order to be watched). Debug logging can be enabled with `GOLOG_LOG_LEVEL="nopfs=debug"`. All blocks are logged to "nopfs-blocks", so logging requests for blocked content can be achieved with `GOLOG_LOG_LEVEL="nopfs-blocks=warn"`: ``` WARN (...) QmRFniDxwxoG2n4AcnGhRdjqDjCM5YeUcBE75K8WXmioH3: blocked (test.deny:9) ``` Interactive/gateway users will also receive errors as responses but with less details: ``` Error: /ipfs/QmQvjk82hPkSaZsyJ8vNER5cmzKW7HyGX5XVusK7EAenCN is blocked and cannot be provided ``` One particularity to keep in mind is that GetMany() will silently drop blocked blocks from the response (a warnings are logged). AddMany() will act similarly and avoid adding blocked blocks. The code implementing all this is actually in nopfs: - https://github.com/ipfs-shipyard/nopfs (main library) - https://github.com/ipfs-shipyard/nopfs/tree/master/ipfs (wrappers) The interpretation of the list rules and block detection is well tested, but a general review might be in order.
|
Added docs, now I am testing + applying changes caused by recent boxo reactor, will open relevant PRs later today. |
This applies changes necessary after ipfs/boxo#459 we need this to unblock ipfs/kubo#10161
Includes dependency update from ipfs-shipyard/nopfs#25
|
I've added tests and the Repro: blocking the root CID of wikipedia, and everything under it: $ cat $IPFS_PATH/denylists/text.deny
/ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/*It correctly blocks things in CLI: $ ipfs resolve /ipns/en.wikipedia-on-ipfs.org/wiki/
Error: /ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/wiki is blocked and cannot be providedbut the gateway returns blocked payload, bypasses the rule somehow: $ curl -is http://127.0.0.1:8080/ipns/en.wikipedia-on-ipfs.org/wiki/ | grep HTTP
HTTP/1.1 200 OK |
CLI works as expected, gateway does not respect the rule (needs investigation)
Yes, I think this is because the gateway here is created without passing the Kubo resolver, but it creates one internally. This is why I added the WithResolver() option. On rainbow currently this works as expected: We can fix this once latest boxo has bubbled to Kubo. |
|
@hsanjuan you can use Boxo's latest commit in main here. We allow Kubo to use a commit version of Boxo and then at release time we ensure that it has a tagged version. |
ah ok, I thought path changes etc had not been bubbled. Can do. |
cosmetic change to error message, this should be more robust
|
@lidel I updated nopfs to a commit that has both ipfs-shipyard/nopfs#28 and ipfs-shipyard/nopfs#27 merged. Eventually we should produce tags for boxo, nopfs and bubble that. |
this ensures we test something other than default sha256 Ref. ipfs-shipyard/nopfs#28
adds missing tests for "no fetch" gateways one can expose, in both cases the offline mode is done by passing custom blockservice/exchange into path resolver, which means global path resolver that has nopfs intercept is not used, and the content blocking does not happen on these gateways. needs to be fixed, but at least now we have tests that fail until it is fixed.
|
Close, but not ready yet: content blocking is not applied to "no fetch" gateways. c99068e adds missing tests for "no fetch" gateways one can expose. In both cases the "no fetch" (non-recursive mode) is implemented by passing custom blockservice/exchange into path resolver, which means global path resolver that has nopfs intercept is not used, and the content blocking does not happen on these gateways:
Needs to be fixed before merge, but at least now we have tests that fail on this PR to surface the problem. I'll look into this on Friday, but if I run out of time before code freeze, this might slip from 0.24 to 0.25. |
this fixes the problem described in #10161 (comment) by adding explicit offline path resolvers that are backed by offline exchange, and using them in NoFetch gateways instead of the default online ones
this fixes the problem described in #10161 (comment) by adding explicit offline path resolvers that are backed by offline exchange, and using them in NoFetch gateways instead of the default online ones
this is the minimum we need right now to make content blocking from ipfs/kubo#10161 return HTTP 410 on rule match
this is the minimum we need right now to make content blocking from ipfs/kubo#10161 return HTTP 410 on rule match
requires ipfs/boxo#497 which is based on top of the boxo already used in kubo master to avoid issues caused by ilater commits in boxo main
There was a problem hiding this comment.
Switched this PR to commits that are in main branches of boxo and nopfs.
TODOs are addressed, and tests pass. Gateway returns HTTP 410.
I am merging this to ensure it is included in 0.24-rc1, allowing wider community to test and provide feedback.
We will switch to tagged release versions before final 0.24.
Once again, thank you @hsanjuan for work on NOpfs and the spec, made this possible.
lidel
changed the title
Fixes #8492.
This introduces "nopfs" as a preloaded plugin into Kubo.
It automatically make Kubo watch *.deny files (syntax from IPIP-383) found in:
/etc/ipfs/denylists$XDG_CONFIG_HOME/ipfs/denylists$IPFS_PATH/denylists(files need to be present before starting in order to be watched).
Debug logging can be enabled with
GOLOG_LOG_LEVEL="nopfs=debug".All blocks are logged to "nopfs-blocks", so logging requests for blocked content can be achieved with
GOLOG_LOG_LEVEL="nopfs-blocks=warn":Interactive/gateway users will also receive errors as responses but with less details:
One particularity to keep in mind is that GetMany() will silently drop blocked blocks from the response (warnings are logged). AddMany() will act similarly and avoid adding blocked blocks.
The code implementing all this is actually in nopfs:
The interpretation of the list rules and block detection is well tested, but a general review might be in order.
TODO
test/cli/content_blocking_test.go)test/cli/content_blocking_test.go)dag exportaccept CID or Path like all other commandstest/cli/content_blocking_test.go