booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted

[rvagg]

This is a bit of a PoC at the moment; enables a mini Trustless Gateway implementation when you only want to serve CARs through /ipfs/. Uses Frisbii but I've copied in the two main modules that are used into this PR because of the libp2p dependency pain that gets stuck at ipni dependencies in Frisbii which are not required here. So cmd/booster-http/frisbii/* are (mostly) unmodified copies from github.com/ipld/frisbii - we only need the handler and the CAR streamer for this.

The transparent switch in implementations by subtle CLI argument changes is probably not optimal for users, but I've got a big integration test in here that pulls in the big trustless pathing test fixture that exercises a whole lot of trustless path retrievals for a fixed 20Mb CAR (which we import across multiple deals here) and it gets run across both implementations in here with the same boost backend, just two different booster-http frontends.

The reason this might be useful are that (1) it's super minimal and very lightweight; there's no intermediate caching of blocks while it builds up state, it just does a traversal straight from the backend blockstore and spits it out as a CAR, (2) it gives us a bit more freedom to experiment with alternative outputs, like what SPARK is after, and (3) gives us useful error output if there's a problem loading the root block, instead of the current empty CAR that we get.

• Update docs

[dirkmc]

Thanks for putting this together @rvagg

We actually have a PR that simplifies the parameters to booster-http, and takes advantage of the boxo gateway's trustless gateway: feat: add support for deserialized response

How would pulling in this frisbii implementation differ from using the boxo gateway?

[willscott]

This direction was well-received at the fil dev summit last week.

• having a separate bifrost-gateway port / process for full files, assuming we have documentation for that install, allows the two endpoints to have different ACLs
• not having the full boxo dependencies directly in booster-http lets us be a bit less worried about dependency versioning.

@rvagg - what's needed to get this to a point where it could be ready for review?

[rvagg]

Changed so this removes --serve-blocks and --serve-files, leaving only --serve-pieces and --serve-cars - http piece, and http trustless.

The whole frisbii directory will go when the ipnisync branch gets merged and we get all the updated dependencies. For now it's an up to date fork of the frisbii components being used.

[rvagg]

removed the frisbii fork now that the libipni branch is merged and dependencies updated so we have no conflicts!

[LexLuthr]

Does this require docs changes? If yes, then can we add some rough documentation to this PR and I can update boost docs.

[rvagg]

yes, working on it, first I'll do so by getting an integration test in here that uses bifrost-gateway to demonstrate that it can be used on top of this, then we can add that to the docs as well so all options are covered.

[rvagg]

includes a full e2e test now that involves bifrost-gateway and lassie to perform some trustless and trusted fetching, these are both installed on demand and are not direct dependencies; I have some cleanup to do before this is ready but it's mostly there now

[rvagg]

Rebased and merged in #1736 and #1737 to this and simplified the commits significantly. There's 4 commits now - the main switch to frisbii, the fixes for compression and addition of --compression-level, the addition of --log-file and removal of custom logging in here, and the refactor of the piece handler to a separate file.

The bulk of the changes here are the addition of extra tests, including an e2e one involving bifrost-gateway to exercise trusted gateway support, and the refactoring of the piece handler does create some annoying diff noise, but it's last in the commit list so you could review that separately to avoid the noise—there are a bunch of changes to the piece handler to support logging and compression that are masked by the refactor.

Further comments about compression are in #1736 and logging, including example output, is in #1737.

[rvagg]

Updated to frisbii@v0.4.1 for a minor byte-range fix but this also brings in some other deps - notably graphsync v0.15, which thankfully doesn't cause problems here now because of the libp2p update (which was the main difference between the head versions on v0.14 and v0.15) and go-graphsync is only used for testing, with boost-graphsync used for builds here.

Also brings in various golang.org/x/ upgrades which come along with golang.org/x/net@v0.17.0 for the http/2 security fixes ipld/frisbii#63

[LexLuthr]

Pending AI to merge in main:

• Rebase on the latest main
• Test in devnet for multi-miner retrievals
• Test in prod for HTTP retrievals
• Perf test to compare with the older HTTP server

[LexLuthr]

DEVNET Testing:

lexluthr@Lexs-Mac ~ % curl -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/piece/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8128k  100 8128k    0     0   100M      0 --:--:-- --:--:-- --:--:--  107M
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/piece/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /piece/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: public, max-age=29030400, immutable
< Content-Length: 8323072
< Content-Type: application/piece
< Etag: "baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy"
< Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
< Vary: Accept-Encoding
< Date: Wed, 25 Oct 2023 11:03:45 GMT
< 
{ [13901 bytes data]
100 8128k  100 8128k    0     0   107M      0 --:--:-- --:--:-- --:--:--  115M
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % l              
zsh: command not found: l
lexluthr@Lexs-Mac ~ % la
zsh: command not found: la
lexluthr@Lexs-Mac ~ % ls b
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
8.1M	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Vary: Origin
< Vary: Accept-Encoding
< Date: Wed, 25 Oct 2023 11:04:25 GMT
< Content-Length: 135
< Content-Type: text/plain; charset=utf-8
< 
{ [135 bytes data]
100   135  100   135    0     0  15180      0 --:--:-- --:--:-- --:--:-- 33750
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car                                                                                                
4.0K	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea.car"
< Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
< Etag: "bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea.car.bjhgn9hgie2db"
< Vary: Accept, Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea
< Date: Wed, 25 Oct 2023 11:04:52 GMT
< Transfer-Encoding: chunked
< 
{ [13680 bytes data]
100 51302    0 51302    0     0  1141k      0 --:--:-- --:--:-- --:--:-- 1318k
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car                                                                                              
52K	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76 HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76.car"
< Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
< Etag: "bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76.car.9e13ncrk162k"
< Vary: Accept, Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76
< Date: Wed, 25 Oct 2023 11:05:24 GMT
< Transfer-Encoding: chunked
< 
{ [3545 bytes data]
100 5009k    0 5009k    0     0  8289k      0 --:--:-- --:--:-- --:--:-- 8348k
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car                                                                                              
5.1M	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car

[LexLuthr]

Tested on filcollins:

lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai http://localhost:7777/piece/baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /piece/baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: public, max-age=29030400, immutable
< Content-Length: 34091302912
< Content-Type: application/piece
< Etag: "baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai"
< Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
< Vary: Accept-Encoding
< Date: Wed, 25 Oct 2023 11:55:08 GMT
< 
{ [3761 bytes data]
  0 31.7G    0 27.3M    0     0  3753k      0  2:27:50  0:00:07  2:27:43 5498k^C
lexluthr@Lexs-Mac ~ % 
lexluthr@Lexs-Mac ~ % 
lexluthr@Lexs-Mac ~ % 
lexluthr@Lexs-Mac ~ % curl -v -o bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca http://localhost:7777/piece/bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /piece/bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Vary: Origin
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< Date: Wed, 25 Oct 2023 11:56:25 GMT
< Content-Length: 103
< 
{ [103 bytes data]
100   103  100   103    0     0    179      0 --:--:-- --:--:-- --:--:--   180
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca 
4.0K	bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca
lexluthr@Lexs-Mac ~ % cat bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca 
"getting piece metadata: piece bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca: not found"
lexluthr@Lexs-Mac ~ % curl -v -o bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca http://localhost:7777/ipfs/bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi.car"
< Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
< Etag: "bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi.car.76dbvvp97jlgt"
< Vary: Accept, Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi
< Date: Wed, 25 Oct 2023 11:57:17 GMT
< Transfer-Encoding: chunked
< 
{ [32225 bytes data]
100 1024k    0 1024k    0     0   429k      0 --:--:--  0:00:02 --:--:--  429k
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca                                                                                           
1.1M	bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca

[LexLuthr]

All tests are complete. I added some fixes for devnet.
I think we are good to merge.