booster-http: implement IPFS HTTP gateway

[dirkmc]

Implements the IPFS gateway API in booster-http

New functionality in booster-http:

• Serve files as CAR files
• Serve the original file (eg video)
• Serve individual blocks in a file
• Filter out blocks containing badbits
• Enable / disable serving raw piece data at /pieces
• Enable / disable serving IPFS API gateway at /ipfs

Example: Serving the original file from a path in a CAR file

booster-http -vv run --api-boost=$BOOST_API_INFO --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO
2023-03-10T16:25:11.118+0100	INFO	booster	booster-http/run.go:266	Using boost API at ws://127.0.0.1:1288/rpc/v0
2023-03-10T16:25:11.119+0100	INFO	booster	lib/api.go:37	Using full node API at ws://127.0.0.1:1234/rpc/v1
2023-03-10T16:25:11.120+0100	INFO	booster	lib/api.go:63	Using storage API at ws://127.0.0.1:2345/rpc/v0
2023-03-10T16:25:11.121+0100	INFO	booster	lib/api.go:105	Miner address: f01000
2023-03-10T16:25:11.169+0100	INFO	booster	booster-http/run.go:184	Starting booster-http node on port 7777 with base path ''
2023-03-10T16:25:11.170+0100	INFO	booster	booster-http/run.go:192	serving IPFS gateway at /ipfs/
2023-03-10T16:25:11.170+0100	INFO	booster	booster-http/run.go:197	serving raw pieces at /piece/
2023-03-10T16:25:13.466+0100	DEBUG	remote-blockstore	remoteblockstore/remoteblockstore.go:47	Get	{"cid": "bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm"}
2023-03-10T16:25:13.467+0100	DEBUG	remote-blockstore	remoteblockstore/remoteblockstore.go:50	Get response	{"cid": "bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm", "size": 58, "error": null}
2023-03-10T16:25:13.467+0100	INFO	remote-blockstore	remoteblockstore/remoteblockstore.go:56	Get	{"cid": "bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm", "size": 58}
2023-03-10T16:25:13.468+0100	DEBUG	remote-blockstore	remoteblockstore/remoteblockstore.go:47	Get	{"cid": "bafybeibk3k3lqqjixzqwohsdi4flj45odotrler4zxf74745udhqns2qlq"}
2023-03-10T16:25:13.471+0100	DEBUG	remote-blockstore	remoteblockstore/remoteblockstore.go:50	Get response	{"cid": "bafybeibk3k3lqqjixzqwohsdi4flj45odotrler4zxf74745udhqns2qlq", "size": 139, "error": null}
2023-03-10T16:25:13.471+0100	INFO	remote-blockstore	remoteblockstore/remoteblockstore.go:56	Get	{"cid": "bafybeibk3k3lqqjixzqwohsdi4flj45odotrler4zxf74745udhqns2qlq", "size": 139}
2023-03-10T16:25:13.471+0100	DEBUG	remote-blockstore	remoteblockstore/remoteblockstore.go:47	Get	{"cid": "bafybeicjmpqebfdqxbqipitj5m6egfrnbp5kxkfzuvgqxv72wvusmcsf7e"}
2023-03-10T16:25:13.474+0100	DEBUG	remote-blockstore	remoteblockstore/remoteblockstore.go:50	Get response	{"cid": "bafybeicjmpqebfdqxbqipitj5m6egfrnbp5kxkfzuvgqxv72wvusmcsf7e", "size": 9255, "error": null}
2023-03-10T16:25:13.474+0100	INFO	remote-blockstore	remoteblockstore/remoteblockstore.go:56	Get	{"cid": "bafybeicjmpqebfdqxbqipitj5m6egfrnbp5kxkfzuvgqxv72wvusmcsf7e", "size": 9255}

New Command Line Parameters

Several new command line parameters have been added to booster-http run:

   --serve-pieces                                           enables serving raw pieces (default: true)
   --serve-blocks                                           serve blocks with the ipfs gateway API (default: true)
   --serve-cars                                             serve CAR files with the ipfs gateway API (default: true)
   --serve-files                                            serve original files (eg jpg, mov) with the ipfs gateway API (default: false)
   --api-filter-endpoint value                              the endpoint to use for fetching a remote retrieval configuration for bitswap requests
   --api-filter-auth value                                  value to pass in the authorization header when sending a request to the API filter endpoint (e.g. 'Basic ~base64 encoded user/pass~'
   --badbits-denylists value [ --badbits-denylists value ]  the endpoints for fetching one or more custom BadBits list instead of the default one at https://badbits.dwebops.pub/denylist.json (default: "https://badbits.dwebops.pub/denylist.json")
   --help, -h                                               show help

Tests:

• Serve CAR file
• Serve original file
• Check badbits filter blocks requests for a filtered cid

Example badbits filter:

[
  {"anchor":"c03f80a19a7ca00929cbcd0181204febbc625fb4b38f51125af3074033d4140c"}
]

fixes #960

[LexLuthr]

We need to bump the "info" version as this is a major updated to http endpoints.

[jacobheun]

Edit: nevermind, see below comment

---

It looks like there is an issue fetching whole car files, I've got this set up locally and rebased on main. I've got a car with cid bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw stored, created with the random deal command.

The latest version of lassie is able to fetch the data over graphsync and bitswap, validated with car inspect, but curling the file causes car inspect to fail.

Looks like it's missing some data:

-rw-r--r--   5120000 Mar 23 18:34 out.car #Curl
-rw-r--r--   5129411 Mar 23 18:26 out2.car
-rw-r--r--   5129411 Mar 23 18:25 out3.car

##### Bitswap #####
$ ./lassie fetch --protocols bitswap -o out3.car --provider=/ip4/127.0.0.1/tcp/8888/p2p/12D3KooWKQy3xToW1Nog6TSsUWEh1Yx8hYwFVF3qbiL4NBegf8pm bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
Fetching bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw from [{12D3KooWKQy3xToW1Nog6TSsUWEh1Yx8hYwFVF3qbiL4NBegf8pm: [/ip4/127.0.0.1/tcp/8888]}].....................................................................................................
Fetched [bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw] from []:
	Duration: 1.203005535s
	  Blocks: 101
	   Bytes: 4.9 MiB

$ car inspect out3.car 
Version: 1
Roots: bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
Root blocks present in data: Yes
Block count: 101
Min / average / max block length (bytes): 5210 / 50744 / 51200
Min / average / max CID length (bytes): 38 / 38 / 38
Block count per codec:
	raw: 100
	dag-pb: 1
CID count per multihash:
	blake2b-256: 101


##### Graphsync #####
$ ./lassie fetch --protocols graphsync -o out2.car --provider=/ip4/127.0.0.1/tcp/50000/p2p/12D3KooWHASLRJuDarGfEiAgBnDoTe7phuMsZDp3avjpLT3hEvNy bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
Fetching bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw from [{12D3KooWHASLRJuDarGfEiAgBnDoTe7phuMsZDp3avjpLT3hEvNy: [/ip4/127.0.0.1/tcp/50000]}].....................................................................................................
Fetched [bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw] from [12D3KooWHASLRJuDarGfEiAgBnDoTe7phuMsZDp3avjpLT3hEvNy]:
	Duration: 201.760668ms
	  Blocks: 101
	   Bytes: 4.9 MiB

$ car inspect out2.car 
Version: 1
Roots: bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
Root blocks present in data: Yes
Block count: 101
Min / average / max block length (bytes): 5210 / 50744 / 51200
Min / average / max CID length (bytes): 38 / 38 / 38
Block count per codec:
	raw: 100
	dag-pb: 1
CID count per multihash:
	blake2b-256: 101


##### curl #####
$ curl -s -o out.car http://localhost:7777/ipfs/bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
$ car inspect out.car 
2023/03/23 18:31:13 invalid header: Invalid majorByte: 0xe1

[jacobheun]

Ah, I remembered this requires headers. We should enumerate the curl's this supports so we can get them added to the docs.

$ curl -s -H "Accept:application/vnd.ipld.car;" -o out.car http://localhost:7777/ipfs/bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw

$ car inspect out.car 
Version: 1
Roots: bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
Root blocks present in data: Yes
Block count: 101
Min / average / max block length (bytes): 5210 / 50744 / 51200
Min / average / max CID length (bytes): 38 / 38 / 38
Block count per codec:
	raw: 100
	dag-pb: 1
CID count per multihash:
	blake2b-256: 101

[jacobheun]

The code looks good! The only other thing I saw aside from documenting the available curls is that the booster-http dashboard doesn't show anything for retrievals (runtime dashboard is fine).

I ran all of the following several times but only got goroutines

# car file
curl -s -H "Accept:application/vnd.ipld.car;" -o /dev/null http://localhost:7777/ipfs/bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
# raw, root block
curl -s -H "Accept:application/vnd.ipld.raw;" -o /dev/null http://localhost:7777/ipfs/bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
# raw content
curl -s -o /dev/null http://localhost:7777/ipfs/bafykbzacecyrmvxl7shftg5qpebpkrdtdzihjnoymc7voyiyomvaydomddxcw
# whole piece
curl -o /dev/null http://localhost:7777/piece/baga6ea4seaqexju64hfftlf5xsvyjahfv2ly3czi2lhrmhleg7xshyfmpcfj2ly

[dirkmc]

I added some metrics:

[jacobheun]

Awesome work, this looks good! I ran some smoke tests locally and verified files are disabled by default and that setting --serve-files=true successfully enables it.