booster-http: disable on-the-fly indexing by default

[dirkmc]

No description provided.

[jacobheun]

What's the consequence of having this disabled? I get the resource savings here by having it off, but if this is disabled that risks a failed retrieval, correct?

If that's the case I think we should default to "make it work out of the box", and then document and allow tuning for performance via params, noting the trade off of shutting certain things off.

[dirkmc]

A piece is stored in a sector.
A CAR file is stored in a piece (the piece contains the CAR file + some padding).

In order to retrieve a CAR file from a sector, we need to know

• where the piece is within the sector
• where the end of the CAR file is within the piece

                            v
[sectorsectorsector{<carfile>piecepadding}sectorsector]

We figure out where the end of the file CAR file is by

• getting the CAR file index from the DAG store
• finding the highest offset in the index
• calculating the size of the last CAR file segment

Indexing-on-the-fly is when

• a client makes a request to retrieve a CAR file
• the piece has not finished indexing
• we build an index of the CAR file on-the-fly in the booster-http process
• we follow the above steps to figure out where the end of the CAR file is

In practice this process is relatively expensive, because booster-http needs to retrieve the entire piece and index it. If there are lots of retrieval requests booster-http will do this repeatedly.

So I think it makes sense to disable this functionality by default. When the client makes a request for a CAR file in a piece that has not been indexed, the SP should notice errors in their logs and manually run commands to index the piece.

[jacobheun]

I'm assuming the manual intervention that's needed here is if indexing fails, is that correct? If it's processing and will eventually succeed I think that's fine. I'd like to understand what the manual intervention pieces of this look like so that we can try and make that more prevalent to SP's, or better yet, automatic.

[dirkmc]

Yes the manual intervention is to fix indexing if it fails 👍

The actual steps depend on exactly went wrong. It could be that the sealing node was down when indexing was attempted, a corrupted datastore, a corrupted CAR file, etc. I think we should address those as part of the boost doctor work.

[jacobheun]

yeah, that seems reasonable to focus that on showcasing with the doctor effort.

[ribasushi]

✔️