Support Data Segment Index

[willscott]

Boost component

• boost daemon - storage providers

Improvement Suggestion

Currently boost indexes pieces using go-car to generate a local index of CIDs contained in the piece for use in retrievals and to notify network indexers.

This format has been extended recently, per the go-data-segment library. This wrapper can be used to parse pieces, in particular by parsing a trailing index if it exists.

as @Kubuxu notes:

Here is the code for parsing the data segment index: https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/parse_index.go
After the index is parsed, you should use IndexData#ValidEntries()` to gather valid data segments on which you can access information of each segment using PieceCID() UnpaddedOffset() UnpaddedLength(): https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/index.go#L156-L173

This is currently managed by the dagstore
I think the lid variant will be this? (can you confirm, @nonsense?)

[honghaoq]

Tracking link (from Jake): #573

[willscott]

This work is progressing in #1495