consider easier ways to let users use a CAR index directly

[mvdan]

Right now, an easy way to transparently use the index is to go through the blockstore package. Ideally we don't want that to be the main option, though - many use cases don't need a blockstore and its extra abstraction layer.

It should be possible to efficiently use an index from a CAR file on disk via OpenReader. The best option right now is to go through Reader.IndexReader and index.ReadFrom. Unfortunately, it has some shortcomings:

1. Always loads the entire index into memory. This is rather wasteful if one just wants to loop over all index entries once, for example. Plus, OpenReader already uses an mmap, which allows for fast sequential or random access.

2. No ability to inspect information about the index. For example, how do I tell if a CAR file has a multihash sorted index? Right now the API just allows this by loading the entire index into memory.

3. Has some footguns; for example, it's a bit too easy to call index.ReadFrom straight on Reader.IndexReader, forgetting that it may be nil.

I'm working around these in the indexer provider, but I'll probably backport some of it in the form of reusable APIs.

This issue would enable #222, I think.
It's unclear to me if/how #95 is related.

cc @masih @willscott

[mvdan]

One easy suggestion is to allow inspecting the index codec from Reader:

func (*Reader) IndexCodec() multicodec.Code

The internal reader is a ReaderAt, so we can read the varint without consuming bytes. I don't even think we need to return an error; in the rare case the index varint is corrupted, we can return a bogus multicodec like 0 (i.e. "identity"). The caller should only consider the result valid if it's a known car index multicodec.

[mvdan]

Another suggestion, which could replace IndexCodec above, is a higher level API to obtain the existing Index in an efficient way:

func (*Reader) Index() (index.Index, error)

It would return nil, nil if the CAR file has no index. It would back up the index with the mmap where possible, or otherwise read and decode the entire index into memory. One could still obtain the multicodec via Index.Codec, and one could type assert the index interface to IterableIndex or specific implementations.