* optimize __len__ of SlicedCorpus

[horpto]

• small refactoring of SlicedCorpus and IndexedCorpus
• remove unnecessary file

[menshikh-iv]

@horpto thanks, you are very active for last time 👍

[piskvorky]

Awesome! These type of fixes are sorely needed. Many thanks @horpto .

[horpto]

Thanks, @piskvorky! It's just little fix. The Worse problem is SlicedCorpus.__iter__: docbyoffset will reopen file for each doc. >< I cannot fix it as I don't want to destroy the abstraction of IndexedCorpus & not all Corpuses support line2doc of something like this.

[piskvorky]

I think we should also implement a binary format, such as extend our Matrix Market to handle binary (.mm, MmCorpus).

It will be efficient and much faster, for those people who care about speed more than readability. The format is pretty trivial too => easy task. CC @menshikh-iv .

[menshikh-iv]

Can you describe your proposal more concretely @piskvorky (I'll create an issue)?

[piskvorky]

I mean, store the (sparse) matrix represented by a corpus in a binary format: less space + faster processing.

For example, store each document as number_of_document_nonzeros (int32) followed by serialized (feature_id, feature_weight) pairs as (int32, float32) = 8 bytes per non-zero element. So, total serialized corpus size = num_docs * 4 + 8 * nnz.

That's one simple option (doesn't support slicing / indexing though).

More optimized binary schemes are possible, such as encoding elements more cleverly (RLE for the values? gray code for feature ids? or just zip? something even cleverer?) to save space, or adding indexing capabilities, but I don't think that's strictly necessary. Certainly not as the first step.