Modifying the inverted index

[abg0148]

Hello,
I was working on a usecase where i wanted to access the inverted index list and modify it to run some experiments. so far i am unable to figure it out. Can someone help out with this usecase. something basic like taking the index and removing certain documents from the postings list, if someone can help share how i can do this.

Thanks!

[abg0148]

i am working on the ms marco dataset

[lintool]

Lucene does provide methods for removing documents, but it's not exposed in Anserini, unfortunately...

You can always take the corpus, remove the documents, and then re-index? With MS MARCO v1, indexing time is pretty fast, so for research purposes this approach should work fine...

[abg0148]

Wouldnt exactly work for me, since same documents might not get pruned for all inverted lists. I will try using pylucene to update the index and then maybe run evaluations using pyserini.
But this seems like a good feature, what do you say @lintool? I can work on this maybe, if you have some time, maybe we can discuss?

[lintool]

Sure, happy to discuss - send a PR?

[abg0148]

Will do!