merge a large index with small index \ adding small collection of docs to a large index

[tommymordo33]

Hi,

I would like to add a small collection of documents to a large prebuilt index. i tried to use the function add_doc_raw() of LuceneIndexer but i got an exception:

jnius.JavaException: JVM exception occurred: cannot change field "id" from doc values type=SORTED to inconsistent doc values type=BINARY java.lang.IllegalArgumentException

There is another option to make it?

Thanks!

###full code###

from pyserini.index.lucene import IndexReader, LuceneIndexer
import tarfile
from urllib.request import urlretrieve
r = random.randint(0, 10000000)
collection_url = 'https://github.com/castorini/anserini-data/raw/master/CACM/lucene-index.cacm.tar.gz'
tarball_name = 'lucene-index.cacm-{}.tar.gz'.format(r)
index_dir = 'index{}/'.format(r)
 _,_ = urlretrieve(collection_url, tarball_name)
tarball = tarfile.open(tarball_name)
tarball.extractall(index_dir)
tarball.close()
searcher = SimpleSearcher(f'{index_dir}lucene-index.cacm')
index_utils = IndexReader(f'{index_dir}lucene-index.cacm')
lucene_index = LuceneIndexer(f'{index_dir}lucene-index.cacm', append=True)
x =  '{ "id": "doc99910294", "contents": "this is the content of bob"}'
index_utils.stats()
lucene_index.add_doc_raw(x)
lucene_index.close()

[lintool]

I believe you're getting that error because the index you're using was built using Lucene 8.

Try this one and see if it resolves the issue: https://github.com/castorini/anserini-data/blob/master/CACM/lucene9-index.cacm.tar.gz

[tommymordo33]

Thanks for your reply!
i tried the lucene9 index and then i got the exception:

jnius.JavaException: JVM exception occurred: cannot change field "contents" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS_AND_FREQS java.lang.IllegalArgumentException.

i tried adding a flag of storePosition to resolve that:
lucene_index = LuceneIndexer(f'{index_dir}lucene9-index.cacm', append=True, args=['-storePositions'])

but then got another excetption:
jnius.JavaException: JVM exception occurred: Option "-index" is required org.kohsuke.args4j.CmdLineException.

adding -index as a flag too, return another exception:
jnius.JavaException: JVM exception occurred: No argument is allowed: org.kohsuke.args4j.CmdLineException.

Do i missed something?

Another question on the same issue:
what is the best way the build a sparse index from a collection (for example Clueweb) over pyserini?

Thanks!
Tommy.

[lintool]

You're trying to change the underlying method of index. They have to match - but not sure this is properly exposed in Pyserini.

What are you actually trying to do? Surely, it can't be adding to CACM index?

[tommymordo33]

I'm working on creating an index using a specific portion of Clueb09 documents for research purposes, and then incorporating it with various subsets of generated documents.

There is a way to do it with pyserini?

Thanks!

[lintool]

So build the index on ClueWeb09, then add documents to it. I don't think trying to debug what's happening with the demo index is helpful...?

[tommymordo33]

Perfect! i will try it.
thank you.