JVM exception occurred: no segments* file found in MMapDirectory

[suamin]

Hi, thank you for this simplified and an excellent tool!

I'm attempting to explore the SimpleSearcher class with DPR but running into an unexpected Java exception:

searcher = SimpleSearcher.from_prebuilt_index('wikipedia-dpr-single-nq-bf')
Attempting to initialize pre-built index wikipedia-dpr-single-nq-bf.
/netscratch/samin/cache/pyserini/indexes/dindex-wikipedia-dpr_single_nq-bf-20200115-cd5034.d1ef9286ddb38633cd052171963c62cb already exists, skipping download.
Initializing wikipedia-dpr-single-nq-bf...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/netscratch/samin/dev/miniconda3/envs/pyserini/lib/python3.7/site-packages/pyserini/search/_searcher.py", line 76, in from_prebuilt_index
    return cls(index_dir)
  File "/netscratch/samin/dev/miniconda3/envs/pyserini/lib/python3.7/site-packages/pyserini/search/_searcher.py", line 51, in __init__
    self.object = JSimpleSearcher(index_dir)
  File "jnius/jnius_export_class.pxi", line 271, in jnius.JavaClass.__init__
  File "jnius/jnius_export_class.pxi", line 385, in jnius.JavaClass.call_constructor
  File "jnius/jnius_utils.pxi", line 91, in jnius.check_exception
jnius.JavaException: JVM exception occurred: no segments* file found in MMapDirectory@/netscratch/samin/cache/pyserini/indexes/dindex-wikipedia-dpr_single_nq-bf-20200115-cd5034.d1ef9286ddb38633cd052171963c62cb lockFactory=org.apache.lucene.store.NativeFSLockFactory@3feb2dda: files: [docid, index] org.apache.lucene.index.IndexNotFoundException

Here are the packages installed:

Package            Version
------------------ -----------
certifi            2021.10.8
charset-normalizer 2.0.9
click              8.0.3
Cython             0.29.25
faiss-cpu          1.7.1.post3
filelock           3.4.0
flatbuffers        2.0
huggingface-hub    0.2.1
idna               3.3
importlib-metadata 4.8.2
joblib             1.1.0
nmslib             2.1.1
numpy              1.21.4
onnxruntime        1.10.0
packaging          21.3
pandas             1.3.5
pip                21.2.2
protobuf           3.19.1
psutil             5.8.0
pybind11           2.6.1
pyjnius            1.4.1
pyparsing          3.0.6
pyserini           0.14.0
python-dateutil    2.8.2
pytz               2021.3
PyYAML             6.0
regex              2021.11.10
requests           2.26.0
sacremoses         0.0.46
scikit-learn       1.0.1
scipy              1.7.3
sentencepiece      0.1.96
setuptools         58.0.4
six                1.16.0
threadpoolctl      3.0.0
tokenizers         0.10.3
torch              1.7.1
tqdm               4.62.3
transformers       4.13.0
typing_extensions  4.0.1
urllib3            1.26.7
wheel              0.37.0
zipp               3.6.0

and javac 11.0.13.

Thanks in advance!

[suamin]

The MD5 hash values for the downloaded index:

>> md5sum dindex-wikipedia-dpr_single_nq-bf-20200115-cd5034.d1ef9286ddb38633cd052171963c62cb/*
169133f4fba54d7eeda481903abe0f8f  /netscratch/samin/cache/pyserini/indexes/dindex-wikipedia-dpr_single_nq-bf-20200115-cd5034.d1ef9286ddb38633cd052171963c62cb/docid
f27ff30376dfef88f999e8076883718c  /netscratch/samin/cache/pyserini/indexes/dindex-wikipedia-dpr_single_nq-bf-20200115-cd5034.d1ef9286ddb38633cd052171963c62cb/index

In the download directory name, the value matches with one mentioned here.

[lintool]

SimpleSearcher is only for inverted indexes (Lucene). DPR uses Faiss - see this guide:
https://github.com/castorini/pyserini/blob/master/docs/experiments-dpr.md

[suamin]

Thank you for the quick answer, following worked:

from pyserini.dsearch import SimpleDenseSearcher, DprQueryEncoder
encoder = DprQueryEncoder('facebook/dpr-question_encoder-multiset-base')
searcher = SimpleDenseSearcher.from_prebuilt_index('wikipedia-dpr-single-nq-bf', encoder)

where encoder can take names from DPR query encoders from HuggingFace models. It might be of consideration to have from_pretrained method available from the query encoders API.

[lintool]

Pull requests welcome!