How use "docid" get the text corresponding to docid？

[FutureForMe]

Hi~
I have obtained the retrieval result using the code(Dense Retrieve FaissSearcher), but I would like to ask how to obtain the corresponding text according to the docid?

Thanks

[lintool]

The Faiss indexes do not store actual documents. You'll have to use the corresponding Lucene indexes. Consult this guide: https://github.com/castorini/pyserini/blob/master/docs/usage-fetch.md

[FutureForMe]

I want to know that I have retrieved the doc id from faiss index (wikipedia-dpr-100w.dpr-single-nq). Which file should I go to for the corresponding doc?

[manveertamber]

Hi @FutureForMe,

You can try something like this

from pyserini.search.lucene import LuceneSearcher
import json

searcher = LuceneSearcher.from_prebuilt_index('wikipedia-dpr-100w')
passage_text = json.loads(searcher.doc(str(docid)).raw())['contents']

[FutureForMe]

Hi @manveertamber, thank you for your reply. However, I noticed a problem. The number of documents in Wikipedia in FAISS is 21015320, but the number of Documents in the Lucene index of Wikipedia is 21015324. Will this amount of inconsistency affect the retrieval results of the document (causing corresponding misalignment of the retrieval results)?

the screenshot of wikipedia from pyserini/pyserini/prebuilt_index_info.py

[manveertamber]

I don't think this is a cause for concern. Documents are identified by string docids, which should be consistent across both. I think the difference in document counts is just because of some empty documents.

[FutureForMe]

Thank you again for your reply. Your reply answered my doubts.

[manveertamber]

hi, does it mean that even if I already have an original Faiss index for searching, I still need to build a Lucene index for looking up raw content, in addition to the original Faiss index?

You'll need some sort of way to lookup the raw content given a docid because faiss doesn't have this. You don't need to build a Lucene index though. Depends on how you have your data stored?