LuceneSearcher + multiprocessing problem

[davidoterof]

Hi,

I'm using LuceneSearcher in a multiprocessing setting, but it is not working as I expected. I crafted the following script to showcase the problem that I have:

from multiprocessing import Process
from pyserini.search.lucene import LuceneSearcher


def do_job(w):
    searcher = LuceneSearcher("...")
    print(w)
    print(searcher.doc(w).contents())
    searcher.close()

processes = []

# creating processes
for w in range(4):
    p = Process(target=do_job, args=(w,))
    processes.append(p)
    p.start()

# completing process
for p in processes:
    p.join()

When running this, it gives a bunch of errors like this:

jnius.JavaException: JVM exception occurred: org/apache/lucene/store/Lock java.lang.NoClassDefFoundError

And every time that I run it, the class name changes without any apparent pattern. I don't know if I'm misunderstanding something, and this is normal behaviour, or the problem is on pyserini's side.

thanks

[lintool]

Instead of trying to parallelize querying in LuceneSearcher on the Python end, use batch_search, which takes advantage of multi-threading on the Java side, which is much more robust and doesn't suffer from GIL-related issues:
https://github.com/castorini/pyserini/blob/master/pyserini/search/lucene/_searcher.py#L173

Hope this helps!

[davidoterof]

Hi! First, thanks for answering

Actually, this solution would not work for our use case. Using multithreading is a thing in the framework we were using, and we had no way to change it. Anyway, we solved the problem by reworking our whole approach and we don't have that problem anymore.

Thanks!

[lintool]

Can you share your solution so that others may benefit from it in the future?

[davidoterof]

Sure! We were trying to deploy a REST endpoint with pyserini+gunicorn with --workers=n (n > 1), where each worker performed some search with LuceneSearcher (an instance globally shared between workers). Our solution was to deploy a REST API with Java and Lucene directly, and consume that API from our gunicorn workers.

[lintool]

Ah, makes sense! Thanks for the info.