Stemming and searching Arabic documents

[Fatima-Haouari]

Hi,
Kindly advise about the following:
When indexing documents using Pyserini. Is stemming done by default? or we should specify this? I managed to index my Arabic collection by specifying the language ar as shown here . However, when I tried to specify the language when searching documents as shown here
searcher.set_language('ar')

I got the below error

 searcher.set_language('ar')
AttributeError: 'SimpleSearcher' object has no attribute 'set_language'

[lintool]

For future reference, please file questions under "Discussions" - Q&A.

The functionality appears to be there:
https://github.com/castorini/pyserini/blob/master/pyserini/search/__main__.py#L143

Are you using a PyPI version? If so, that feature may not be on PyPI yet. We'd recommend a dev installation to get access to the latest features: https://github.com/castorini/pyserini/blob/master/docs/installation.md#development-installation

[Fatima-Haouari]

Thanks a lot for the quick response. I re-installed pyserini and it worked perfectly fine.
Just a question. Is stemming done by default when indexing?

[lintool]

For English, yes, for other languages, we use Lucene-provided analyzers, so whatever their default behaviors are. See for details: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexCollection.java#L762

[Fatima-Haouari]

Thanks a lot for your help.