ERROR:mteb.evaluation.MTEB:Error while evaluating QuoraRetrieval: 'SimCSEWrapper' object has no attribute 'stop_multi_process_pool'

[xinghaow99]

Hi! Thanks for this easy-to-use repo!
While I'm getting this error when running the example script https://github.com/embeddings-benchmark/mtebscripts/blob/main/run_array_simcse.py on retrieval benchmarks like QuoraRetrieval.
How do I evaluate on retrieval tasks when using my own model with a wrapper?

[Muennighoff]

Hi! Thanks for this easy-to-use repo! While I'm getting this error when running the example script https://github.com/embeddings-benchmark/mtebscripts/blob/main/run_array_simcse.py on retrieval benchmarks like QuoraRetrieval. How do I evaluate on retrieval tasks when using my own model with a wrapper?

Hey sorry for the trouble, can you try with beir==1.0.0?
I also just updated the mteb package not to use parallel encoding, which should solve the issue if you upgrade to the latest mteb version. Let me know if you still face problems!

[xinghaow99]

Hi! Thanks for this easy-to-use repo! While I'm getting this error when running the example script https://github.com/embeddings-benchmark/mtebscripts/blob/main/run_array_simcse.py on retrieval benchmarks like QuoraRetrieval. How do I evaluate on retrieval tasks when using my own model with a wrapper?

Hey sorry for the trouble, can you try with beir==1.0.0? I also just updated the mteb package not to use parallel encoding, which should solve the issue if you upgrade to the latest mteb version. Let me know if you still face problems!

beir==1.0.0 solves the problem! Thanks for the help!