Add Korean Text Search Tasks to MTEB

[taeminlee]

Hello MTEB maintainers,

I am currently working on a project that involves implementing an embedding model for Korean text search. Through my work, I realized the need for a benchmark, and that's how I came across MTEB. However, I noticed that MTEB does not currently support tasks specifically for Korean language. To address this gap, I have added tasks for Korean text search.

Limitations

At the moment, the implementation does not support a wide range of tasks like PL-MTEB or C-MTEB.

To-Do

I plan to add various tasks using Korean corpora in the near future. For example, I'm considering the addition of tasks like klue-sts.

I believe this enhancement will significantly benefit researchers and developers working with Korean language text search and analysis. I am looking forward to your feedback and suggestions on this addition.

Thank you for considering my contribution.

[Muennighoff]

Amazing! Adding Korean would be huge :)

From my side, we can pretty much already merge this and then you can add more whenever you want. But we can also leave the PR open if you prefer!

[Muennighoff]

I think we can remove this. This should be done in the encode method of the model. If the model has an encode_corpus function then it will automatically use that

[taeminlee]

I agree with your suggestion to remove this section. Thanks for pointing this out!

[taeminlee]

Amazing! Adding Korean would be huge :)

From my side, we can pretty much already merge this and then you can add more whenever you want. But we can also leave the PR open if you prefer!

Thank you for your feedback. I concur with proceeding with the current merge. :)

[Pang-dachu]

@taeminlee
한국어 데이터 MTEB 평가를 어떻게 진행하면 될까요 ?

현재 [run_mteb_korean.py] 파일을 실행하면 평가 task들에 대해서 아래와 같은 오류가 발생합니다.

INFO:main:Running task: Ko-miracl
WARNING:mteb.evaluation.MTEB:WARNING: Unknown tasks: Ko-miracl.   

Known tasks: 8TagsClustering,AFQMC,ATEC,AllegroReviews,AmazonCounterfactualClassification,AmazonPolarityClassification,AmazonReviewsClassification,AngryTweetsClassification,ArguAna,ArguAna-PL,ArxivClusteringP2P,ArxivClusteringS2S,AskUbuntuDupQuestions,BIOSSES,BQ,BUCC,Banking77Classification,BiorxivClusteringP2P,BiorxivClusteringS2S,BlurbsClusteringP2P,BlurbsClusteringS2S,BornholmBitextMining,CBD,CDSC-E,CDSC-R,CLSClusteringP2P,CLSClusteringS2S,CMedQAv1,CMedQAv2,CQADupstackAndroidRetrieval,CQADupstackEnglishRetrieval,CQADupstackGamingRetrieval,CQADupstackGisRetrieval,CQADupstackMathematicaRetrieval,CQADupstackPhysicsRetrieval,CQADupstackProgrammersRetrieval,CQADupstackStatsRetrieval,CQADupstackTexRetrieval,CQADupstackUnixRetrieval,CQADupstackWebmastersRetrieval,CQADupstackWordpressRetrieval,ClimateFEVER,CmedqaRetrieval,Cmnli,CovidRetrieval,DBPedia,DBPedia-PL,DKHateClassification,DalajClassification,DanishPoliticalCommentsClassification,DuRetrieval,EcomRetrieval,EmotionClassification,FEVER,FiQA-PL,FiQA2018,HotpotQA,HotpotQA-PL,IFlyTek,ImdbClassification,JDReview,LCQMC,LccSentimentClassification,MMarcoReranking,MMarcoRetrieval,MSMARCO,MSMARCO-PL,MSMARCOv2,MTOPDomainClassification,MTOPIntentClassification,MassiveIntentClassification,MassiveScenarioClassification,MedicalRetrieval,MedrxivClusteringP2P,MedrxivClusteringS2S,MindSmallReranking,MultilingualSentiment,NFCorpus,NFCorpus-PL,NQ,NQ-PL,NoRecClassification,NordicLangClassification,NorwegianParliament,Ocnli,OnlineShopping,PAC,PAWSX,PPC,PSC,PolEmo2.0-IN,PolEmo2.0-OUT,QBQTC,Quora-PL,QuoraRetrieval,RedditClustering,RedditClusteringP2P,SCIDOCS,SCIDOCS-PL,SICK-E-PL,SICK-R,SICK-R-PL,STS12,STS13,STS14,STS15,STS16,STS17,STS22,STSB,STSBenchmark,ScalaDaClassification,ScalaNbClassification,ScalaSvClassification,SciDocsRR,SciFact,SciFact-PL,SprintDuplicateQuestions,StackExchangeClustering,StackExchangeClusteringP2P,StackOverflowDupQuestions,SweRecClassification,T2Reranking,T2Retrieval,TNews,TRECCOVID,TRECCOVID-PL,Tatoeba,TenKGnadClusteringP2P,TenKGnadClusteringS2S,ThuNewsClusteringP2P,ThuNewsClusteringS2S,Touche2020,ToxicConversationsClassification,TweetSentimentExtractionClassification,TwentyNewsgroupsClustering,TwitterSemEval2015,TwitterURLCorpus,VideoRetrieval,Waimai.

[Muennighoff]

@taeminlee 한국어 데이터 MTEB 평가를 어떻게 진행하면 될까요 ?

현재 [run_mteb_korean.py] 파일을 실행하면 평가 task들에 대해서 아래와 같은 오류가 발생합니다.

INFO:main:Running task: Ko-miracl
WARNING:mteb.evaluation.MTEB:WARNING: Unknown tasks: Ko-miracl.   

Known tasks: 8TagsClustering,AFQMC,ATEC,AllegroReviews,AmazonCounterfactualClassification,AmazonPolarityClassification,AmazonReviewsClassification,AngryTweetsClassification,ArguAna,ArguAna-PL,ArxivClusteringP2P,ArxivClusteringS2S,AskUbuntuDupQuestions,BIOSSES,BQ,BUCC,Banking77Classification,BiorxivClusteringP2P,BiorxivClusteringS2S,BlurbsClusteringP2P,BlurbsClusteringS2S,BornholmBitextMining,CBD,CDSC-E,CDSC-R,CLSClusteringP2P,CLSClusteringS2S,CMedQAv1,CMedQAv2,CQADupstackAndroidRetrieval,CQADupstackEnglishRetrieval,CQADupstackGamingRetrieval,CQADupstackGisRetrieval,CQADupstackMathematicaRetrieval,CQADupstackPhysicsRetrieval,CQADupstackProgrammersRetrieval,CQADupstackStatsRetrieval,CQADupstackTexRetrieval,CQADupstackUnixRetrieval,CQADupstackWebmastersRetrieval,CQADupstackWordpressRetrieval,ClimateFEVER,CmedqaRetrieval,Cmnli,CovidRetrieval,DBPedia,DBPedia-PL,DKHateClassification,DalajClassification,DanishPoliticalCommentsClassification,DuRetrieval,EcomRetrieval,EmotionClassification,FEVER,FiQA-PL,FiQA2018,HotpotQA,HotpotQA-PL,IFlyTek,ImdbClassification,JDReview,LCQMC,LccSentimentClassification,MMarcoReranking,MMarcoRetrieval,MSMARCO,MSMARCO-PL,MSMARCOv2,MTOPDomainClassification,MTOPIntentClassification,MassiveIntentClassification,MassiveScenarioClassification,MedicalRetrieval,MedrxivClusteringP2P,MedrxivClusteringS2S,MindSmallReranking,MultilingualSentiment,NFCorpus,NFCorpus-PL,NQ,NQ-PL,NoRecClassification,NordicLangClassification,NorwegianParliament,Ocnli,OnlineShopping,PAC,PAWSX,PPC,PSC,PolEmo2.0-IN,PolEmo2.0-OUT,QBQTC,Quora-PL,QuoraRetrieval,RedditClustering,RedditClusteringP2P,SCIDOCS,SCIDOCS-PL,SICK-E-PL,SICK-R,SICK-R-PL,STS12,STS13,STS14,STS15,STS16,STS17,STS22,STSB,STSBenchmark,ScalaDaClassification,ScalaNbClassification,ScalaSvClassification,SciDocsRR,SciFact,SciFact-PL,SprintDuplicateQuestions,StackExchangeClustering,StackExchangeClusteringP2P,StackOverflowDupQuestions,SweRecClassification,T2Reranking,T2Retrieval,TNews,TRECCOVID,TRECCOVID-PL,Tatoeba,TenKGnadClusteringP2P,TenKGnadClusteringS2S,ThuNewsClusteringP2P,ThuNewsClusteringS2S,Touche2020,ToxicConversationsClassification,TweetSentimentExtractionClassification,TwentyNewsgroupsClustering,TwitterSemEval2015,TwitterURLCorpus,VideoRetrieval,Waimai.

Maybe it was cuz you weren't installing from source? I released a new MTEB version so if you upgrade to 1.1.2 it should be in there