1.6.1

1.6.1 (2024-04-11)

Documentation

• docs: Update mmteb (#338)

docs: update mmteb (bee4244)

Fix

• fix: missing json and updated tests to not run in editable mode (#340)

• fix: Added json files to pyproject.toml

• ci: avoid using -e when installing for tests (17c809d)

1.6.0

1.6.0 (2024-04-10)

Documentation

• docs: Update points.md (#337) (2a3f9e6)

Feature

• feat: Added new language code standard (#326)

• fix: Added initial language code suggestion

• docs: updated task metadata description

• fix: changed folder structure to iso 639-3 codes

• fix: Updated all language tags

• clean: removed accidental results commit

• fix: Add trusting of remote code to remove warning

• fix: Added formatting

• fix: trust remote code the flores dataset

• docs: Added point for language rewrite

• fix: reran linter after merge

• fix: Added corrections from review

• fix: Updated languages for newly added datasets

• docs: added points for new annotations (f0daece)

1.5.6

1.5.6 (2024-04-10)

Documentation

• docs: add points and affiliation for MartinBernstorff (#335)

docs: update points.md (2903cb4)

Fix

• fix: Added medical qa dataset (#333)

• Added news classification dataset.

• Fixes on suggestions

• Added new medical qa dataset

• Update model run files and model path

• Added points for dataset.

• Fixes

---

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (80acc3e)

Unknown

• Update pull_request_template.md (84cffa2)

1.5.5

1.5.5 (2024-04-09)

Fix

• fix: Improve logging when the revision is None (#329) (404587b)

1.5.4

1.5.4 (2024-04-08)

Fix

• fix: Multiple dataset fixes (#328)

• fix: remove time of run (as it does not relate to the model itself). Time of run should be on the dataset results

• fix: fixes the PawsX datasets

• docs: Updated points

• fix: flores clustering

• fix: mulitple dataset fixes

• docs: updated points

• fix: added missing dataset_transform to multitask task

• syle: ran formatter

• fix: correctly fix pawsX (84408f7)

1.5.3

1.5.3 (2024-04-08)

Documentation

• docs: Added point for SEB (#318)

• docs: added points for seb

• docs: added points for seb (ca64fc7)

• docs: Small fixes in readme.md (#317)

Fix typos in readme.md (ede12c8)

Fix

• fix: Added English news classification dataset (#323)

• Fix typos in readme.md

• Added news classification dataset.

• Added news classification dataset.

• Fixes on suggestions

• Update docs/mmteb/points.md

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

---

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (4d21807)

Unknown

• Fix name (d69bf94)

• Add law datasets (#311)

• add command

• add datasets

• reformat dataset

• Rephrase description

• Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py

• Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py

• Update mteb/init.py

• Update scripts/run_mteb_law.py

• Update scripts/run_mteb_law.py

• Update mteb/init.py

• Update mteb/tasks/Retrieval/init.py

• Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py

• Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py

• Update mteb/tasks/Retrieval/law/LegalQuADRetrieval.py

• Update mteb/tasks/Retrieval/law/LegalQuADRetrieval.py

• Update scripts/run_mteb_law.py

• Update mteb/tasks/Retrieval/law/LegalSummarizationRetrieval.py

• Update mteb/tasks/Retrieval/law/LegalSummarizationRetrieval.py

• Update mteb/tasks/Retrieval/law/LeCaRDv2Retrieval.py

• Update mteb/tasks/Retrieval/law/LeCaRDv2Retrieval.py

• Rename GerDaLIRRetrieval.py to GerDaLIRSmallRetrieval.py

• Update mteb/tasks/Retrieval/init.py

• Update GerDaLIRSmallRetrieval.py

Add metadata

• Update GerDaLIRSmallRetrieval.py

Update metadata

• Update AILACasedocsRetrieval.py

Update AILACasedocsRetrieval metadata

• Update AILAStatutesRetrieval.py

Update AILAStatutesRetrieval metadata

• Update LeCaRDv2Retrieval.py

Update LeCaRDv2Retrieval metadata

• Update LegalBenchConsumerContractsQARetrieval.py

Update LegalBenchConsumerContractsQARetrieval metadata

• Update LegalBenchCorporateLobbyingRetrieval.py

Update LegalBenchCorporateLobbyingRetrieval metadata

• Update LegalQuADRetrieval.py

Update LegalQuADRetrieval metadata

• Update LegalSummarizationRetrieval.py

Update LegalSummarizationRetrieval metadata

• Update AILACasedocsRetrieval.py

Update AILACasedocsRetrieval

• Update AILACasedocsRetrieval.py

Update AILACasedocsRetrieval metadata

• Update AILAStatutesRetrieval.py

Update AILAStatutesRetrieval metadata

• Update GerDaLIRSmallRetrieval.py

Update GerDaLIRSmallRetrieval metadata

• Update LeCaRDv2Retrieval.py

Update LeCaRDv2Retrieval metadata

• Update LegalBenchConsumerContractsQARetrieval.py

• Update LegalBenchCorporateLobbyingRetrieval.py

• Update LegalQuADRetrieval.py

• Update LegalSummarizationRetrieval.py

• Update AILACasedocsRetrieval.py

• Update AILAStatutesRetrieval.py

• Update GerDaLIRSmallRetrieval.py

• Update LeCaRDv2Retrieval.py

• move dataset language folder

• update order

---

Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com> (6e3f419)

• Fix PawsX eval splits (#316) (15285d4)

1.5.2

1.5.2 (2024-04-04)

Fix

• fix: Minor fixes to metadata (#315)

• Update MindSmallReranking.py

• fix: Updated wrong metadata (e0eddf9)

Unknown

• Adding French team contribution points (#302)

• Update points.md

• Update docs/mmteb/points.md

• Update points.md

• Update points.md (23c9fdd)

1.5.1

1.5.1 (2024-04-03)

Fix

• fix: Added tests for checking datasets (#307)

• fix: Fixed hf_hub_name for WikiCitiesClustering

• Added points for this PR and a 3 other minor dataset fixes

• feat: Added tests which validated that datasets are available

• fix: Updated hf references and revisions to multiple datasets

• Added points for submission

• fix: Added suggestions from the review

• Apply suggestions from code review

Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com>

• fix: sped up async test for whether datasets exist

• fix: Updated revisions

• fix: reuploaded scandeval datasets

• fix: Applied formatter

---

Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com> (8d804f4)

1.5.0

1.5.0 (2024-04-02)

Feature

• feat: Allow extending the load_dataset parameters in custom tasks inheriting AbsTask (#299)

• Allow extending the load_dataset parameters

• format

• Fix test

• remove duplicated logic from AbsTask, now handled in the metadata

• add tests

• remove comments, moved to PR

• format

• extend metadata dict from super class

• Remove additional load_data

• test: adding very high level test

• Remove hf_hub_name and add test

• Fix revision in output file

---

Co-authored-by: gbmarc1 <marcantoine.belanger@shopify.com> (953780d)

1.4.1

1.4.1 (2024-04-01)

Fix

• fix: hf_hub_name for WikiCitiesClustering (#305)

• fix: Fixed hf_hub_name for WikiCitiesClustering

• Added points for this PR and a 3 other minor dataset fixes (b447235)