Add law datasets

[ShuangLI59]

Checklist for adding MMTEB dataset

• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• I have considered the size of the dataset and reduced it if it is too big (2048 examples is typically large enough for most tasks)
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.
• I have added points for my submission to the POINTS.md file.

[Muennighoff]

Looks great amazing job! Tagging @KennethEnevoldsen to make sure you are okay with having a law folder?
This will be a new leaderboard tab detailed here: https://huggingface.co/spaces/mteb/leaderboard/discussions/90

I think lateron we will separate languages and domains and maybe allow people to select combinations of them via dropdown boxes or similar 🧐

@KennethEnevoldsen do you know why the tests are failing? It seems unrelated to the PR 🧐

[KennethEnevoldsen]

These need to e filled out (for all datasets). Previous datasets don't have them but future datasets should.

[ShuangLI59]

How to do the e filled out? Will your team do this for us?

[KennethEnevoldsen]

The guide describes how to fill these out. Generally, we expect it when new datasets are added to MTEB as the ones adding the dataset are more knowledgeable about the dataset.

[KennethEnevoldsen]

Generally though here is a short guess:

        date=, # you best guess on date of when the data was generated (from, to) 
        form="written", # assumed 
        domains=["Legal", "Non-fiction"], # assumed
        task_subtypes=[],
        license=,  # needs be specified
        socioeconomic_status=["high"], # assumed but since it is law it is probably correct
        annotations_creators=, # required
        dialect=[], # assuming there are no dialects
        text_creation="found", # assumed
        bibtex_citation=, # if there is none just leave it as None otherwise please specify it 
        n_samples=, # check using python
        avg_character_length=, # check using python

[KennethEnevoldsen]

@ShuangLI59 thanks for the submission I believe this is a great addition to MTEB. We still need a few things before we can merge this is, but it should be quick enough to add.

Please make sure to also run the models as specified on the checklist (this is important as we do not test to run on all datasets so to make sure you integrate it correctly please specify those as well.

[KennethEnevoldsen]

Looks great amazing job! Tagging @KennethEnevoldsen to make sure you are okay with having a law folder?
This will be a new leaderboard tab detailed here: https://huggingface.co/spaces/mteb/leaderboard/discussions/90

I would move it to the "en" folder. Legal should be tagged as domain in the metadata.

I think lateron we will separate languages and domains and maybe allow people to select combinations of them via dropdown boxes or similar 🧐

We should also add into the interface of MTEB to filter based on domain as well (that should be doable with the current metadata).

[Muennighoff]

Looks great amazing job! Tagging @KennethEnevoldsen to make sure you are okay with having a law folder?
This will be a new leaderboard tab detailed here: https://huggingface.co/spaces/mteb/leaderboard/discussions/90

I would move it to the "en" folder. Legal should be tagged as domain in the metadata.

I think lateron we will separate languages and domains and maybe allow people to select combinations of them via dropdown boxes or similar 🧐

We should also add into the interface of MTEB to filter based on domain as well (that should be doable with the current metadata).

The thing is that it's a mix of English, German & Chinese datasets so they would be all separate then even though they form one benchmark

[KennethEnevoldsen]

The thing is that it's a mix of English, German & Chinese datasets so they would be all separate then even though they form one benchmark

Ah didn't see that aspect. I think grouping them in languages is what I would do for now though. Potentially we could do:

tasks/{type}/{lang}/{major domain}/my_task.py

But since many datasets can have multiple domains I would not do that. Instead I would specify it (as you do) in a task list.

PS: You might consider adding the "Norwegian Courts" dataset (also legal, but not sure it is relevant). PR #315 updated its metadata

[KennethEnevoldsen]

@KennethEnevoldsen do you know why the tests are failing? It seems unrelated to the PR 🧐

I think it is a brief network error (sadly happens when your test relies on external resources). Rerunning the test should solve the issue otherwise I will have a look at it.

[Muennighoff]

@ShuangLI59 can you still move the datasets to the language folders? Then I think we can merge!

[ShuangLI59]

@Muennighoff Yes the datasets have been moved to the language folders

[ShuangLI59]

Hi @Muennighoff , I noticed that "GerDaLIR" should be changed to "GerDaLIRSmall" in scripts/run_mteb_law.py
Could you help update this?

TASK_LIST_RETRIEVAL_LAW = [
"LegalSummarization",
"LegalBenchConsumerContractsQA",
"LegalBenchCorporateLobbying",
"AILACasedocs",
"AILAStatutes",
"LeCaRDv2",
"LegalQuAD",
"GerDaLIRSmall",
]