feat: Added new language code standard

[KennethEnevoldsen]

This adds a new language code standard using an iso 639-3 code as well as an iso code for scripts.

I believe the benchmarks are currently preserved as they should be. I also believe all tasks run as intended (I have tested a few of the relevant candidates).

A few things we would probably need to fix:

•  MTEB(task_langs=[...]), currently uses just the user-specified splits, we should instead transfer it to use the standard language codes.
  • Currently the above filter also only filters within tasks (not monolingual tasks). I believe it should do both.
  • While we are changing it, we could also add a few other filters based on domains etc.
•  The above will cause a need to update the benchmark lists (shouldn't be too big)
•  This change does not lead to changes (at least as I understand it) in the results, but it probably should. We have already discussed standardizing the format of the results.

The suggestions above will be major breaking which changes the way the benchmark interface works as well as how results are stored.

supersedes: #319
fixes: #251

[KennethEnevoldsen]

@MartinBernstorff, @Muennighoff and @imenelydiaker it would be great to have this PR review before it diverge too much

[MartinBernstorff]

Good work @kenneth! Huge diff though, so might very well have missed something, but the stuff I noticed all seems very minor.

[MartinBernstorff]

My impression is that providing these "shortcuts" tend to be annoying for maintenance, but your mileage may vary.

[KennethEnevoldsen]

Annoying for maintenance and nice for use. My guess would be that it removes confusion due to the mapping dicts (leading to less issues). All a guess though. I will need to for:

MTEB(task_langs=[...]), currently uses just the user-specified splits, we should instead transfer it to use the standard language codes.

So rather than implement it in the MTEB object I believe it is better places here.

[KennethEnevoldsen]

I will merge this in now due to the many merge conflicts I keep having to solve. Once it is in we can basically launch MMTEB as the other changes are only nice to have, but not need to have and don't cause issues for people adding datasets

[Muennighoff]

I will merge this in now due to the many merge conflicts I keep having to solve. Once it is in we can basically launch MMTEB as the other changes are only nice to have, but not need to have and don't cause issues for people adding datasets

Good with me! Only had a quick look but looks great, really nice work!

[Muennighoff]

Do we have to change the language codes in all existing result files?

[KennethEnevoldsen]

@Muennighoff atm. I don't believe it changes the results file at all (still uses the hf_name for the multilingual dataset e.g. "fr-en") and for the monolingual datasets, it doesn't include a lang-id. The same is the case for MTEB(task_langs="fr-en"), which should use the same mappings as before and still match the hf-name.