Fix PawsX eval splits

[imenelydiaker]

Fix for issue #309. Eval splits didn't have the correct name.

[KennethEnevoldsen]

Can you run one of the smaller models on this just to check that it works without issue?

[MartinBernstorff]

Looks good! Agree with Kenneth.

As a meta-question, maybe it's possible to check if the splits exist in the dataset without downloading it? Might be worth it if we get more errors like this.

[imenelydiaker]

Looks good! Agree with Kenneth.

As a meta-question, maybe it's possible to check if the splits exist in the dataset without downloading it? Might be worth it if we get more errors like this.

Yep was thinking about adding it to the test that checks if a dataset + revision id exists

[KennethEnevoldsen]

As a meta-question, maybe it's possible to check if the splits exist in the dataset without downloading it? Might be worth it if we get more errors like this.

As I understand there is no API for this (at least documented), but you could probably do something like:

>>> ds = load_dataset("paws-x", "en", revision="8a04d940a42cd40658986fdd8e3da561533a3646", streaming=True)
>>> ds.keys()
dict_keys(['train', 'test', 'validation'])

It only download metadata I believe:

Downloading builder script: 100%|██████████| 6.54k/6.54k [00:00<00:00, 9.68MB/s]
Downloading metadata: 100%|████████████████| 17.6k/17.6k [00:00<00:00, 9.76MB/s]
Downloading readme: 100%|██████████████████| 11.8k/11.8k [00:00<00:00, 7.18MB/s]

Might be a way to generalize the existing tests.

[KennethEnevoldsen]

Yep was thinking about adding it to the test that check if a dataset + revision id exists

@imenelydiaker this test already exists (based on a recent pr)

[imenelydiaker]

Can you run one of the smaller models on this just to check that it works without issue?

Yep I used average_word_embeddings_komninos with mteb CLI and it worked 🙂

[imenelydiaker]

As a meta-question, maybe it's possible to check if the splits exist in the dataset without downloading it? Might be worth it if we get more errors like this.

As I understand there is no API for this (at least documented), but you could probably do something like:

>>> ds = load_dataset("paws-x", "en", revision="8a04d940a42cd40658986fdd8e3da561533a3646", streaming=True)
>>> ds.keys()
dict_keys(['train', 'test', 'validation'])

It only download metadata I believe:

Downloading builder script: 100%|██████████| 6.54k/6.54k [00:00<00:00, 9.68MB/s]
Downloading metadata: 100%|████████████████| 17.6k/17.6k [00:00<00:00, 9.76MB/s]
Downloading readme: 100%|██████████████████| 11.8k/11.8k [00:00<00:00, 7.18MB/s]

Might be a way to generalize the existing tests.

So you just need to download the dataset in streaming mode to not store it locally and check the keys of the dataset object? Looks like a good alternative 🤔

[imenelydiaker]

Should I merge the fix and we'll open another PR for the test?

[MartinBernstorff]

Yeah, I'll give it a stab today 👍