Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multilingual Hellaswag tasks #332

Merged
merged 60 commits into from
Oct 1, 2024
Merged

Multilingual Hellaswag tasks #332

merged 60 commits into from
Oct 1, 2024

Conversation

hynky1999
Copy link
Collaborator

Goal

Add Hellaswag tasks supporting about 36 unique languages.

Since it uses template it support all 3 types of formulation out of the box :)

How to test:

lighteval accelerate --output_dir=./tmp --custom_tasks="lighteval.tasks.multilingual.tasks" --tasks="custom|{task}|0|0" --model_args=pretrained=gpt2 --override_batch_size=1 --max_samples=100 --save_details

where task in hellaswag_tel_cf|0|0 hellaswag_tur_cf|0|0 hellaswag_tha_cf|0|0

Comments

  • We talked about removing the suite all together. Since right now we use suite I decided to use custom suite. I can switch to multilingual or whatever

hynky1999 and others added 30 commits September 5, 2024 17:40
@hynky1999
add multilignaul dynamic generative metrics
5c69eb0
@hynky1999
Merge branch 'main' into geneartive_dynamic_metrics
39c4220
@hynky1999
Merge branch 'geneartive_dynamic_metrics' into config_templates
2a5cdca
@hynky1999
draft
2df9a08
@hynky1999
finish multichoice config
95729ee
@hynky1999
Merge branch 'main' into geneartive_dynamic_metrics
3aa0579
@hynky1999
update tokenizers + install nltk reqs
b8f90a9
@hynky1999
use punkt tab
f5a8717
@hynky1999 @NathanHB
Update src/lighteval/utils/imports.py
227f572 
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
@hynky1999 @NathanHB
Update src/lighteval/metrics/normalizations.py
d80b3ba 
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
fix imports
532bdad
remove unused import
75f7ac5
@NathanHB
Merge branch 'main' into geneartive_dynamic_metrics
f99e330
@clefourrier
Merge branch 'main' into geneartive_dynamic_metrics
92daf90
@NathanHB
Merge branch 'main' into geneartive_dynamic_metrics
f2a801d
finish implementation of templates + move stuff around
91d9d4f
resolve nits
9356cc6
when in rome do as romans do (handle error messages the same way)
0fbc731
@hynky1999
fix utils
fa1fa83
@hynky1999
Merge branch 'geneartive_dynamic_metrics' into config_templates
db36e16
@hynky1999
nicers tests + fix them
44aeecf
@hynky1999
nicer todo
2bff963
@hynky1999
add nice doscrings 📃
3c9eb21
@hynky1999
add even more docstring
4216ae2
@hynky1999
nit
d8f56b8
@hynky1999
fix test
f26e88c
@hynky1999
add multilingual to dev group
111d615
@hynky1999
merge nli, add languagees to literals
7ca4239
@hynky1999
translation literals
22eeddb
@hynky1999
add nli
7faaa8a
hynky1999 and others added 15 commits September 30, 2024 18:21
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
ca865bd 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
1cc1187 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
d64251f 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
9806fab 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
35d7e6d 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
e560738 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999 @clefourrier
Update src/lighteval/tasks/multilingual/tasks.py
99524c5 
Co-authored-by: Clémentine Fourrier <22726840+clefourrier@users.noreply.github.com>
@hynky1999
add two new tasks + docs
150c76f
@hynky1999
Merge branch 'multilnag_nli_tasks' of github.com:huggingface/lighteva…
4e6100d 
…l into multilnag_nli_tasks
@hynky1999
Merge remote-tracking branch 'origin/main' into multilnag_nli_tasks
7b561fe
@hynky1999
Merge branch 'multilnag_nli_tasks' into multilang_copa_task
f9c7134
@hynky1999
add nice docs
b219e07
@hynky1999
Merge branch 'multilang_copa_task' into hellaswag_tasks
8899a1c
@hynky1999
update hellaswag with docs
fcad0e8
@hynky1999
move hellaswag to lighteval suite
43809ab
@hynky1999 hynky1999 changed the base branch from multilang_copa_task to main September 30, 2024 18:16
@hynky1999 hynky1999 changed the title Multilingual Hellaswg tasks Multilingual Hellaswag tasks Oct 1, 2024
clefourrier
clefourrier approved these changes Oct 1, 2024
View reviewed changes
src/lighteval/tasks/templates/utils/translation_literals.py
@@ -349,4 +349,5 @@ def __getattribute__(self, name: str) -> str:
Language.SORANI: TranslationLiterals(language=Language.SORANI),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future, would be nice to sort this one alphabetically to avoid duplicates

@hynky1999 hynky1999 merged commit 994fe73 into main Oct 1, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clefourrier clefourrier clefourrier approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hynky1999 @clefourrier @NathanHB