docs: first stab at point attribution #280
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MartinBernstorff
changed the title
KennethEnevoldsen
requested review from
imenelydiaker and
Muennighoff
and removed request for
KennethEnevoldsen
|
Shouldn't this PR also update the actual scores in |
|
@Muennighoff we wanted to simply discuss it before we added it in |
|
edited by KennethEnevoldsen (it was just a some compressed message)
|
|
@Myahr208 it seems like there is something wrong with your formatting |
Sure it looks good to me! |
imenelydiaker
approved these changes
Mar 26, 2024
|
@imenelydiaker I would love you thoughts on this PR? |
@KennethEnevoldsen I approved the PR, everything looks good to me! 🚀 You guys did a great job! |
|
@KennethEnevoldsen and @MartinBernstorff PR #302 was merged, can you please add your affiliations and merge this PR? |
ci: Added tests
update Makefile and test_all_abstasks.py
BREAKING CHANGE: Bump version
…different dataset (#297)
* docs: added information related to the automatic release * docs: removed test-parallel from docs * docs: minor additions to contributing guidelines * ci: removed changelog As it already present in the git releases * Apply suggestions from code review Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com> --------- Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
…-terrier (#292) * ci: Added windows to test suite * feat: Changed to pytrec-eval-terrier to add support for windows installs
* fix: Fixed hf_hub_name for WikiCitiesClustering * Added points for this PR and a 3 other minor dataset fixes
…eriting AbsTask (#299) * Allow extending the load_dataset parameters * format * Fix test * remove duplicated logic from AbsTask, now handled in the metadata * add tests * remove comments, moved to PR * format * extend metadata dict from super class * Remove additional load_data * test: adding very high level test * Remove hf_hub_name and add test * Fix revision in output file --------- Co-authored-by: gbmarc1 <marcantoine.belanger@shopify.com>
* fix: Fixed hf_hub_name for WikiCitiesClustering * Added points for this PR and a 3 other minor dataset fixes * feat: Added tests which validated that datasets are available * fix: Updated hf references and revisions to multiple datasets * Added points for submission * fix: Added suggestions from the review * Apply suggestions from code review Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com> * fix: sped up async test for whether datasets exist * fix: Updated revisions * fix: reuploaded scandeval datasets * fix: Applied formatter --------- Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com>
* Update points.md * Update docs/mmteb/points.md * Update points.md * Update points.md
* Update MindSmallReranking.py * fix: Updated wrong metadata
Fix typos in readme.md
* docs: added points for seb * docs: added points for seb
* add command * add datasets * reformat dataset * Rephrase description * Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py * Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py * Update mteb/__init__.py * Update scripts/run_mteb_law.py * Update scripts/run_mteb_law.py * Update mteb/__init__.py * Update mteb/tasks/Retrieval/__init__.py * Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py * Update mteb/tasks/Retrieval/law/GerDaLIRRetrieval.py * Update mteb/tasks/Retrieval/law/LegalQuADRetrieval.py * Update mteb/tasks/Retrieval/law/LegalQuADRetrieval.py * Update scripts/run_mteb_law.py * Update mteb/tasks/Retrieval/law/LegalSummarizationRetrieval.py * Update mteb/tasks/Retrieval/law/LegalSummarizationRetrieval.py * Update mteb/tasks/Retrieval/law/LeCaRDv2Retrieval.py * Update mteb/tasks/Retrieval/law/LeCaRDv2Retrieval.py * Rename GerDaLIRRetrieval.py to GerDaLIRSmallRetrieval.py * Update mteb/tasks/Retrieval/__init__.py * Update GerDaLIRSmallRetrieval.py Add metadata * Update GerDaLIRSmallRetrieval.py Update metadata * Update AILACasedocsRetrieval.py Update AILACasedocsRetrieval metadata * Update AILAStatutesRetrieval.py Update AILAStatutesRetrieval metadata * Update LeCaRDv2Retrieval.py Update LeCaRDv2Retrieval metadata * Update LegalBenchConsumerContractsQARetrieval.py Update LegalBenchConsumerContractsQARetrieval metadata * Update LegalBenchCorporateLobbyingRetrieval.py Update LegalBenchCorporateLobbyingRetrieval metadata * Update LegalQuADRetrieval.py Update LegalQuADRetrieval metadata * Update LegalSummarizationRetrieval.py Update LegalSummarizationRetrieval metadata * Update AILACasedocsRetrieval.py Update AILACasedocsRetrieval * Update AILACasedocsRetrieval.py Update AILACasedocsRetrieval metadata * Update AILAStatutesRetrieval.py Update AILAStatutesRetrieval metadata * Update GerDaLIRSmallRetrieval.py Update GerDaLIRSmallRetrieval metadata * Update LeCaRDv2Retrieval.py Update LeCaRDv2Retrieval metadata * Update LegalBenchConsumerContractsQARetrieval.py * Update LegalBenchCorporateLobbyingRetrieval.py * Update LegalQuADRetrieval.py * Update LegalSummarizationRetrieval.py * Update AILACasedocsRetrieval.py * Update AILAStatutesRetrieval.py * Update GerDaLIRSmallRetrieval.py * Update LeCaRDv2Retrieval.py * move dataset language folder * update order --------- Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
* Fix typos in readme.md * Added news classification dataset. * Added news classification dataset. * Fixes on suggestions * Update docs/mmteb/points.md Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> --------- Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
* fix: remove time of run (as it does not relate to the model itself). Time of run should be on the dataset results * fix: fixes the PawsX datasets * docs: Updated points * fix: flores clustering * fix: mulitple dataset fixes * docs: updated points * fix: added missing dataset_transform to multitask task * syle: ran formatter * fix: correctly fix pawsX
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kenneth and I took a first stab at attributing points for the contributions so far.
These are meant as an opening for a discussion, not at all a final list, so definitely feel free to suggest changes!
@KennethEnevoldsen
4 add dataset annotations for size
1 added dataset annotations one dataset
2 added ci
3 updated readme x 3 (installation instruction, adding dataset, mmteb etc.)
1 folder structure
7 reviewed pr x 7
= 18
@MartinBernstorff
Merged PRs:
Reviews:
Total: 12