HF: switch conditional checks to self.backend from AUTO_MODEL_CLASS
#2353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
baberabb
requested review from
haileyschoelkopf and
lintangsutawika
as code owners
haileyschoelkopf
requested changes
Sep 26, 2024
There was a problem hiding this comment.
Thanks a bunch for working on this!!
Broadly this is the right approach, but I had a different thought on how we should handle things while maintaining user back-compatibility--described in my PR comments. That should also ensure that we still respect novel subclass-overridden self.AUTO_MODEL_CLASS values.
|
@haileyschoelkopf ready to review again! Still need to test, but wanted to make sure if I understood you correctly. |
# Conflicts: # tests/models/test_huggingface.py
haileyschoelkopf
approved these changes
Oct 8, 2024
|
We discussed offline and resolved comments! |
mariagrandury
pushed a commit
to somosnlp/lm-evaluation-harness
that referenced
this pull request
Oct 9, 2024
…S` (EleutherAI#2353) * switch conditional checks to `self.backend` * nit * nit * commit feedback * fix test; update precommit hooks * add escape hatch for custom self.AUTO_MODEL_CLASS * add escape hatch for custom self.AUTO_MODEL_CLASS * fix * move assertion * add logging messages * update AUTO_MODEL_CLASS behavior in _get_backend --------- Co-authored-by: haileyschoelkopf <hailey@eleuther.ai>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The conditions in HFLM now check for either
causalorseq2seqrather than checking for theAUTO_MODEL_CLASS