-
Notifications
You must be signed in to change notification settings - Fork 27.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve gguf tensor processing #34515
Improve gguf tensor processing #34515
Conversation
Hey @SunMarc! I made some refactoring changes regarding tensor GGUF processing. Please, take a look. What do you think? |
4b412ab
to
a090a84
Compare
Hey @SunMarc! Just a kind ping, does it make sense to have these changes? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for making tensor processing cleaner. I think it makes sense to do that as it will be easier to read the code. Left a few minor comments. Can you have a second look @Isotr0py ?
a090a84
to
ddcea0f
Compare
8876cc6
to
98810a3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM! Thanks for improving this!
0060aac
to
254b073
Compare
Hey @SunMarc! Do you have any further comments or it looks good to you? |
Hey @SunMarc! Just a kind reminder in order to not loose this PR. |
254b073
to
4153231
Compare
4153231
to
386d4d8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @VladOS95-cyber as always ! We were away for a week, sorry for the delay. LGTM, just a nit
if name is None: | ||
continue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe add a comment above to say that you modified name to None in order to skip the processing done below as we already did it above with processor.process
. Or we can refactor a bit to also include what we have below to be included in processor.process
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I very much like this split! Thanks @VladOS95-cyber!
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Feel free to merge when happy @SunMarc |
* add tensor processing system to separate logic for models * format refactoring * small fix * make some methods private * move custom methods to processors * refactor tensor processing * format fix
* add tensor processing system to separate logic for models * format refactoring * small fix * make some methods private * move custom methods to processors * refactor tensor processing * format fix
What does this PR do?
Adding more and more architectures with custom weights conversion is making tensor processing system bigger, less flexible and readable.
Improve GGUF tensor processing system in modeling_gguf_pytorch_utils.py by adding TensorProcessor class for general cases without custom weights changes and all necessary separated tensor processor classes based on architecture with custom process logic.
Before submitting
Pull Request section?
to it if that's the case. Link: Community contribution: Adding GGUF support for more architectures #33260
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Regarding the task @SunMarc @LysandreJik @ArthurZucker .