Add Nemotron GGUF Loading Support #34725
Conversation
| @@ -129,6 +129,9 @@ def load_gguf_checkpoint(gguf_checkpoint_path, return_tensors=False): | |||
| ) | |||
| model_size = m.group().strip("-") # only keeps `7b` | |||
|
|
|||
| if "nemotron" in architecture: | |||
There was a problem hiding this comment.
Why you explicitly assign architecture to updated one if it is the same?
There was a problem hiding this comment.
I'm sorry if my answer seems obvious, but isn't it for addressing cases where the "architecture" does not only contain "nemotron"? I took reference on what you did for the qwen2moe, so I think It's better to also do it for nemotron. But I tested it without these lines and it passes through. What do you think? And thank you for reviewing! As this is my first time contributing, please let me know if anything seems odds or is there any better implementation. Thank you!
There was a problem hiding this comment.
No, for qwen2moe, I explicitly assigned another architecture name, because gguf file contains qwen2moe, but later, execution chain expects to get qwen2_moe for config, model processing and so on. You provided the same name "nemotron". So, there is no reason to explicitly assign updated architecture to the same name and even to mention nemotron, because gguf processing takes it from config by default.
There was a problem hiding this comment.
I'm sorry for misreading the code and thank you for pointing it out! I have deleted the unnecessary lines in the new commit. Please let me know if there is something needs to be fixed.
There was a problem hiding this comment.
No worries, other changes is fine from my perspective
There was a problem hiding this comment.
Thank you for reviewing!
|
Could you please take a look at this PR? @SunMarc @lyaronskaya @ArthurZucker . |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
* Add Nemotron GGUF Loading Support * fix the Nemotron architecture assignation --------- Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Add Nemotron GGUF Loading Support * fix the Nemotron architecture assignation --------- Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
What does this PR do?
Add Nemotron GGUF loading support
Before submitting
Pull Request section?
to it if that's the case. Link: Community contribution: Adding GGUF support for more architectures #33260
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@SunMarc @LysandreJik @ArthurZucker