Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extremely slow model loading for BNB 4bit saved weights #29810

Closed
2 of 4 tasks
aliencaocao opened this issue Mar 22, 2024 · 2 comments
Closed
2 of 4 tasks

Extremely slow model loading for BNB 4bit saved weights #29810

aliencaocao opened this issue Mar 22, 2024 · 2 comments

Comments

@aliencaocao
Copy link
Contributor

aliencaocao commented Mar 22, 2024

System Info

  • transformers version: 4.39.0
  • Platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
  • Python version: 3.10.12
  • Huggingface_hub version: 0.19.4
  • Safetensors version: 0.4.1
  • Accelerate version: 0.28.0
  • Accelerate config: not found
  • PyTorch version (GPU?): 2.2.1+cu121 (True)
  • Tensorflow version (GPU?): 2.16.1 (True)
  • Flax version (CPU?/GPU?/TPU?): not installed (NA)
  • Jax version: not installed
  • JaxLib version: not installed
  • Using GPU in script?: yes
  • Using distributed or parallel set-up in script?: no

Who can help?

@SunMarc @ArthurZucker

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

I am converting the original llava1.6 mistral 7b into BNB 4 bit saved model by inserting model.save_pretrained() at https://github.com/haotian-liu/LLaVA/blob/7440ec9ee37b0374c6b5548818e89878e38f3353/llava/model/builder.py#L114
The saved model is mere 4.6GB: https://huggingface.co/panoyo9829/llava-v1.6-mistral-7b-bnb-4bit
However, it takes a good 77.8 seconds on a AMD 5800X (8core16 thread) to load it. It takes 9.2seconds on a 80 core Intel Xeon CPU to load. This seems too slow. I am on a NVMe SSD but I see close to 0% disk IO.

Expected behavior

When loading fp16 models, it take less than 30 seconds and disk IO is very much occupied, while when loading a BNB 4bit model, i see almost 0 disk IO.
Something related to loading BNB model must have been affecting it a lot.
Possible regression from #26037

I am fully aware that HF now supports llava-next natively and I have yet to try saving bnb 4bit weights from the hf converted weights, but my point is this is not related to the model itself since fp16 loads fine.

@aliencaocao
Copy link
Contributor Author

My bad, was actually a WSL2 bug...microsoft/WSL#4197

@aliencaocao
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant