error running with --load-in-4bit

[iChristGit]

Loading llama-7b...
Traceback (most recent call last):
File "D:\MachineLearning\TextWebui\text-generation-webui\server.py", line 194, in
shared.model, shared.tokenizer = load_model(shared.model_name)
File "D:\MachineLearning\TextWebui\text-generation-webui\modules\models.py", line 94, in load_model
from llama import load_quant
ModuleNotFoundError: No module named 'llama'
Press any key to continue . . .

Windows11, 3090Ti
Tried 7B, 13B and 30B.

--load-in-8bit works
commit:
026d60b

[oobabooga]

4-bit requires additional installation steps. See here:

https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model#4-bit-mode

[iChristGit]

4-bit requires additional installation steps. See here:

https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model#4-bit-mode

Didnt see that note, I followed the steps, getting error regarding cuda, although i have cuda v11.8 running and the webui is working
After last step (python setup_cuda.py install)

D:\MachineLearning\TextWebui\text-generation-webui\repositories\GPTQ-for-LLaMa>python setup_cuda.py install
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8'
running install
D:\MachineLearning\Python\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
D:\MachineLearning\Python\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
writing quant_cuda.egg-info\PKG-INFO
writing dependency_links to quant_cuda.egg-info\dependency_links.txt
writing top-level names to quant_cuda.egg-info\top_level.txt
D:\MachineLearning\Python\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'quant_cuda.egg-info\SOURCES.txt'
writing manifest file 'quant_cuda.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_ext
Traceback (most recent call last):
File "D:\MachineLearning\TextWebui\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 4, in
setup(
File "D:\MachineLearning\Python\lib\site-packages\setuptools_init_.py", line 87, in setup
return distutils.core.setup(**attrs)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\core.py", line 177, in setup
return run_commands(dist)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\core.py", line 193, in run_commands
dist.run_commands()
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\dist.py", line 968, in run_commands
self.run_command(cmd)
File "D:\MachineLearning\Python\lib\site-packages\setuptools\dist.py", line 1217, in run_command
super().run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\dist.py", line 987, in run_command
cmd_obj.run()
File "D:\MachineLearning\Python\lib\site-packages\setuptools\command\install.py", line 74, in run
self.do_egg_install()
File "D:\MachineLearning\Python\lib\site-packages\setuptools\command\install.py", line 123, in do_egg_install
self.run_command('bdist_egg')
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\cmd.py", line 317, in run_command
self.distribution.run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools\dist.py", line 1217, in run_command
super().run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\dist.py", line 987, in run_command
cmd_obj.run()
File "D:\MachineLearning\Python\lib\site-packages\setuptools\command\bdist_egg.py", line 165, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "D:\MachineLearning\Python\lib\site-packages\setuptools\command\bdist_egg.py", line 151, in call_command
self.run_command(cmdname)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\cmd.py", line 317, in run_command
self.distribution.run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools\dist.py", line 1217, in run_command
super().run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\dist.py", line 987, in run_command
cmd_obj.run()
File "D:\MachineLearning\Python\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\command\install_lib.py", line 112, in build
self.run_command('build_ext')
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\cmd.py", line 317, in run_command
self.distribution.run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools\dist.py", line 1217, in run_command
super().run_command(command)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\dist.py", line 987, in run_command
cmd_obj.run()
File "D:\MachineLearning\Python\lib\site-packages\setuptools\command\build_ext.py", line 79, in run
_build_ext.run(self)
File "D:\MachineLearning\Python\lib\site-packages\setuptools_distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "D:\MachineLearning\Python\lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
_check_cuda_version(compiler_name, compiler_version)
File "D:\MachineLearning\Python\lib\site-packages\torch\utils\cpp_extension.py", line 382, in _check_cuda_version
torch_cuda_version = packaging.version.parse(torch.version.cuda)
File "D:\MachineLearning\Python\lib\site-packages\pkg_resources_vendor\packaging\version.py", line 49, in parse
return Version(version)
File "D:\MachineLearning\Python\lib\site-packages\pkg_resources_vendor\packaging\version.py", line 264, in init
match = self._regex.search(version)
TypeError: expected string or bytes-like object

[MarkSchmidty]

You need to follow the Windows specific GPTQ 4bit compilation instructions in this issue on GPTQ-for-LLaMA: qwopqwop200/GPTQ-for-LLaMa#11 (comment)

[iChristGit]

You need to follow the Windows specific GPTQ 4bit compilation instructions in this issue on GPTQ-for-LLaMA: qwopqwop200/GPTQ-for-LLaMa#11 (comment)

I am getting more errors, Is that only because of Visual studio 2022?
I have to use 2019 version? or maybe wait for new development

[oobabooga]

In another issue someone said that Visual studio 2022 doesn't work and that an older version was needed.

I can't confirm but it could be worth trying.

[iChristGit]

In another issue someone said that Visual studio 2022 doesn't work and that an older version was needed.

I can't confirm but it could be worth trying.

I also tried it in Linux new installation, same error
is there any workaround?

[okageblade]

2019 works. Use native tools command prompt. 30b 4bit takes 40 seconds to respond on my 3090 however, so YMMV on its usability.

[iChristGit]

2019 works. Use native tools command prompt. 30b 4bit takes 40 seconds to respond on my 3090 however, so YMMV on its usability.

After i compile it can I go back to VS2022 or I need to stay 2019?
I have a lot of stuff that needs VS2022

[okageblade]

You can install it alongside 2022. Just need it for the install but there's no harm in just keeping it in case you need it again I suppose.

[iChristGit]

You can install it alongside 2022. Just need it for the install but there's no harm in just keeping it in case you need it again I suppose.

I installed the 2019 version, now getting CUDA Extension not installed
after finishing the compiling
I really dont know, maybe its beacuse I used to automatic installer for the webui?

[okageblade]

Make sure the conda env you install the extension on and the one that runs the server.py is the same and activated

[iChristGit]

Make sure the conda env you install the extension on and the one that runs the server.py is the same and activated

I am really a newbie haha
I use the automatic installer so I have the install files folder with env
And I copied to commands from here
qwopqwop200/GPTQ-for-LLaMa#11 (comment)

changed the relevant paths to where my conda stuff is.. I dont really understand how to fix that

[gsgoldma]

why is this closed, wasn't his issue unresolved?

[github-actions]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.