Illegal hardware instruction in quantize step

[tunglee35]

• Ran into this error on a Macbook Pro M1

./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2
[1]    18452 illegal hardware instruction  ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2

• What I've tried:

  • Run main on the model-f16, still have the same error
  • Convert the 13B model, still same error in quantize step

• Env:
  Darwin Tungs-MacBook-Pro.local 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000 x86_64

[prusnak]

What does cc --version and c++ --version say?

[brozkrut]

This error happens when binary is compiled for x86_64 instead of arm64.

This is common mistake if you are compiling under conda like environment.

Check binary quantize with:
file quantize

On Apple Silicon Macs it should print:
quantize: Mach-O 64-bit executable arm64

If it shows x86_64 then you have to recompile after exiting conda environment:
make clean
make

[tunglee35]

@prusnak it said

Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[tunglee35]

Solved by re-install conda env. Thanks guys

[bowbowbow]

@brozkrut

Thank you, it works as you said.

[JerryYao80]

@brozkrut I got the error in Windows 7:

ERROR: /app/.devops/tools.sh: line 40 6 Illegal instruction ./main $arg2

when I executed :

--quantize "/models/7B/ggml-model-f16.bin" "/models/7B/ggml-model-q4_0.bin" 2

and I checked my quantize file:

quantize: ELF 64-bit LSB shared object, x86-64, version 1 <GNU/Linux> ...... for GNU/Linux 3.2.0, not stripped

But the architecture of docker image named ghcr.io/ggerganov/llama.cpp:full is amd64:

docker image ghcr.io/ggerganov/llama.cpp:full|grep Architecture
"Architecture": "amd64"

and my environment is:

Docker Toolbox 1.13.1
docker client: 1.13.1 os/arch: windows 7 /amd64
docker server:19.03.12 os/arch:ubuntu 22.04 /amd64

SO I'M CONFUSED:
1 Must I re-compile quantize?
2 How to re-compile if I use docker image?
3 Is there anywhere that I can download directly?

[amanreddy77]

This error happens when binary is compiled for x86_64 instead of arm64.

This is common mistake if you are compiling under conda like environment.

Check binary quantize with: file quantize

On Apple Silicon Macs it should print: quantize: Mach-O 64-bit executable arm64

If it shows x86_64 then you have to recompile after exiting conda environment: make clean make can you explain me clearly

[amanreddy77]

Solved by re-install conda env. Thanks guys

can you please explain me