Skip to content

AMD ROCm
Compare
Choose a tag to compare
@jllllll jllllll released this 04 Aug 07:27
· 97 commits to main since this release
rocm
7578ef0
This commit was signed with the committer’s verified signature.
jllllll
GPG key ID: 7FCD00C417935797
Learn about vigilant mode.

All wheels built for AVX2 CPUs for now.

Linux

Wheels Built for ROCm 5.4.2, 5.5 and 5.6.1.

Windows

Should be considered experimental and may not work at all. Windows ROCm is very new.

To test it, you will need ROCm for Windows: https://www.amd.com/en/developer/rocm-hub/hip-sdk.html
Consult the possibly inaccurate GPU compatibility chart here: https://rocm.docs.amd.com/en/docs-5.5.1/release/windows_support.html
If your GPU isn't on that list, or it just doesn't work, you may need to build llama-cpp-python manually and hope your GPU is compatible.
Another option is to do this: ggerganov/llama.cpp#1087 (comment)

Pre-0.1.80 wheels built using ggerganov/llama.cpp#1087

Installation

To install, you can use this command:

python -m pip install llama-cpp-python --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/rocm5.5

This will install the latest llama-cpp-python version available from here for ROCm 5.5. You can change rocm5.5 to change the ROCm version.
Supported ROCm versions:

  • Windows
    • 5.5.1
  • Linux
    • 5.4.2 5.5 5.6.1
    • Some adjacent versions of ROCm may also be compatible.
      For example, 5.4.1 should be compatible with the 5.4.2 wheel.

GitHub Actions workflow here: https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/blob/main/.github/workflows/build-wheel-rocm.yml

Assets 1,995
Loading