09 ‐ Docker
Docker Compose is a way of installing and launching the web UI in an isolated Ubuntu image using only a few commands.
In order to create the image as described in the main README, you must have Docker Compose installed (2.17 or higher is recommended):
~$ docker compose version
Docker Compose version v2.21.0
The installation instructions for various Linux distributions can be found here:
https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
Use these commands to launch the image:
cd text-generation-webui
ln -s docker/{nvidia/Dockerfile,nvidia/docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
# Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model
docker compose up --build
By @loeken.
A video walking you through the setup can be found here:
in the the “software updater” update drivers to the last version of the prop driver.
to switch using to new driver
sudo apt update
sudo apt-get install curl
sudo mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-compose -y
sudo usermod -aG docker $USER
newgrp dockercurl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/ubuntu22.04/amd64 /" | \
sudo tee /etc/apt/sources.list.d/nvidia.list > /dev/null
sudo apt update
sudo apt install nvidia-docker2 nvidia-container-runtime -y
sudo systemctl restart dockergit clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
download and place the models inside the models folder. tested with:
4bit https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1483891617 https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1483941105
8bit: https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1484235789
edit .env values to your needs.
cp .env.example .env
nano .envdocker compose up --buildmanjaro/arch is similar to ubuntu just the dependency installation is more convenient
sudo mhwd -a pci nonfree 0300rebootyay -S docker docker-compose buildkit gcc nvidia-docker
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl restart docker # required by nvidia-container-runtimecontinue at 5. clone the repo
A video walking you through the setup can be found here:
install package manager (https://chocolatey.org/ )
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
choco install nvidia-display-driver cuda git docker-desktop
wsl --install
after reboot enter username/password in wsl
clone the repo and edit .env values to your needs.
cd Desktop
git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
COPY .env.example .env
notepad .env
download and place the models inside the models folder. tested with:
4bit https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1483891617 https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1483941105
8bit: https://github.com/oobabooga/text-generation-webui/pull/530#issuecomment-1484235789
docker compose up
on older ubuntus you can manually install the docker compose plugin like this:
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.17.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose
export PATH="$HOME/.docker/cli-plugins:$PATH"
An external repository maintains a docker wrapper for this project as well as several pre-configured 'one-click' docker compose variants (e.g., updated branches of GPTQ). It can be found at: Atinoda/text-generation-webui-docker.