Gather ideas of how to shape the Dockerfile for alltalk beta

[Paladinium]

I would like to start a discussion of how to build a Dockerfile for alltalk beta, but also what the requirements are. This should support the process of crafting actual Dockerfiles.

Goal of having Docker

The advantages of Docker are quite obvious: There is no need for local installations and dealing with dependencies and incompatibilities. Instead, a Docker image should provide everything needed to run alltalk. Also, it makes it easier to ship alltalk to production.

Requirements

Just to give an idea of what kind of requirements are possible:

• When starting the image using docker run, alltalk should be up and running without the need for downloading dependencies. Startup time should therefore be in the order of seconds.
• All dependency versions should be pinned for the sake of reproducible builds (e.g. avoid conda install -y pytorch::faiss-cpu and instead use conda install -y pytorch::faiss-cpu=1.9.0)
  • On a running docker environment, complete requirements files for pip and conda can separately be extracted and added as additional alltalk requirements files.
  • This should allow for --no-deps to be used
• The docker images should be published to docker hub such that own Dockerfiles can be written using e.g. FROM alltalk/alltalk:2.0
• In turn, when building the image using docker buildx build, there should be the possibility of "configuring" the image, e.g. by passing build args, e.g. for picking the model to use or providing some other means of configuration, e.g. by mounting config files, without having to modify the actual source code of alltalk.
  • It might be required only for internal purposes, e.g. when publishing docker images per model, e.g. alltalk-xtts:2.0 or alltalk-piper:2.0
• The ability to use GPUs should be provided out-of-the box
• Extended features should be enabled/available by default such as
  • DeepSpeed
  • RVC
  • Multi-Engine-Manager
• Adhere to best practices, e.g.
  • Security: not running the application itself as root
  • Layering: Make sure Docker layers are cacheable (which they are not when running scripts)
• Forward logs to console
• ...

Misc

Feel free to provide examples of Dockerfiles or state your further/other requirements.

[Paladinium]

An example of a Dockerfile which packages alltalk and picking xtts as model, yet not fulfilling many of the requirements, is:

FROM continuumio/miniconda3:24.7.1-0

SHELL ["/bin/bash", "-l", "-c"]
ENV SHELL=/bin/bash
ENV HOST=0.0.0.0
ENV DEBIAN_FRONTEND=noninteractive
ENV CUDA_DOCKER_ARCH=all

RUN <<EOR
apt-get update
# apt-get upgrade -y
apt-get install --no-install-recommends -y libaio-dev espeak-ng ffmpeg gcc g++ portaudio19-dev \
  git curl wget expect \
  freeglut3-dev vim

apt-get clean && rm -rf /var/lib/apt/lists/*
EOR

WORKDIR /alltalk

#RUN conda init bash
ARG INSTALL_ENV_DIR=/alltalk/alltalk_environment/env
ENV CONDA_AUTO_UPDATE_CONDA="false"
RUN <<EOR
    conda create -y -n "alltalk" -c conda-forge python=3.11.9
    conda activate alltalk
    conda install -y pytorch=2.2.1 torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
    conda install -y nvidia/label/cuda-12.1.0::cuda-toolkit=12.1
    conda install -y pytorch::faiss-cpu
    conda install -y conda-forge::ffmpeg
    conda clean -a && pip cache purge
EOR

# Install python dependencies (cannot use --no-deps because requirements are not complete)
COPY system/config system/config
COPY system/requirements/requirements_standalone.txt system/requirements/requirements_standalone.txt
COPY system/requirements/requirements_parler.txt system/requirements/requirements_parler.txt
ENV PIP_CACHE_DIR=/alltalk/pip_cache
RUN <<EOR
    conda activate alltalk

    mkdir /alltalk/pip_cache
    pip install --no-cache-dir --cache-dir=/alltalk/pip_cache -r system/requirements/requirements_standalone.txt
    pip install --no-cache-dir --cache-dir=/alltalk/pip_cache --upgrade gradio==4.32.2

    # Deepspeed:
    curl -LO https://github.com/erew123/alltalk_tts/releases/download/DeepSpeed-14.0/deepspeed-0.14.2+cu121torch2.2-cp311-cp311-manylinux_2_24_x86_64.whl
    pip install --no-cache-dir --cache-dir=/alltalk/pip_cache deepspeed-0.14.2+cu121torch2.2-cp311-cp311-manylinux_2_24_x86_64.whl
    rm -f deepspeed-0.14.2+cu121torch2.2-cp311-cp311-manylinux_2_24_x86_64.whl

    # Parler:
    pip install --no-cache-dir --no-deps --cache-dir=/alltalk/pip_cache -r system/requirements/requirements_parler.txt

    conda clean --all --force-pkgs-dirs -y && pip cache purge
EOR

# Writing scripts to start alltalk:
RUN <<EOR
    cat << EOF > start_alltalk.sh
#!/usr/bin/env bash
source ~/.bashrc
conda activate alltalk
python script.py
EOF
    cat << EOF > start_finetune.sh
#!/usr/bin/env bash
source ~/.bashrc
export TRAINER_TELEMETRY=0
conda activate alltalk
python finetune.py
EOF
    cat << EOF > start_diagnostics.sh
#!/usr/bin/env bash
source ~/.bashrc
conda activate alltalk
python diagnostics.py
EOF
    chmod +x start_alltalk.sh
    chmod +x start_environment.sh
    chmod +x start_finetune.sh
    chmod +x start_diagnostics.sh
EOR

COPY . .

# Create scripts to execute firstrun.py:
# Options are: 1 = piper, 2 = vits, 3 = xtts
# TODO: Pass this option as ARG to the Dockerfile
RUN echo $'#!/usr/bin/env bash \n\
source ~/.bashrc \n\
conda activate alltalk \n\
python ./system/config/firstrun.py' > ./start_firstrun.sh
RUN echo $'#!/usr/bin/env expect \n\
expect_before timeout { exit 1 } \n\
set timeout 600 \n\
spawn ./start_firstrun.sh \n\
expect "Enter your choice" \n\
send -- "3\r" \n\
expect eof' > ./start_firstrun_noninteractive.sh

# Run start_firstrun_noninteractive.sh:
RUN chmod +x start_firstrun.sh start_firstrun_noninteractive.sh
RUN ./start_firstrun_noninteractive.sh

RUN mkdir -p /alltalk/outputs
RUN mkdir -p /root/.triton/autotune

# Enabling deepspeed for all models (TODO: isn't there a config option?):
RUN find . -name model_settings.json -exec sed -i -e 's/"deepspeed_enabled": false/"deepspeed_enabled": true/g' {} \;

## Start alltalk exposing the API port:
RUN chmod +x start_alltalk.sh
EXPOSE 7851
ENTRYPOINT ["sh", "-c", "./start_alltalk.sh"]

This can be built with docker buildx build -t alltalk:latest . and can be run with e.g. docker run -p 7851:7851 --gpus=all -it alltalk:latest

[erew123]

Hi @Paladinium What you have put above sounds great :) and I would love to have your help to do this. However, I have a small bit of code to complete withing the script.py code so that it identifies the Gradio interface is running within Docker. The API address will work fine, but the Gradio interface has to be pointed to the external IP address of wherever the API address is running from. If you are running docker on a local machine, it should (I think) work, but if you are running it somewhere on the internet in the cloud, it wont work.

Ive already done Google Colab, so that automatically picks up the tunnel addresses and pushes the IP/URL into the code as necessary when AllTalk starts up and I am kind of doing the same with when it detects a docker style environment. I have about 95% of the code written (somewhere on my home machine) but then this happened link here a couple of months ago and I have been traveling away from home for a couple of months now.

I still dont have a timeline when I will be back and able to finish it, it could be a week, it could be a month and I suppose there could be other possible timelines too (will make sense when you read the link).

I think I will be updating the Pytorch version in future, which may knock a few other requirements versions in the requirements files, but I don't think that would cause you any blips. However, please see my notes about the transformers versions on the link I gave above. Its out of my hands getting a later version of transformers for Coqui TTS streaming to work, its handled by idiap, but they do appear to be discussing it/working on it https://github.com/idiap/coqui-ai-TTS. Also, Microsoft are finally starting to release pre-built wheels for Pytorch, so we may be able to do a generic link to just pip install deepspeed in future, but chatting with Logan at MS, they are currently having a few issues and of course, it will also depend on the Pytorch version used, so thats a drive to update it one day soon.

So, you should be able to get a docker build running on the API and call it with Command prompt CURL/SillyTavern Extension/TGWUI Remote Extension/, but the Gradio interface wont work yet*. I am not great on Docker at all, but my brief glance says what you have done above and suggest looks good to me! :)

Hopefully I can get back to you soon, or maybe remote into my home machine and try check the code and post it up, though I have to be careful to make sure it doesnt knock something else out of whack and Im not exactly on the best laptop in the world to be able to remote control my home machine for coding. :/

If anyone else has any other thoughts/ideas and would like to get involved in this chat, it would be great!

And thanks again @Paladinium

[vlrevolution]

@Paladinium Ive managed to finish the updates for Gradio allowing you to manually set the docker IP/URL in the interface. What I may also do at some point is allow a choice on startup in docker for Gradio's public shared address.

However, for now, if I have gotten things correct (which I havnt tested fully in docker if Im honest) you should be able to update the Gradio interface to point to the Docker IP/URL for the API, when you know what it is:

0c9034c

Sorry Ive not gotten to the PR yet, things are on/off very busy for me outside of all this and when I come back to do/test bits, there are support ticket requests etc to deal with. Will be on to it shortly though.

Thanks

This setting for docker ip/host usign ngrok testing it in a vast.ai machine is not currently fixing the access issue. Maybe there's something wrong still?

[Paladinium]

@vlrevolution : Did you use the example I posted as 1st comment in this discussion using the code base from the alltalkbeta branch?

Just to be clear: This was an example and not yet fully completed. I was busy contributing at other places in this project, so no progress so far.

[vlrevolution]

@vlrevolution : Did you use the example I posted as 1st comment in this discussion using the code base from the alltalkbeta branch?

Just to be clear: This was an example and not yet fully completed. I was busy contributing at other places in this project, so no progress so far.

No, I was talking about being able to change the docker ip in the panel, the image that @erew123 posted in his reply. I think right now if it detects docker it breaks the url it sends the api requests to in gradio, even if you change the ip in the panel (has no effect). I had to comment out the docker detection code to get it to use even the correct localhost ip (using ssh tunneling).

[erew123]

@vlrevolution The new update should (I say should) work correctly with docker, as far as the Gradio interface goes. I have not tested in docker, only faked the flag in the code to say its in docker and test that it works.

Obviously though @Paladinium will probably be testing a docker build at some point in the future.

[vlrevolution]

@vlrevolution The new update should (I say should) work correctly with docker, as far as the Gradio interface goes. I have not tested in docker, only faked the flag in the code to say its in docker and test that it works.

Obviously though @Paladinium will probably be testing a docker build at some point in the future.

Thank you, I will test in near future and report back

[erew123]

I came across this for optimising docker image sizes. Its supposedly good, but I don't know enough about docker to say how easy this would be to use or what impact it may have. I'm leaving it here for reference as I'm happy to give it a shot when a docker image gets built https://github.com/slimtoolkit/slim

Claims of up to 30x reduction in size

[vlrevolution]

Looking at issues tho shouldnt get hopes up. AI docker images are already very hard to get working in the first place

…

On Sun, 8 Dec 2024, 11:24 erew123, ***@***.***> wrote: I came across this for optimising docker image sizes. Its supposedly good, but I don't know enough about docker to say how easy this would be to use or what impact it may have. I'm leaving it here for reference as I'm happy to give it a shot when a docker image gets built https://github.com/slimtoolkit/slim Claims of up to 30x reduction in size — Reply to this email directly, view it on GitHub <#378 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQC2ZKTXQUU7RESCD7MYMMD2EQF6BAVCNFSM6AAAAABP5BHJT2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNBZHA2DGNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RenNagasaki]

@vlrevolution it sounds like you managed to get a docker image working, if yes, would you mind sharing that one with us? I'd love to test that. 😄

[vlrevolution]

The actual Dockerfile can be seen here:
https://github.com/pytorch/pytorch/blob/main/Dockerfile

[Paladinium]

I have a docker file for which I intend to create a PR in this project soon for the alltalkbeta branch.

[RenNagasaki]

That sounds awesome!

[Paladinium]

Here we go: #457

The PR yet has to be approved by @erew123 .

Looking forward getting feedback from you.

[vlrevolution]

Nice! I will test it tomorrow, thanks for the efforts! 🌞