Skip to content

Latest commit

 

History

History
111 lines (75 loc) · 3.4 KB

README.md

File metadata and controls

111 lines (75 loc) · 3.4 KB
Raw

Safetensors vs CKPT

Vidéo du tutoriel : https://youtu.be/FcBLxDyOwrA.

Template original : https://github.com/NeuronAddict/python-ai-template

Template for python container for AI

This repo extend https://github.com/NeuronAddict/python-template (python template).

It add to this template :

  • notebook facilities
  • nbclean with pre-commit to add book

Doc for the initial template

When you build a python container with a requirement.txt without explicit versions, it sometimes fail (too often!).

Why? This is because without versions in this file, your build use the last versions of all dependencies. This work at a moment (when develop) but can fail when YOU build code.

To avoid that, this template help us to provides a unique build for a commit.

  • Docker rootless container for python
  • Include only python dependencies

How To

set python dependencies

  • Add needed dev dependencies in dev-requirements.in. There are not deployed on final container (like pytest).
  • Add run dependencies (packages you need to run in production) in requirements.in. Add versions only if needed.

Exemple

requirements.in

django
psycopg2

set dev env

$ python -m venv venv # may be python3
$ source venv/bin/activate
$ pip install pip-tools # only when dev-requirements.txt is void
$ # remove your local index-url and options to not leak this on github
$ pip-compile --no-emit-index-url --no-emit-options requirements.in
$ # add a constraint to dev-requirements to avoid a different version on dev env
$ pip-compile --constraint=requirements.txt --no-emit-index-url --no-emit-options dev-requirements.in 
$ pip install -r requirements.txt
$ pip install -r dev-requirements.txt

Commit all requirements files (in and txt). Like this your commit give a unique (and reproducible!) build.

Now all *.txt files has unique versions and your commit work now AND anytime in the future.

update packages

To update or add packages, simply use pip-tools :

$ # remove your local index-url and options to not leak this on github
$ pip-compile --upgrade --no-emit-index-url --no-emit-options requirements.in
$ # add a constraint to dev-requirements to avoid a different version on dev env
$ pip-compile --constraint=requirements.txt --no-emit-index-url --no-emit-options dev-requirements.in 
$ # now *.txt files are updated with new versions
$ pip install -r dev-requirements.txt
$ pip install -r requirements.txt
$ git add *
$ git commit

https://pypi.org/project/pip-tools/

This create a new commit with all new versions. You can make a rollback if needed.

Good practices

Exclude files from docker container

Ignore useless files with a dockerignore to not include they in your container :

  • Dockerfile # can leak info
  • .git (can be very big and leak a lot of info)
  • *.md # useless
  • venv/ (may be big!)
  • doc (should not be in a container)
  • *.in
  • dev-requirements.txt
  • .gitlab-ci.yaml, Jenkinsfile (can leak info)
  • .dockerignore itself to not leak ignored containers

One commit, one build

Create a new container version when upgrade packages to keep versions consistent accross time. A commit should not give a different build when rebuild after a long time.

Use staged build

Like the example, use a staged build to :

  • Add all required packages on build container
  • Add only required packages on production build and clean apt-get cache in same layer

https://docs.docker.com/build/building/multi-stage/