Use pre-commit hooks in tox configuration

[waxlamp]

This extends #2045 to use the same pre-commit hooks to perform linting and formatting via tox.ini. In particular, this "refactors" the specification of what versions of codespell and ruff to use to just one place: the pre-commit config file.

The variants of the pre-commit hooks that are needed to replace the tox.ini commands are listed in the pre-commit config using the manual stage specifier with appropriate aliases, meaning they will not be invoked as a git hook ever, and enabling them to be invoked by name in the tox config.

This PR also modifies the dev Dockerfile to build it to run as the user from the host system. That is, instead of running as root inside the container, it will instead run as the same (non-root) user who is doing development work (including building this container, etc.). That enables the use of pre-commit both inside and outside the container, since pre-commit requires examining Git status, etc. Without this modification, you get errors about "dubious ownership" of the Git repository.

I am totally willing to peel off the Dockerfile changes to a separate PR if we require further discussion, especially if some of these ideas are suitable for general use with Docker containers. Let me know what you think, @mvandenburgh.

[yarikoptic]

There are some quirks w.r.t. running this via Docker:

1. Because the permissions for the .git directory are not those of root,

IMHO nobody should run dev tools as root -- should be mapped to the real UID via --user option etc. (in https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L198C1-L199C19 we passed UID and GID env vars) -- what container we are talking about to try on?

[mvandenburgh]

Should the difficulties around running linting inside docker be documented in DEVELOPMENT.md?

[yarikoptic]

when do git and tox need to run inside docker environment?

[waxlamp]

IMHO nobody should run dev tools as root -- should be mapped to the real UID via --user option etc. (in https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L198C1-L199C19 we passed UID and GID env vars) -- what container we are talking about to try on?

Nobody is running dev tools as root 🙂. Docker containers, by default, run as root inside the container (even when launched by a non-root user on the host system). Your example is using Singularity, which was designed in part to work with non-root users by default.

when do git and tox need to run inside docker environment?

Tox needs to run inside the container precisely when the dev wants it to. For example, I often run the command docker compose run --rm django tox -e lint etc. to perform linting in my dev environment. Git does not need to run inside the container, though pre-commit uses git to determine how to run its hooks, which is where I was running into trouble.

Resonant applications provides this container as a convenience. Adopting pre-commit in dandi-archive caused some of the "rev matching" problems to surface, so I'm looking into how to fix it to provide a generally better experience across Resonant apps.

In any case, I am working on a way to build the container so that it runs as the same user as the dev who built it. If that works, then it will provide parity between host OS and Docker container. Stay tuned for updates.

[yarikoptic]

There are some quirks w.r.t. running this via Docker:

1. Because the permissions for the .git directory are not those of root,

IMHO nobody should run dev tools as root -- should be mapped to the real UID via --user option etc. (in https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L198C1-L199C19 we passed UID and GID env vars) -- what container we are talking about to try on?

FWIW -- checked that it works:

❯ docker compose run --rm --user "$(id -u)" django touch 123
WARN[0000] /home/yoh/proj/dandi/dandi-archive/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] /home/yoh/proj/dandi/dandi-archive/docker-compose.override.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Creating 3/0
 ✔ Container dandi-archive-minio-1     Running                                                                                                                                 0.0s 
 ✔ Container dandi-archive-rabbitmq-1  Running                                                                                                                                 0.0s 
 ✔ Container dandi-archive-postgres-1  Running                                                                                                                                 0.0s 

❯ ls -ld 123
-rw-r--r-- 1 yoh root 0 Oct 28 12:18 123

so just add --user "$(id -u)" to your invocation of tox within container. Depending on scripts, you might also benefit from passing env vars as I pointed our so more complete would be --user "$(id -u)" -e "UID=$(id -u)" -e "GID=$(id -g)" (potentially improved to your liking)

[yarikoptic]

note: might need to clean up "root owned" artifacts from prior runs of tox first, and then it seems to work just fine

❯ sudo rm -rf .tox/lint/log .ruff_cache
❯ docker compose run --rm --user "$(id -u)" -e "UID=$(id -u)" -e "GID=$(id -g)" django tox -e lint
WARN[0000] /home/yoh/proj/dandi/dandi-archive/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] /home/yoh/proj/dandi/dandi-archive/docker-compose.override.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Creating 3/0
 ✔ Container dandi-archive-postgres-1  Running                                                                                                                                 0.0s 
 ✔ Container dandi-archive-minio-1     Running                                                                                                                                 0.0s 
 ✔ Container dandi-archive-rabbitmq-1  Running                                                                                                                                 0.0s 
lint: commands[0]> ruff --version
ruff 0.6.9
lint: commands[1]> ruff check
All checks passed!
lint: commands[2]> ruff format --check
155 files already formatted
lint: commands[3]> codespell .
  lint: OK (0.34=setup[0.06]+cmd[0.01,0.02,0.02,0.23] seconds)
  congratulations :) (0.44 seconds)

[waxlamp]

so just add --user "$(id -u)" to your invocation of tox within container

This mostly works--depending on how the container image was built you can still run into problems. Plus, having to add the --user flag on every invocation is annoying and error-prone (you might forget to do it, etc.)

note: might need to clean up "root owned" artifacts from prior runs of tox first, and then it seems to work just fine

This is really the core of the issue--having to remember the --user flag, plus the fact that your root user might be "polluting" your dev directory makes for an unergonomic dev experience.

But I'm experimenting with moving these things to the build phase so that the regular commands "just work". Stand by, I have an incoming fix for all this (I hope).

[yarikoptic]

note: might need to clean up "root owned" artifacts from prior runs of tox first, and then it seems to work just fine

This is really the core of the issue--having to remember the --user flag, plus the fact that your root user might be "polluting" your dev directory makes for an unergonomic dev experience.

The core issue is :

IMHO nobody should run dev tools as root -- should be mapped to the real UID via --user option etc. (in https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L198C1-L199C19 we passed UID and GID env vars) -- what container we are talking about to try on?

Nobody is running dev tools as root 🙂. Docker containers, by default, run as root inside the container (even when launched by a non-root user on the host system).

and the fact that "Docker containers" are not animate, they do not run themselves. They are executed by people who run them and as a result running dev tools as root. It is just a fact.

Tox needs to run inside the container precisely when the dev wants it to.

It is something to remember about and to not execute docker with any dev tools/scripts without precautions being taken -- ideally to avoid that generally. Same advice for sudo - too many people often did sudo pip install causing all kinds of pains later on -- all of that is IMHO just a bad practice.

In the longer run it might be better to look to switch to podman and having fake root inside the container space so touched bind-mounted files would still be fine.

I have expressed my opinion ("never run dev tools as root" - whether through docker or sudo or ...), and will quiet down on this particular issue.

[waxlamp]

note: might need to clean up "root owned" artifacts from prior runs of tox first, and then it seems to work just fine

This is really the core of the issue--having to remember the --user flag, plus the fact that your root user might be "polluting" your dev directory makes for an unergonomic dev experience.

The core issue is :

IMHO nobody should run dev tools as root -- should be mapped to the real UID via --user option etc. (in https://github.com/ReproNim/containers/blob/master/scripts/singularity_cmd#L198C1-L199C19 we passed UID and GID env vars) -- what container we are talking about to try on?

Nobody is running dev tools as root 🙂. Docker containers, by default, run as root inside the container (even when launched by a non-root user on the host system).

and the fact that "Docker containers" are not animate, they do not run themselves. They are executed by people who run them and as a result running dev tools as root. It is just a fact.

Tox needs to run inside the container precisely when the dev wants it to.

It is something to remember about and to not execute docker with any dev tools/scripts without precautions being taken -- ideally to avoid that generally. Same advice for sudo - too many people often did sudo pip install causing all kinds of pains later on -- all of that is IMHO just a bad practice.

In the longer run it might be better to look to switch to podman and having fake root inside the container space so touched bind-mounted files would still be fine.

I have expressed my opinion ("never run dev tools as root" - whether through docker or sudo or ...), and will quiet down on this particular issue.

I did not mean to be dismissive--I was just pointing out a reality of how most people run Docker containers. As I said, I'm working on a solution that addresses this issue head on (it is similar to what you suggested, but even better).