Split CI dependencies and add ./deps:/deps mount.

[KevinMind]

Fixes: mozilla/addons#15234
Fixes: mozilla/addons#15034

Description

• Introduced a new script install_deps.py to streamline the installation of pip and npm dependencies based on specified targets.
• Updated docker-compose.yml to include a new volume for dependencies and adjusted service configurations accordingly.
• Modified Makefile-docker to incorporate checks for CI dependencies and streamline the update process.

Context

Adds a dependency mount to load dependencies from the host. Additionally optimizes the logic for installing dependencies in one batch.

Which dependencies to run is set during make up and not "mutable" because we pass it to the container via the .env file. to change it, re-run make up with a different value.

Note

You can cheat and install development depenednecies in a production running container is you run the command inside the container shell, where make will accept your OLYMPIA_DEPS=development argument

Testing

The value of OLYMPIA_DEPS defaults to the value of DOCKER_TARGET but can be overriden so there are 4 different scenarios.

Standard development mode

make up DOCKER_TARGET=development

Expect:

• volume called data_deps is created
• it is synced to your ./deps folder.
• containers start up and install prod/dev dependencies.
• Theoretically this should re-enable tracking in IDE
• running make check in the container should pass

Dev mode with Prod dependencies

make up DOCKER_TARGET=development OLYMPIA_DEPS=production

Expect:

• everything from above, but missing development dependencies.
• pytest will not work

Important

You will see in the logs during make up that the ./deps directory is cleaned. That is because we are running a development image but specifying we want production dependencies. To ensure we don't end up with stale dependencies we have to clean up the directory and freshly install, otherwise some dev dependencies could be left behind. we DON'T do this on local images that are installing development dependencies as we should not need any removed

Prod mode

This should happen if the DOCKER_TARGET is production

make up DOCKER_TARGET=production

Expect:

• Everything from above AND
• ./deps directory is cleared and dependencies are freshly installed
• expect only prod dependencies to be installed
• verify this with make check and (for example) that debug toolbar is missing on the web
• no pytest or ruff
• verify with make check

Prod mode with Dev dependencies

This does not currently work on a running container We set the OLYMPIA_DEPS variable during startup and make does not support passing arguments from the host makefile, into the container, and into the host again, at least not easily.

So to run prod mode with testing, run a production build with dev dependencies.

make up DOCKER_TARGET=production OLYMPIA_DEPS=development

This will build the production image and install the development dependencies at runtime.

Expect:

• Everything from above AND
• now you should be able to run pytest and ruff

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[KevinMind]

@diox we can simplify this if we decide it's okay to add everything in ci.txt to prod.txt but that means those dependencies ship to production. I'm happy either way.

[eviljeff]

@KevinMind CI is failing with ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref d292b4cc-acf7-406d-b50f-82233a76aea4::5khdhdz6z7m0rspdr4p5sj9wn: "/requirements/ci.txt": not found

[KevinMind]

@KevinMind CI is failing with ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref d292b4cc-acf7-406d-b50f-82233a76aea4::5khdhdz6z7m0rspdr4p5sj9wn: "/requirements/ci.txt": not found

I split off the second part of this patch as it was not strictly necessary and left that line. It's gone now.

[eviljeff]

I just did make up and got

Error response from daemon: failed to mount local volume: mount /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/3827bdbcbfb76834bf7a50fd8746e67e97d66dbb404c4ef6a62eef6bf805c55f:/var/lib/docker/volumes/addons-server_data_deps/_data, flags: 0x1000: no such file or directory

(I didn't do anything from the testing section, to check the scenario for a typical use just after the patch has been merged onto master) (edited)

[KevinMind]

@eviljeff can you try now? I added a .gitkeep in the ./deps directory. If that dir doesn't exist then docker will bail creating the volume.

[eviljeff]

works locally but:

• I expected DOCKER_TARGET=production to also remove the javascript formatting utilities (it didn't), not just ruff
• CI is failing