build: update backend dependency management to pdm + dockerfile

[spwoodcock]

Closes #4642

Looks like this issue has been open for a while.

Great suggestion by @frafra to use PDM. Thank you.

Notes:

• I updated the Dockerfile.backend to reflect the changes.
• I also added the --no-cache flag to the final stage apk install command to save some space.
• I pinned the Python version number for version solving to >=3.7,<=3.8.
• The values for pyproject.toml were taken from the repo. Let me know if they need to change.

[spwoodcock]

This is only a partial solution, as there are multiple references to requirements.txt throughout the repo.

python-venv is also used, but is now redundant with using pdm.

I have a commit on this branch that I have not pushed yet, removing requirements.txt and modifying any docs / workflows with reference to it (14 files).
Let me know if you would like me to push.

[Aadesh-Baral]

Thanks for the PR @swoodcoc. I think it will be easier to review if you push it and maybe you can also describe what the user needs to do to set up the project (like instead of pip install -r requirements.txt) and how this change benefits the project.

[spwoodcock]

No worries, I'll add it now.
I was worried it would increase the scope, as it also affects CI and deployment instructions.

I have updated all references to requirements.txt and using venv that I could find in the repo.

As for the benefit:

• PDM uses a dependency solver to check for compatibility between all dependencies.
• If a package is updated in the future that is incompatible with other package version, then it should be flagged during build.
• I also think PEP 582 dependency management is a lot simpler and reduces the need for virtual environments.

[spwoodcock]

So it actually looks like you can't pip install dependencies directly from a pyproject.toml (I have never had to before).
There is a related issue pypa/pip#8049

I wanted to keep the edits to CI etc simple by pip installing the already solved dependency list.

To achieve this we can

pip install toml && python -c 'import toml; c = toml.load("pyproject.toml")
print("\n".join(c["project"]["dependencies"]))' | pip install -r /dev/stdin

I will update.

[spwoodcock]

Everything should be complete to the best of my knowledge.

There seem to be quite a few Dockerfiles for the backend to maintain - I updated them all, as I wasn't sure which ones are redundant or still used by external processes.

[spwoodcock]

Also on a side note: the Dockerfile using quay.io/hotosm/base-python-image as a base uses Python 3.8.7, while the Dockerfile.backend uses 3.7. Perhaps a refactor is required to manage a consistent Python version.

The dependency install worked without issue (no conflicts) using Python 3.9, so I feel like both could be updated to that anyway.

[Aadesh-Baral]

Some dependencies failed to install while setting up locally without docker but no such issues while using docker, other than that this looks good to me. Also, we may need to update the doc mentioning that the python version set in pyproject.toml should be preinstalled on the system before setting up locally without docker.

[spwoodcock]

Thanks for the feedback, I'm more than happy to go through the docs and update as requested.

As for the package installs, are you able to pip install the packages, but not pdm install? I would be surprised if this is the case.

I imagine the reason install failed is because of missing libraries on your Linux system, such as proj-dev.

A local install requires all the libs packaged into the docker image to be installed on your machine:

tasking-manager/Dockerfile

Lines 11 to 22 in 4757148

	RUN apk update && \
	apk add \
	postgresql-dev \
	gcc \
	g++ \
	python3-dev \
	musl-dev \
	libffi-dev \
	geos-dev \
	proj-util \
	proj-dev \
	make

The Dockerfile above uses Alpine, so the packages would be named differently for example on a Debian based distro (e.g. Ubuntu). proj-dev --> libproj-dev etc.

[spwoodcock]

I rebased and added the suggested changes to the docs.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[eternaltyro]

Related to #5705

[spwoodcock]

I could update this PR and rebase with the current develop, if of interest.

[eternaltyro]

Thank you, @spwoodcock . We are currently swamped and unable to review your upgrades immediately. Can you please give us a week or two to get develop branch cleaned-up and in order? I will keep you posted or you can follow the issues here on GH.

[eternaltyro]

@ramyaragupathy @Aadesh-Baral let us discuss this in our next meeting and go over timelines and effort involved.

[spwoodcock]

Of course @eternaltyro, no rush at all! Send me a message over any platform when you have time for this 👍

[eternaltyro]

@spwoodcock I have left some comments here - mostly prompts and reminders to discuss these. Everything else looks good. This is a significant quality upgrade to the Backend infrastructure. Thank you very much for taking the time and effort to do this.

@Aadesh-Baral Let's discuss this once we meet next week. Please hold merging even after current merge conflicts have been resolved. cc: @dakotabenjamin @ramyaragupathy

[eternaltyro]

I understand MAINTAINER label is deprecated and we can simply use LABELS for it. But what's the reason for using hotosm.org.maintainer as key? Docker suggests tld.domain.sub.maintainer but I'm not sure if it is standard. If it is, we can switch this to org.hotosm.maintainer.

Documentation about this is sparse though.

[spwoodcock]

I don't think there are any standards around label keys as far as I know, we can definitely change it 👍

[spwoodcock]

I'll try to find time to revise this PR soon (particularly the dockerfile). There are a few refinements I could include, plus Python 3.10.

[eternaltyro]

I'll try to find time to revise this PR soon (particularly the dockerfile). There are a few refinements I could include, plus Python 3.10.

Thank you, @spwoodcock . Let me know once you do, and I'll give it another review.

[spwoodcock]

I should get around to updating this at the weekend - got a pretty full work week ahead 👍

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
12 Code Smells

No Coverage information
0.0% Duplication

[spwoodcock]

I just rebased with develop and added the requirements.txt file back in for now.

The file can be deleted after merging #5642 and updating the pyproject.toml.

Ready to merge 👍

[eternaltyro]

Merged. All good so far. Thanks @spwoodcock for your time and labour. It's much appreciated 🙏🏼

[eternaltyro]

PEP 582 has been rejected! - https://discuss.python.org/t/pep-582-python-local-packages-directory/963/430

We will continue to use venv for now until our move to containers is done.

[spwoodcock]

Yes it has, but that doesn't mean it's useless.
Here is a great summary: https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/

PDM defaults to venv based installs. PEP582 support is optional, but is a much better alternative IMO.

The rejection shouldn't affect this PR.
PDM is used to export requirements for the dockerfile and to install via venv for running locally.

[eternaltyro]

Yes it has, but that doesn't mean it's useless. Here is a great summary: https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/

PDM defaults to venv based installs. PEP582 support is optional, but is a much better alternative IMO.

I agree. But we have to play nice with standards and conventions that have buy in from the community also.

The rejection shouldn't affect this PR. PDM is used to export requirements for the dockerfile and to install via venv for running locally.

Agreed. I am only concerned about PEP582; the rest of the features PDM brings are still useful. Going forward, we will use venv, preferably managed by PDM and conforming to PDM defaults. All this is moot anyway with containers taking over.