Proposal: Use pip-tools for dependency management

[aiven-anton]

Manually pinning exact versions in requirements.* files is tedious, using pip-compile from pip-tools we can automate the process of updating dependencies. We'll move the current requirements files to .in equivalents and generate the .txt files from those.

This also allows starting to use package hashes (with --generate-hashes).

---

An issue tangential enough to not deserve a separate issue is we should use Github's archive endpoints instead of git URLs for non-release dependencies. This has two advantages:

• Works with above mentioned hashing.
• Removes dependency on git for installation (this is a win for containerized environments).

---

I think it's also worth discussing the role of setup.py. Right now we have duplicated dependency specifications which makes it unclear which is authoritative. This dependency in setup.py doesn't exist on PyPI. Also, karapace doesn't exist on PyPI (but the name has been reserved).

Since karapace is mainly an application, do we really have reason to use setuptools? I'm guessing that the historical reason is to make use of entry_points. To me it seems like the installation related problems is worth looking for other ways to set those up than setuptools.

[aiven-anton]

Introducing --generate-hashes + --require-hashes has a dependency on this PR: #580.

[aiven-anton]

Note: we should also change to use -c requirements.txt instead of -r requirements.txt in requirements-dev.in when enabling hashing.