Explore pip-tools for dependency management

[caleb-johnson]

Fixes #901

I opened this PR to play around with pip-tools and continue the discussion in #901.

With this tool, instead of only specifying our top-most dependencies, we allow pip-tools to create our requirements.txt from a requirements.in. The requirements.in is very similar to our current requirements.txt, but it generally has its version restrictions relaxed to the maximum extent. This allows pip-tools to do much of the dependency management for us. The requirements.txt generated includes a fully resolved dependency tree, reducing the chances of conflicts during installation.

• Convert requirements.txt files to .in files
• Create .txt files from the .in files using pip-compile from pip-tools
• Create a CI job for updating the requirements.txt files
  • Weekly during a time when packages aren't being actively released (if a package is released between creation of our requirements files, there could be discrepancies)
  • Any time a requirements.in file changes
  • Automatically creates a PR
• Update documentation to reflect new way to install

[caleb-johnson]

I was getting errors when this was unpinned. One requirements file would install ray 2.3 and one would install ray 2.6, which led to conflicts between the dev and production envs.

[Tansito]

Thanks @caleb-johnson ! I will start taking a look into this too. Thank you to start with! ❤️

[psschwei]

I'm assuming the answer to this is yes, but we should confirm that dependabot will handle updates on requirements.in files

[Tansito]

I'm assuming the answer to this is yes, but we should confirm that dependabot will handle updates on requirements.in files

Well, I would not assume it to be honest. Definitely one of the things that we will need to see 😂

[caleb-johnson]

Kind of annoying to have to update these files (migrations 0011, 0012) in this PR, but the black version being resolved for the gateway wants this fixed :)

[caleb-johnson]

This may be totally redundant. Need to double check

[IceKhan13]

@caleb-johnson @Tansito any updates on this? should we close it?

[caleb-johnson]

@caleb-johnson @Tansito any updates on this? should we close it?

I think it is OK to close for now, and if @Tansito or anyone decides they would like to migrate to pip-tool, this PR can be revived or used as reference.