Do development tools belong in requirements.in?

[WaylonWalker]

I am of the opinion that "no they do not, they belong in a requriements_dev.in file". Tools like black, flake8, jupyter, and
ipython are not needed to run pipelines and do not belong in the core requirements file.

Is the argument for simplicity? I can totally see a valid argument that many people using the standard template are new to packaging python libraries and they would prefer a simple workflow in that regard.

What are your thoughts? I am curious to hear why they are in the default template, I do not generally see packages like this in requirements files in other templates/projects.

[mzjp2]

I believe the reason for that is that users expect that once they run kedro install, all future kedro <x> commands run without needing to install more requirements (especially on a new project).

Since kedro lint and kedro test exist, that execute black and isort and other development tools, I can see why they belong inside Kedro's requirements.txt.

This is because Kedro is slightly meta in that it brings an opiniated set of tools to run on your project. If kedro exclusively used black to ensure its own codebase was formatted, but there was 0 mention of import black or subprocess.run("black") inside its codebase, I would 100% agree with you that it should remain in dev_requirements.txt.

It's worth noting that those tools are not inside Kedro's own requirements.txt but are inside the generated project's one. Are you advocating for splitting those out into the project's dev_requirements.txt and then changing kedro install to install from files? I could somewhat see the argument there, kedro install --dev and kedro install could then be a thing? Interesting discussion!

[WaylonWalker]

I completely understand that the default template is opinionated and I may not be the target audience for it. I trust that you guys are trying to meet the target user somewhere in between "too much tooling for me" and doing everything perfectly correct.

As I start down the path of thinking what the ideal is, I think I end up with something that comes very close to pre-commit. It auto installs any tool that is missing into its own sandbox environment that will not clash versions with the libraries in your project. Ideally it would be nice to just replace kedro's internal lint commands with subprocess.run("pre-commit", "run", "--all"). Since you already have split format and lint, it may require two pre-commit configs. I still think you run into a situation where pre-commit itself might not be installed, so you would have to catch that exception. At least with pre-commit you are safe to install the latest version and get the same results across multiple machines, unlike installing an unpinned version of flake8 or black

I think kedro install --dev would be a fantastic solution. Maybe to have fewer changes it would install everything by default, and only required packages on kedro install --prod. Then there is less change for the end-user, and folks setting up ci can change over to add --prod.

[astrojuanlu]

Do development tools belong in requirements.in?

I agree they do not :) Addressed for starters in kedro-org/kedro-starters#238