Plan to improve ci/cd + automated release setup of kedro-plugins

[merelcht]

Description

Following up on kedro-org/kedro-plugins#4

Currently kedro-plugins has one top level Makefile, each plugin has it's own .pre-commit-config.yaml and pyproject.toml. Look at whether it would be better for these configuration files to be unified or for each plugin to have it's own files.

@AntonyMilneQB Raised a good question about whether the pre-commit hooks actually still work without the file being at the root of the repo.

For this issue come up with a plan to improve the setup:

• Decide what parts exactly need to be improved
• Come up with a proposal of breaking this down and how to implement
• Discuss in Technical Design when needed

[deepyaman]

@AntonyMilneQB Raised a good question about whether the pre-commit hooks actually still work without the file being at the root of the repo.

Having .pre-commit-config.yaml at the root of the repo is also recognizable by other tools, such as pre-commit.ci (which we could consider using).

There's no great answer for this, but I've seen pre-commit/pre-commit#466 (comment). Keep in mind that the pre-commit author also says that it's not really designed for monorepos (although it can be made to work).

@noklam raised a good point:

The main drawback is if you are working on the smaller repositories like kedro-airflow , you still have to run the linting for the full datasets repo which takes more time

Maybe it makes sense to just use something like tox to manage the way you run tests in something like this?

[jmholzer]

In order to install the Kedro dependency correctly, we need to add pip install -r requirements.txt under install-test-requirements in the case that $plugin is equal to kedro-datasets in the kedro-plugins Makefile.

[deepyaman]

I would be open to putting together a PoC with tox, if you (others) are interested @jmholzer.

[deepyaman]

Preface

The below breakdown of challenges and solutions aims to:

1. provide tactical solutions to existing challenges (primary motivation)
2. use kedro-plugins as a lightweight testbed for tooling for better practices/other improvements, rather than on the more complex main kedro repo (secondary motivation)

Challenges

• pre-commit configuration is maintained per-plugin
  • Not ideal because part of the idea of a monorepo is that configuration for consistent styling, etc. can be shared.
  • pre-commit doesn't have first-class monorepo support (e.g. support hierarchical config); see the author's guidance on monorepo support
• Makefile doesn't support local testing of individual plugins in isolated environments
  • Need to create/manage an environment per plugin (not documented, on your own)
  • Using kedro_plugins conda environment for every plugin in CI is not reasonably transferrable to local environment
• It's not clear which changes affect which plugins
  • Denoted inconsistently using tags like [kedro-docker] on PR/issues (nonstandard solution)

Proposal

• Choose tool to allow testing a plugin across build environments using a single command kedro-plugins#512 Depending on the outcome, also do the below:
  • [P0] Use tox per-plugin to allow testing a plugin across build environments using a single command (tox) -- rough estimate: 2 days, risks: conda-installed packages may need to be handled differently, accessing pre-commit config above plugin root is less clean
    • Recommend not using tox4 for now until more confident about broader plugin support (e.g. tox-conda)
  • [P0] Enforce tox-based checks in CI
    • This is fairly easy to set up in GitHub Actions using https://pypi.org/project/tox-gh-actions/, which could be a good use case for proving out Migrate kedro-plugins from CircleCI to Github Actions #2153 -- rough estimate: 1 day
• [P0] Move pre-commit config to the top level of kedro-plugins #2337 -- rough estimate: 0.5 days
• [P0] Switch to using pyproject.toml for setuptools config on kedro-plugins #2334
• [P0] Add nightly build or trigger on Kedro core change #2335
• [P1] Enforce use of Conventional Commits with per-plugin scopes with commitlint -- rough estimate: trivial, <0.5 days
• [P1] Use pre-commit.ci to enforce pre-commit checks
• [P2] Set up automated release note generation based on Conventional Commits -- rough estimate: 1 day
• [P2] [SPIKE] Automate full release process based on commits using Conventional Changelog ecosystem, python-semantic-release, or similar
• [P2] [SPIKE] Explore moving supported lint checks to ruff, which does support hierarchical config -- rough estimate: 1-2 days, including set up with pre-commit
• [P2] [SPIKE] Explore use of more fully-functioned build tool (e.g. flit, hatch, poetry, pdm) to cover more steps (e.g. requirements management, release, etc.) through a single tool -- rough estimate: 1-2 days for flit, not sure for the rest

[noklam]

Agree with the challenges but I would order them in reverse order.

• It's not clear which changes affect which plugins
  • Denoted inconsistently using tags like [kedro-docker] on PR/issues (nonstandard solution)

This creates overhead for maintenance, particularly since the changes are quite sparse (not everyone is familiar with all the plugins, usually one person touch a small part of it). It's quite important to isolate the changes source of error.

• Makefile doesn't support local testing of individual plugins in isolated environments
  • Need to create/manage an environment per plugin (not documented, on your own)
  • Using kedro_plugins conda environment for every plugin in CI is not reasonably transferrable to local environment

As mentioned, it's not documented and I have a separate conda environment for development. This probably affects us as developers more as most contributors will only touch one of the plugin? Nevertheless, it's great to have good practices

• pre-commit configuration is maintained per-plugin
  • Not ideal because part of the idea of a monorepo is that configuration for consistent styling, etc. can be shared.
  • pre-commit doesn't have first-class monorepo support (e.g. support hierarchical config); see the author's guidance on monorepo support

I agree this could be an improvement, but I am not sure whether this is the top priority. Most of the configurations are only set once from the beginning and we rarely touch the configuration. It definitely violates the DRY principle, but I don't find the hierarchical config helps a lot.

I would add one more challenge:

• The CI is not run regularly - for kedro we have frequent changes so any dependencies change will be found out pretty quickly. For plugins, there could be a dependency change but we only have new commits after a few months. In this case, it is unclear whether the new changes break the CI or something in between happens, often it takes a long time to isolate the error.

[merelcht]

As discussed in technical design on 15/2:

1. This issue becomes a milestone: https://github.com/kedro-org/kedro/milestone/30
2. All P0 points are converted into issues
3. When all P0 issues have been completed, we'll revisited what more needs to be done. Most likely in technical design sessions.

[deepyaman]

One additional note to keep in mind that I just remembered; we need to address the behavior when multiple plugins are modified (or just make sure this doesn't happen in the future approach):

[astrojuanlu]

Potentially relevant for the automated release https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/

[astrojuanlu]

I think most of #2333 (comment) has already been done, except for kedro-org/kedro-plugins#512

[merelcht]

Closing this because the CI/CD of kedro-plugins has been hugely improved already. I created a new milestone for testing the plugins individually: https://github.com/kedro-org/kedro/milestone/60