-
Notifications
You must be signed in to change notification settings - Fork 901
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spike: How can we make upgrading Kedro projects easier? #3960
Comments
Adding some context from #3959 (we wrote the same issue at the same time 😄) DescriptionOur users find it difficult to upgrade Kedro in their projects. Just from a couple of recent user interviews: User 1:
User 2:
There are also very clear signs that old Kedro versions tend to live on for a very long time: People like @inigohidalgo have been reporting their long journey to upgrade from old Kedro versions in our public Slack. And finally, we also have lots of internal evidence as well that big projects get stuck on old Kedro versions. Is this a problem?One could argue: "it it works, don't touch it". So the fact that Kedro is pinned to a specific version is not necessarily a bad thing. However, with resource constrained teams maintaining many projects, each of them with slightly different versions of Kedro, this can become a mess to maintain. For those teams who would wish to have a uniform Kedro versioning, we should provide a more clean upgrade path. What has been doneFor 0.19 we went ahead and added a detailed migration guide https://docs.kedro.org/en/latest/resources/migration.html#migrate-an-existing-project-that-uses-kedro-0-18-to-use-0-19 Where to go from hereHowever, it seems to not yet be enough. What else can we do to make these migrations easier? |
Originally posted by @datajoely in #3959 (comment) |
Maybe we could draw inspiration from database migration tools as well, like |
Adding another option which operate at a lower level: https://libcst.readthedocs.io/en/latest/codemods_tutorial.html, or pyupgrade |
I agree with @datajoely's view: our upgrade path was particularly complex because we started with kedro very early on in its life, and a lot of custom components were built through a MCK engagement. Those custom components relied on internal kedro behavior which was later deprecated ( I like things like cruft or copier for keeping various of our own projects in sync but I don't see how you can reasonably expect to use those in aiding upgrades. Anyways I would expect that as you near 1.0, this should become less of an issue, with breaking changes being less frequent, and also less breaking in general, with simpler and clearer upgrades. |
I think there's at least 3 sources of friction when it comes to upgrades:
For 1, I think even though we're not super strict, if we do see breakage we fix it. There might be breaking things we're not seeing. For 2, we're already adding deprecation warnings, but arguably we could do more. We just started writing migration guides in 0.18 -> 0.19. We could also think of "linting" tools that detect deprecated functionality. It's probably too much work though. For 3 though we're not doing anything of substance beyond telling people to recreate their project structure and it's where https://github.com/copier-org/copier could definitely help. Anything that I'm missing? |
Only the fact that advanced users will likely patch things like the context, sessions and custom CLIs which make it near impossible to upgrade. I still expect the session to improve through the deployment workstream, so it's hard to say if we're actually done there. |
https://fediverse.zachleat.com/@zachleat/112689087055371089
|
Description
Users have frequently mentioned that upgrading between major versions in Kedro is difficult. The main difficulty seems to be with upgrading the project structure when template changes happened in the breaking releases.
Perhaps a tool like cruft or copier could help here, or even a python script that users can apply to go from e.g.
0.17.0
->0.19.0
and0.18.0
->0.19.0
The task here is to do a spike investigating the different options and deciding on the best way forward.
Context
I didn't fully appreciate the issue until we recently talked to a team that explained they have around 40 projects with Kedro, but not all have an active maintenance team. This results in them having different Kedro versions for the projects. The biggest struggle is updating the template, because that is a very manual job an requires someone understanding what the changes are between breaking Kedro versions.
Possible Solutions
Extra Info
Earlier mentions to upgrading issues and/or curft/copier:
kedro new
without creating a new directory #681The text was updated successfully, but these errors were encountered: