Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent hooks firing during using plugin cli #220

Open
DmitriyLamzin opened this issue Feb 17, 2023 · 2 comments
Open

Prevent hooks firing during using plugin cli #220

DmitriyLamzin opened this issue Feb 17, 2023 · 2 comments

Comments

@DmitriyLamzin
Copy link
Contributor

DmitriyLamzin commented Feb 17, 2023

I believe the plugin should prevent firing pipeline hooks during using its cli, as it creates unpredictable issues.

Example with Mlflow plugin:
We use kedro-mlflow plugin
We have in the project two envs: local and remote.
Remote env contains catalogs proper paths to the data in cloud storage, and the config for mlflow on the cloud and kubeflow config.
The developer/cicd doesn't have access from the local machine to the remote mlflow API
When the developer/cicd tries to compile a pipeline or upload it using --env remote following happens:

The instance of ContextHelper is created:

  1. Kedro Session is created
  2. Kedro context is initialized
  3. after_context_created hook is triggered
  4. catalog instance is retrieved
  5. after_catalog_created hook is triggered
  6. Mlflow plugin has after_catalog_created hook which tries to access to mlflow API
  7. Developer/cicd does not have access to mlflow API on remote env
  8. compiling/uploading fails

This behavior is not needed and not expected. The kubeflow plugin only needs to get catalog and pipeline parameters and proper KFP configs to create proper KFP.

@DmitriyLamzin
Copy link
Contributor Author

If this proposal looks fine I'd like to assign this to myself

@marrrcin
Copy link
Contributor

@szczeles / @em-pe what do you think? We've been actually relying on the fact that the hooks are invoked in some projects. Maybe you @DmitriyLamzin could add this as a flag in the CLI on the plugin level, like: kedro kubeflow --disable-hooks compile etc.?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants