MLFlow credentials will not update on existing or new Kubeflow Notebooks #30
Comments
|
I managed to reproduce the issue with one of our notebook tests by following the steps outlined in this issue. |
|
Minimal instructions to reporoduce.
|
|
Looks like key is not updated in user's namespace secret. Should it be?
|
|
After the updating key, the relation |
|
Candidate for fix in 1.8 |
|
This is expected behavior let me explain: Resource dispatcher is responsible for distributing manifests across the namespaces with specific label. It is also responsible for updating those manifests in case the relation with resource dispatcher changes. That is what is happening. If you mlflow kubeflow plus resource dispatcher and you change the password for minio in mlflow bundle. resource dispatcher will update manifests in target namespaces. Its demonstrated also in Ivan's comment as he is pointing that the relation changes. The error you are describing is just related how mounted secrets in Kubernetes work. Because we mount secret as environment variable to user's notebook, the environment will not automatically change on secret change (the secret in namespace changed but the notebook environment not). In order to use new environment with new value you need to restart pod. If we want to automatically change the environment secret we would need to somehow monitor for secret change (e.g. with vault sidecar injector https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar) for now its out of scope. I will close this issue for now . |
|
Reopennig looks like locally I had metacontroller v3 which is not in 1.7 bundle for kubeflow (there is v2) . We downgraded the metacontroller to v2 because kubeflow-profile-controller had issues with v3. We need v3 for resource dispatcher to correctly update the manifests in user namespaces. |
|
Fixed by updating kfp-profile-controller to use Decorator controller canonical/kfp-operators#344 and then updating metacontroller to v3 canonical/metacontroller-operator#94 |
Versions
Kubernetes platform: Microk8s (v1.24.16)
Juju Agents: 2.9.43
Juju CLI: 2.9.44-ubuntu-amd64
resource-dispatcher: edge (rev 78)
Kubeflow: 1.7/stable
MLFlow: 2.1/edge
Rest of charm versions visible on
juju statusReproduce
The integration between mlflow and kubeflow notebooks was working for me until I decided to update mlflow-minio credentials
juju config mlflow-minio secret-key=miniominio juju run-action mlflow-server/0 get-minio-credentials --wait unit-mlflow-server-0: UnitId: mlflow-server/0 id: "4" results: access-key: minio secret-access-key: miniominio status: completed timing: completed: 2023-08-07 05:00:50 +0000 UTC enqueued: 2023-08-07 05:00:47 +0000 UTC started: 2023-08-07 05:00:48 +0000 UTCLogs
Now testing the S3 credentials on existing, as well as on new notebooks I would still be getting the old secre-key
I tried:
A. Removing and adding secret relation, but still the same result
B. Removing the application and redeploying, but still the same result
I could not find any workaround to it so far
The only thing I could do was rollback the secret-key change
Which, I am not sure if it is intended but after re-setting to no value I got the same secret-key I had before changing its value
juju config mlflow-minio --reset secret-key juju run-action mlflow-server/0 get-minio-credentials --wait unit-mlflow-server-0: UnitId: mlflow-server/0 id: "6" results: access-key: minio secret-access-key: 9TJTFVPTXBZ8QB1YZBP0JUYOIID6XC status: completed timing: completed: 2023-08-07 05:21:08 +0000 UTC enqueued: 2023-08-07 05:21:02 +0000 UTC started: 2023-08-07 05:21:07 +0000 UTCThe text was updated successfully, but these errors were encountered: