Added requeue in five seconds to processLocalDatasetCOS if !AuthProvided #318
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Due to my pipeline to deploy application generate resources on Kubernetes in once pipeline triggered, such as External secrets, Dataset, Deployment. Dataset pull secrets from the secret store while it is not ready-to-use (too fast). Then, I have noticed some time Datashim didn't work as It should be (PVC is not created).
I went dive to the log of dataset-controller and found this
No useable secret provided for authentication
before looping ofPVC is not created yet
, andReconciling Dataset
as I show below.2023-11-08 11:21:59 | 1.6994173194031293e+09 ERROR controller_datasetinternal Failed to initialise {"Dataset.Namespace": "xxx-dev", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx", "error": "No useable secret provided for authentication"}
2023-11-08 11:22:21 | 1.6994173412709296e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx, "namespace": "xxx-dev", "Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-08 11:22:21 | 1.699417341270836e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx-dev", "Request.Namespace": "xxx-dev", "Request.Name": "xxx"}
2023-11-08 11:22:16 | 1.6994173362702456e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx", "Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-08 11:22:16 | 1.6994173362701066e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx-dev", "Request.Namespace": " xxx", "Request.Name": "xxx"}
.
.
.
[it seems deadlock because controller_datasetinternal]
Adding requeue to the
processLocalDatasetCOS
incontroller_datasetinternal
solved my problem. the log after fixing is below.1.699502566537274e+09 ERROR controller_datasetinternal Failed to initialise {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx", "error": "No useable secret provided for authentication"}
2023-11-09 15:08:01 | 1.6995172812466173e+09 INFO controller_datasetinternal All good and we shouldnt reconcile! {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset"}
2023-11-09 15:08:01 | 1.699517281239933e+09 INFO controller_datasetinternal Creating new secrets {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Secret.Namespace": "xxx", "Secret.Name": "x"}xx
2023-11-09 15:08:01 | 1.6995172812319953e+09 INFO controller_datasetinternal Creating new pvc {"Dataset.Namespace": "xxx", "Dataset.Name": "x", "Method": "processLocalDataset", "PxxVC.Namespace": "xxx", "PVC.Name": "x"}xx
2023-11-09 15:08:01 | 1.699517281223015e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "x", "Request.Namespaxxce": "xxx", "Request.Name": "xxx"}
2023-11-09 15:08:01 | 1.69951728122279e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxx", "Request.Namespace": "x", "Request.Naxxme": "xxx"}
2023-11-09 15:08:01 | 1.6995172812229505e+09 INFO controller_datasetinternal Creating new configMap {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "createConfigMapforObjectStorage", "configMap.namespace": "xxx", "configMap.Name": "xxx"}
2023-11-09 15:08:01 | 1.699517281222926e+09 INFO controller_datasetinternal Authentication info has been successfully retrieved {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx"}
2023-11-09 15:08:01 | 1.6995172812229085e+09 INFO controller_datasetinternal Error: secret namespace is same as dataset namespace, allowed {"Dataset.Namespace": "xxx", "Dataset.Name": "xxx", "Method": "processLocalDataset", "Dataset.Name": "xxx"}
2023-11-09 15:08:01 | 1.699517281222897e+09 INFO controller_datasetinternal All good, proceed {"Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-09 15:08:01 | 1.6995172812227738e+09 INFO controller_datasetinternal Reconciling DatasetInternal {"Request.Namespace": "xxxx", "Request.Name": "xxx"}
2023-11-09 15:07:56 | 1.6995172762218883e+09 INFO controller.dataset PVC is not created yet {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxxx", "namespace": "xxx", "Request.Namespace": "xxx", "Request.Name": "xxx"}
2023-11-09 15:07:56 | 1.6995172762218397e+09 INFO controller.dataset Reconciling Dataset {"reconciler group": "com.ie.ibm.hpsys", "reconciler kind": "Dataset", "name": "xxx", "namespace": "xxxx, "Request.Namespace": "xxxx, "Request.Name": "xxx"}
this shows that deadlock doesn't exist anymore.