DBInstance does not retry Create for invalid spec values

[RedbackThomson]

Describe the bug
If a password is not specified when first creating an RDS DBInstance CR, the Create* call fails and sets the CR into a terminal condition. If you update the password field after this, the call is not re-attempted and the instance is forever stuck in a terminal state.

Steps to reproduce

1. Create a DBInstance with an invalid masterUserPassword value
2. Watch for the DBInstance to become terminal
3. Update the masterUserPassword value to be valid
4. See that the DBInstance never changes

Expected outcome
The DBInstance should be created with the new masterUserPassword if the initial Create failed.

Environment

• Kubernetes version
• Using EKS (yes/no), if so version?
• AWS service targeted (S3, RDS, etc.) RDS

[RedbackThomson]

From our Slack conversation regarding this issue:

When we generate the RDS controller, we set which error responses should be classified as terminal. I can see that InvalidParameterValue is one of those. It might be worth it for us to remove that one from the list, since that blocks your use case.

[ack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[vijtrip2]

/lifecycle frozen

[Samze]

I also bumped into this. This is especially problematic when applying the DBInstance & Secret at the same time, it introduces a race condition. If the Secret is created first the DBInstance will succeed, if the DBInstance is created first then it will fail and never reconcile. Ideally it would periodically re-reconcile on an exponential backoff.

[RedbackThomson]

@Samze I see that in our logic for retrieving fields from Secret, we will throw an error if the Secret is not found - https://github.com/aws-controllers-k8s/runtime/blob/main/pkg/runtime/reconciler.go#L121 . This should cause the reconciler to requeue (without any backoff) until it succeeds. Is this the scenario you've been facing? Or have you been supplying an empty or invalid value inside the Secret, which is what's being used to create the instance?

[Samze]

@RedbackThomson thanks for your response. I'm pointing DBInstance to a non-existing secret. Interesting, I'm not seeing any requeue and I see DBInstance get a " terminal condition" condition, which I assume means it won't retry.

This is what I'm doing:

1. Create DBInstance with reference to non-existing secret.
2. Check status, it shows.

  conditions:
  - message: kubernetes secret not found
    status: "True"
    type: ACK.Terminal
  - lastTransitionTime: "2022-06-03T18:25:10Z"
    message: Resource synced successfully
    reason: resource is in terminal condition
    status: "True"
    type: ACK.ResourceSynced

3. Apply secret
4. Wait for changes. Nothing changes

For reference I'm using 0.0.24 of the ACK RDS Operator.

Also didn't mean to hijack this thread, happy to make another issue if needed.

[RedbackThomson]

Oh yeah this should not be a terminal condition. We definitely want to retry when we see that. If you could please put this into a separate issue, this is no longer just an RDS bug - it's a bug with the general way we handle secrets (and errors). I see the problem in the code, though!

[Samze]

Thanks, created #1318

[brucegucode]

issue-1318 is closed, we can close this one too.
#1318

[brucegucode]

/close

[ack-bot]

@brucegucode: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jkatz]

/close

[ack-bot]

@jkatz: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.