Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retrying release for an already released table #246

Open
alok87 opened this issue Jul 14, 2021 · 2 comments
Open

Retrying release for an already released table #246

alok87 opened this issue Jul 14, 2021 · 2 comments
Labels
bug Something isn't working p3 not urgent, intermittent issue

Comments

@alok87
Copy link
Contributor

alok87 commented Jul 14, 2021

Operator released a table, then the async status update did not happen somehow, and since status did not get updated in the next reconcile it found the topic is not released and tried to release it again and got stuck in this loop ever since.

Need to make sure the status update happen, then only consider the release as valid!
@alok87 alok87 added the bug Something isn't working label Jul 14, 2021
@alok87
Copy link
Contributor Author

alok87 commented Jul 14, 2021 •
edited
Loading

Guessing: The issue is happening because the release query to rename the table was stuck due to some reason, and the reconciler process got stuck waiting for it, when the reconciliation timed out, it went to try again, but the query has done its job in Redshift and the status could not get updated for it as the reconciler timedout!

@alok87
Copy link
Contributor Author

alok87 commented Jul 14, 2021 •
edited
Loading

Found: status update was not happening after multiple release tries for a table, this could be related to status API not giving 200 for a particular payload or something. Need to tie the release success with status getting updated success!

Reducing the default of MaxTopicRelease from 50 to 5 and marking the issue as P3.

@alok87 alok87 added the p3 not urgent, intermittent issue label Jul 14, 2021
alok87 added a commit that referenced this issue Jul 14, 2021
@alok87
Do not release many tables in one iteration
379aeda 
Status update calls fail when many are updated together
#246
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working p3 not urgent, intermittent issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@alok87