Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Platform][2.4] Backup deletion script failure should not cause retries #7988

Closed
sb-yb opened this issue Apr 9, 2021 · 2 comments
Closed

[Platform][2.4] Backup deletion script failure should not cause retries #7988

sb-yb opened this issue Apr 9, 2021 · 2 comments
Assignees
@sb-yb
Labels
area/platform Yugabyte Platform priority/high High Priority
Milestone
2.7.x

Comments

@sb-yb
Copy link
Contributor

sb-yb commented Apr 9, 2021

When backup reference in yw db is so old that node does not exist anymore. The backup script fails. We should ignore such failure otherwise it keeps retrying and clogs the task framework.
Ideally in future we should have exponential backoff + limit on max retries.
@sb-yb sb-yb self-assigned this Apr 9, 2021
@sb-yb
Copy link
Contributor Author

sb-yb commented Apr 9, 2021

Because #7780 is now fixed; this bug has become critical. We should expect lot of old backup deletion attempts and many of them will fails since they are actually already deleted manually as a workaround.

@sb-yb sb-yb added the priority/high High Priority label Apr 9, 2021
@sb-yb
Copy link
Contributor Author

sb-yb commented Apr 9, 2021

FYI @CharlotteRose

WesleyW added a commit that referenced this issue Apr 9, 2021
@WesleyW
[#7988][Platform]: Fix yb_backup.py exit handlers
978205b 
Summary:
The `__file__` variable is not set when the `atexit` handlers are called. Instead, store the value
beforehand and save it.

Note: even if the exit handlers error, the return code will not change.

Test Plan: yb_backup

Reviewers: arnav, daniel, sb-yb

Reviewed By: sb-yb

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D11206
sb-yb added a commit that referenced this issue Apr 10, 2021
@sb-yb
[#7988][Platform] Make delete backup task best effort.
9d00b8b 
Summary:
- It will always transition backup to deleted state - no matter what and ignore all the errors.
- Removed ton of redundant and unnecessary code.

Test Plan: WIP

Reviewers: wesley

Reviewed By: wesley

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D11207
@streddy-yb streddy-yb added this to the 2.7.x milestone Apr 12, 2021
WesleyW added a commit that referenced this issue Apr 12, 2021
@WesleyW
[Backport 2.4][#7988][Platform]: Fix yb_backup.py exit handlers
606b2cd 
Summary:
The `__file__` variable is not set when the `atexit` handlers are called. Instead, store the value
beforehand and save it.

Note: even if the exit handlers error, the return code will not change.

Test Plan: yb_backup

Reviewers: arnav, daniel, sb-yb

Reviewed By: sb-yb

Subscribers: yugaware, jenkins-bot

Differential Revision: https://phabricator.dev.yugabyte.com/D11209
sb-yb added a commit that referenced this issue Apr 12, 2021
@sb-yb
[#7988][2.4][Platform] Make delete backup task best effort.
8d4f173 
Summary:
Backport for ignoring backup deletion failures.
 Original commit: D11207 / 9d00b8b

Test Plan: Unit test

Reviewers: spotachev, wesley

Reviewed By: wesley

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D11224
WesleyW added a commit that referenced this issue May 3, 2021
@WesleyW
[Backport 2.2][#7988][Platform]: Fix yb_backup.py exit handlers
ca88e8e 
Summary:
The `__file__` variable is not set when the `atexit` handlers are called. Instead, store the value
beforehand and save it.

Backports D11206

Test Plan: Check yb_backup

Reviewers: arnav, daniel, sb-yb, jvigil

Reviewed By: jvigil

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D11445
YintongMa pushed a commit to YintongMa/yugabyte-db that referenced this issue May 26, 2021
@WesleyW @YintongMa
[yugabyte#7988][Platform]: Fix yb_backup.py exit handlers
3fcdce8 
Summary:
The `__file__` variable is not set when the `atexit` handlers are called. Instead, store the value
beforehand and save it.

Note: even if the exit handlers error, the return code will not change.

Test Plan: yb_backup

Reviewers: arnav, daniel, sb-yb

Reviewed By: sb-yb

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D11206
YintongMa pushed a commit to YintongMa/yugabyte-db that referenced this issue May 26, 2021
@sb-yb @YintongMa
[yugabyte#7988][Platform] Make delete backup task best effort.
2e33f05 
Summary:
- It will always transition backup to deleted state - no matter what and ignore all the errors.
- Removed ton of redundant and unnecessary code.

Test Plan: WIP

Reviewers: wesley

Reviewed By: wesley

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D11207
@ryan-ally ryan-ally mentioned this issue Nov 15, 2023
Open
@nyndyny nyndyny mentioned this issue Nov 15, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sb-yb sb-yb

Labels
area/platform Yugabyte Platform priority/high High Priority
Projects
None yet
Milestone
2.7.x
Development

No branches or pull requests
2 participants
@streddy-yb @sb-yb