Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

log-backup: log backup task may silently broken by a failed initialization #16056

Closed
YuJuncen opened this issue Nov 23, 2023 · 1 comment · Fixed by #16071
Closed

log-backup: log backup task may silently broken by a failed initialization #16056

YuJuncen opened this issue Nov 23, 2023 · 1 comment · Fixed by #16071
Labels
affects-6.5 affects-7.1 affects-7.5 component/backup-restore Component: backup, import, external_storage severity/major type/bug The issue is confirmed as a bug.

Comments

@YuJuncen
Copy link
Contributor

Bug Report

What version of TiKV are you using?

v6.5.5, but this probably happens on all current versions.

What operating system and CPU are you using?

a general x64 cpu.

Steps to reproduce

  1. Begin a task, with one store has a temporary connective issue to PD.
  2. Wait.

What did you expect?

The log backup task should either:

  • Fail.
  • Run normally.

What did happened?

The log backup lag increases to infinity. Until we evict all leaders in the store failed to connect to PD during loading the task.
It reports the log (thankfully...) and did nothing more.

[2023/11/07 19:29:51.821 +00:00] [WARN] [errors.rs:171] [""backup stream meet error""] [verbose_err="Etcd(...)"] [err="..."] [context=""failed to register backup stream task log-scheduled-backup-manager to router: ranges not found"
@YuJuncen YuJuncen added the type/bug The issue is confirmed as a bug. label Nov 23, 2023
@BornChanger
Copy link
Contributor

/component backup-restore

@ti-chi-bot ti-chi-bot bot added the component/backup-restore Component: backup, import, external_storage label Nov 24, 2023
@YuJuncen YuJuncen mentioned this issue Nov 27, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Nov 30, 2023
@YuJuncen @ti-chi-bot
log-backup: make initialize failure fatal error, release memory while…
fd98944 
… task stopped. (#16071)

close #16056, close #16070

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 30, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 30, 2023
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16071
9b34274 
close tikv#16056, close tikv#16070

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
This was referenced Nov 30, 2023
Merged
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Nov 30, 2023
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16071
c870c15 
close tikv#16056, close tikv#16070

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this issue Dec 8, 2023
@ti-chi-bot @YuJuncen @iosmanthus
log-backup: make initialize failure fatal error, release memory while…
8e5ea99 
… task stopped. (#16071) (#16108)

close #16056, close #16070

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: hillium <yujuncen@pingcap.com>
Co-authored-by: iosmanthus <dengliming@pingcap.com>
ti-chi-bot bot added a commit that referenced this issue Dec 21, 2023
@ti-chi-bot @YuJuncen @ti-chi-bot
log-backup: make initialize failure fatal error, release memory while…
0998d1f 
… task stopped. (#16071) (#16110)

close #16056, close #16070

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: hillium <yujuncen@pingcap.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@3pointer 3pointer mentioned this issue Jan 30, 2024
Closed
ti-chi-bot bot added a commit that referenced this issue Jan 31, 2024
@ti-chi-bot @YuJuncen @ti-chi-bot
log-backup: make initialize failure fatal error, release memory while…
2dba329 
… task stopped. (#16071) (#16109)

close #16056, close #16070

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: hillium <yujuncen@pingcap.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 component/backup-restore Component: backup, import, external_storage severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

log-backup: make initialize failure fatal error, release memory while task stopped. YuJuncen/tikv
2 participants
@YuJuncen @BornChanger