Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

one tikv has no leader and slow down br restore speed #50566

Closed
Tracked by #50701
fubinzh opened this issue Jan 18, 2024 · 2 comments · Fixed by #50776
Closed
Tracked by #50701

one tikv has no leader and slow down br restore speed #50566

fubinzh opened this issue Jan 18, 2024 · 2 comments · Fixed by #50776
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Jan 18, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. use br to restore a full backup of 30TB 500k table cluster to a 20 TiKV nodes new cluster.
/br  restore  full "-s" "gcs://aaa/bbb-500k-table-30tb-backup-with-stats-3" "-u" "http://tc-pd:2379" "--check-requirements=false" "--ddl-batch-size=128" "--tikv-max-restore-concurrency=128" "--granularity=coarse-grained" "--send-credentials-to-tikv=true" "--checksum=false"

2. What did you expect to see? (Required)

all tikv should have restore tasks.

3. What did you see instead (Required)

One TiKV has no leader, and slow down the br restore speed.

image

4. What is your TiDB version? (Required)

[root@br-0 tmp]# /br -V
Release Version: v7.6.0-pr50518
Git Commit Hash: 4888198
Git Branch: heads/refs/tags/v7.6.0-pr50518
Go Version: go1.21.5
UTC Build Time: 2024-01-18 02:50:49
Race Enabled: false

@fubinzh fubinzh added the type/bug The issue is confirmed as a bug. label Jan 18, 2024
@fubinzh
Copy link
Author

fubinzh commented Jan 18, 2024

/component br
/severity major

@3pointer
Copy link
Contributor

This is because evict-slow-store triggeered by PD. we can fix this issue by remove this scheduler during restore.

@Leavrth Leavrth added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Feb 4, 2024
ti-chi-bot bot pushed a commit that referenced this issue Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants