Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

qps dorp to zero after inject pdleader io hang #6124

Closed
Lily2025 opened this issue Mar 9, 2023 · 4 comments · Fixed by #6182
Closed

qps dorp to zero after inject pdleader io hang #6124

Lily2025 opened this issue Mar 9, 2023 · 4 comments · Fixed by #6182
Assignees
@lhy1024
Labels
affects-7.0 severity/critical type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Lily2025 commented Mar 9, 2023

Bug Report

What did you do?

inject pdleader io hang
https://tcms.pingcap.net/dashboard/executions/plan/1655438
http://perf.pingcap.net:3000/d/WOw1JWmVk/ha-test-case-detail?orgId=1&refresh=10s&var-case_id=2820188
c8d6c931-8096-412e-90d3-3273eb313ce9
719777a4-a367-4d9f-92bb-d5526dec6c19

What did you expect to see?

QPS can recover within 2min

What did you see instead?

QPS drop to zero until fault recover
c8d6c931-8096-412e-90d3-3273eb313ce9
719777a4-a367-4d9f-92bb-d5526dec6c19

clinc : https://clinic.pingcap.com.cn/portal/#/orgs/31/clusters/7208067513104263723?from=1678303062&to=1678304612
logs: http://minio.pingcap.net:38888/buckets/test-infra-testground/browse/YXJjaGl2ZS9lbmRsZXNzLWhhLXRlc3QtdHBzLTE2NTU0MzgtMS02MDM=

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v6.7.0-alpha
Edition: Community
Git Commit Hash: 8cd4b16
Git Branch: heads/refs/tags/v6.7.0-alpha
UTC Build Time: 2023-03-08 11:32:54
2023-03-09T03:17:55.301+0800 INFO
@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Mar 9, 2023
@Lily2025
Copy link
Author

Lily2025 commented Mar 9, 2023

/type bug
/severity critical
/assign lhy1024

@lhy1024
Copy link
Contributor

lhy1024 commented Mar 9, 2023

it should be only affected the master branch after #6046, I will Investigate it

@lhy1024 lhy1024 mentioned this issue Mar 20, 2023
Merged
ti-chi-bot added a commit that referenced this issue Mar 20, 2023
@lhy1024 @ti-chi-bot
etcdutil: revert etcd client in pd legacy server (#6182)
1e49be5 
close #6124

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Mar 20, 2023
Merged
ti-chi-bot added a commit that referenced this issue Mar 21, 2023
@ti-chi-bot @lhy1024
etcdutil: revert etcd client in pd legacy server (#6182) (#6186)
37cc8de 
close #6124, ref #6182

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
CabinfeverB pushed a commit to CabinfeverB/pd that referenced this issue Mar 22, 2023
@ti-chi-bot @lhy1024 @CabinfeverB
etcdutil: revert etcd client in pd legacy server (tikv#6182) (tikv#6186)
b487635 
close tikv#6124, ref tikv#6182

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
CabinfeverB pushed a commit to ti-chi-bot/pd that referenced this issue Mar 22, 2023
@ti-chi-bot @lhy1024 @CabinfeverB
etcdutil: revert etcd client in pd legacy server (tikv#6182) (tikv#6186)
730bf49 
close tikv#6124, ref tikv#6182

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
CabinfeverB pushed a commit to ti-chi-bot/pd that referenced this issue Mar 22, 2023
@ti-chi-bot @lhy1024 @CabinfeverB
etcdutil: revert etcd client in pd legacy server (tikv#6182) (tikv#6186)
5a1c531 
close tikv#6124, ref tikv#6182

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>
@lhy1024 lhy1024 mentioned this issue Apr 25, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Apr 25, 2023
@lhy1024 @ti-chi-bot
etcdutil: revert etcd client without multi endpoint (#6374)
574461f 
ref #6124

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@rleungx
Copy link
Member

rleungx commented Jun 2, 2023

@lhy1024 Please add more detailed information about it when you close an issue. What is the root cause?

@lhy1024
Copy link
Contributor

lhy1024 commented Jun 2, 2023

@lhy1024 Please add more detailed information about it when you close an issue. What is the root cause?

Just revert this behavior and test it some times.

In #6046, I use multi endpoint to connect etcd, but its keepalive only detect network rather than leader healthy, so when meet pd leader io hang, the connect cannot update.

rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@lhy1024 @ti-chi-bot @rleungx
etcdutil: revert etcd client without multi endpoint (tikv#6374)
f40e7f8 
ref tikv#6124

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lhy1024 lhy1024

Labels
affects-7.0 severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

etcdutil: revert etcd client in pd legacy server lhy1024/pd
5 participants
@VelocityLight @lhy1024 @rleungx @ti-chi-bot @Lily2025