PD panic caused by go-queue

[Yisaer]

Bug Report

What did you do?

In the internal test, pd panic whe there was lots of merge operator. Here is the panic stack:

panic: runtime error: index out of range [2] with length 2

goroutine 227117623 [running]:
github.com/phf/go-queue/queue.(*Queue).PushBack(0xc0aedd95c0, 0x2577da0, 0xc2a1bc2a90)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/phf/go-queue@v0.0.0-20170504031614-9abe38d0371d/queue/queue.go:154 +0x2e2
github.com/tikv/pd/pkg/movingaverage.(*AvgOverTime).Add(0xc0aedd95f0, 0x0, 0x28fa6ae00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/pkg/movingaverage/avg_over_time.go:80 +0x9b
github.com/tikv/pd/pkg/movingaverage.(*TimeMedian).Add(0xc0aedd9650, 0x0, 0x28fa6ae00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/pkg/movingaverage/time_median.go:48 +0x9c
github.com/tikv/pd/server/statistics.(*dimStat).Add(0xc2d1b63880, 0x0, 0x28fa6ae00)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/statistics/hot_peer.go:46 +0x6d
github.com/tikv/pd/server/statistics.(*hotPeerCache).updateHotPeerStat(0xc019ff1480, 0xc217d78460, 0xc04cd90280, 0x0, 0x0, 0x28fa6ae00, 0x2)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/statistics/hot_peer_cache.go:417 +0x91
github.com/tikv/pd/server/statistics.(*hotPeerCache).CheckRegionFlow(0xc019ff1480, 0xc1a8252600, 0xc19efb6780, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/statistics/hot_peer_cache.go:195 +0x548
github.com/tikv/pd/server/statistics.(*HotCache).CheckWrite(...)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/statistics/hot_cache.go:42
github.com/tikv/pd/server/cluster.(*RaftCluster).CheckWriteStatus(...)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/cluster/cluster.go:1478
github.com/tikv/pd/server/cluster.(*RaftCluster).processRegionHeartbeat(0xc004fd0120, 0xc1a8252600, 0x6047461c, 0x26dc04d9)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/cluster/cluster.go:548 +0x12c
github.com/tikv/pd/server/cluster.(*RaftCluster).HandleRegionHeartbeat(0xc004fd0120, 0xc1a8252600, 0x4bee660, 0x278b43928)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/cluster/cluster_worker.go:34 +0x39
github.com/tikv/pd/server.(*Server).RegionHeartbeat(0xc000392dc0, 0x3286860, 0xc2f4adc590, 0x32a0c40, 0xc000392dc0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/src/github.com/pingcap/pd/server/grpc_service.go:452 +0x866
github.com/pingcap/kvproto/pkg/pdpb._PD_RegionHeartbeat_Handler(0x2771120, 0xc000392dc0, 0x327f960, 0xc0e3e528c0, 0xd246d8, 0x20)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20210204054616-1c1ed89bb167/pkg/pdpb/pdpb.pb.go:5444 +0xb0
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1(0x2771120, 0xc000392dc0, 0x327f960, 0xc0e3e528c0, 0x18, 0xc39a152410)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:71 +0xf1
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1(0x2771120, 0xc000392dc0, 0x32807a0, 0xc17e5a3e00, 0xc0e3e528a0, 0xc39a1523c0, 0x4c19648, 0x25de740)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 +0xeb
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1(0x2771120, 0xc000392dc0, 0x32807a0, 0xc17e5a3e00, 0x4c1aee0, 0x2668100)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:74 +0x99
go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1(0x2771120, 0xc000392dc0, 0x32807a0, 0xc17e5a3e00, 0xc0e3e528a0, 0xc39a1523c0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20191023171146-3cf2f69b5738/etcdserver/api/v3rpc/interceptor.go:225 +0x14e
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1(0x2771120, 0xc000392dc0, 0x32807a0, 0xc17e5a3e00, 0xc0e3e528a0, 0x2e4daf0, 0x326da40, 0xc14ad4ade0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:79 +0x14f
google.golang.org/grpc.(*Server).processStreamingRPC(0xc005137c80, 0x32897a0, 0xc20d65f080, 0xc0b8a16000, 0xc0018505a0, 0x423f4c0, 0x0, 0x0, 0x0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1244 +0x47a
google.golang.org/grpc.(*Server).handleStream(0xc005137c80, 0x32897a0, 0xc20d65f080, 0xc0b8a16000, 0x0)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1317 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc19c996984, 0xc005137c80, 0x32897a0, 0xc20d65f080, 0xc0b8a16000)
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/jenkins/agent/workspace/build_pd_multi_branch_master/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:720 +0xa1

What version of PD are you using (pd-server -V)?

760fb65

[Yisaer]

When the panic happen, the cluster is doing lots of merge operation. As the region will send a heartbeat after merge finish. The panic might be caused by handling the same regions'heartbeats from after merge and regular heartbeat.

[nolouch]

closed by #3506