Restore fails due to PD returns Unavailable not leader

[overvenus]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

Restore a backup. BR exits unexpectedly

Error: rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader

2021/08/04 11:04:15.816 +08:00] [ERROR] [restore.go:26] [“failed to restore”] [error=“rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader”] [errorVerbose=“rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader\ngithub.com/tikv/pd/client.(*client).ScanRegions\n\tgithub.com/tikv/pd@v0.0.0-20210105112549-e5be7fd38659/client/client.go:598\ngithub.com/pingcap/br/pkg/restore.(*pdClient).ScanRegions\n\tgithub.com/pingcap/br@/pkg/restore/split_client.go:385\ngithub.com/pingcap/br/pkg/restore.PaginateScanRegion\n\tgithub.com/pingcap/br@/pkg/restore/util.go:491\ngithub.com/pingcap/br/pkg/restore.(*RegionSplitter).Split\n\tgithub.com/pingcap/br@/pkg/restore/split.go:103\ngithub.com/pingcap/br/pkg/restore.SplitRanges\n\tgithub.com/pingcap/br@/pkg/restore/util.go:432\ngithub.com/pingcap/br/pkg/restore.(*tikvSender).splitWorker\n\tgithub.com/pingcap/br@/pkg/restore/pipeline_items.go:234\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“main.runRestoreCommand\n\tgithub.com/pingcap/br@/cmd/br/restore.go:26\nmain.newFullRestoreCommand.func1\n\tgithub.com/pingcap/br@/cmd/br/restore.go:97\ngithub.com/spf13/cobra.(*Command 1).execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:56\nruntime.main\n\truntime/proc.go:203”]
[2021/08/04 11:04:15.816 +08:00] [ERROR] [main.go:58] [“br failed”] [error=“rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader”] [errorVerbose=“rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader\ngithub.com/tikv/pd/client.(*client).ScanRegions\n\tgithub.com/tikv/pd@v0.0.0-20210105112549-e5be7fd38659/client/client.go:598\ngithub.com/pingcap/br/pkg/restore.(*pdClient).ScanRegions\n\tgithub.com/pingcap/br@/pkg/restore/split_client.go:385\ngithub.com/pingcap/br/pkg/restore.PaginateScanRegion\n\tgithub.com/pingcap/br@/pkg/restore/util.go:491\ngithub.com/pingcap/br/pkg/restore.(*RegionSplitter).Split\n\tgithub.com/pingcap/br@/pkg/restore/split.go:103\ngithub.com/pingcap/br/pkg/restore.SplitRanges\n\tgithub.com/pingcap/br@/pkg/restore/util.go:432\ngithub.com/pingcap/br/pkg/restore.(*tikvSender).splitWorker\n\tgithub.com/pingcap/br@/pkg/restore/pipeline_items.go:234\nruntime.goexit\n\truntime/asm_amd64.s:1357”] [stack=“main.main\n\tgithub.com/pingcap/br@/cmd/br/main.go:58\nruntime.main\n\truntime/proc.go:203”]

2. What did you expect to see?

Restore success.

3. What did you see instead?

Failed.

4. What version of BR and TiDB/TiKV/PD are you using?

v4.0.13

See https://asktug.com/t/topic/95904

[YuJuncen]

Maybe tikv/pd#4797

[3pointer]

pingcap/tidb#38948 may fix it