”write to tikv with no leader returned“ if we meet NotLeader which is caused by on-going schedule

[D3Hunter]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

it's a corner case, and not easy to reproduce in normal run, even with very large mount of data, below is the timeline which may trigger this case:

• before we pause scheduler by key-range for a region, there's ongoing schedule which'll not be interrupted by pause
• lightning start import, but got NotLeader from tikv when ingest, and the ongoing schedule changes leader and peer-list at same time, but lightning only update leader on this case:
  

  tidb/br/pkg/lightning/backend/local/region_job.go

  Line 598 in 477fbaf

  Leader: newLeader,

• then on retry write we fail on this error
  

  tidb/br/pkg/lightning/backend/local/region_job.go

  Lines 350 to 351 in 477fbaf

  	if leaderID == region.Peers[i].GetId() {
  	leaderPeerMetas = resp.Metas

  
  

  tidb/br/pkg/lightning/backend/local/region_job.go

  Lines 358 to 363 in 477fbaf

  	if len(leaderPeerMetas) == 0 {
  	log.FromContext(ctx).Warn("write to tikv no leader",
  	logutil.Region(region), logutil.Leader(j.region.Leader),
  	zap.Uint64("leader_id", leaderID), logutil.SSTMeta(meta),
  	zap.Int64("kv_pairs", totalCount), zap.Int64("total_bytes", totalSize))
  	return errors.Errorf("write to tikv with no leader returned, region '%d', leader: %d",

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

[D3Hunter]

cc @lance6716

[D3Hunter]

5.4 don't has this problem since writeAndIngestByRange always retry on error even non-retriable

[lance6716]

but got NotLeader from tikv

lightning get this error when write or ingest?

[D3Hunter]

on ingest. updated