Disk full can make PD always schedule base on stale information

[BusyJay]

Bug Report

What did you do?

Loading data while delete a store.

What did you expect to see?

The store can be successfully deleted.

What did you see instead?

There are still one peer on the target store not deleted. From logs we can see the region finishes joint state change correctly, but PD still thinks it's not finish.

[2023/02/13 16:08:25.503 +08:00] [INFO] [region.go:640] ["region ConfVer changed"] [region-id=4897] [detail="Remove peer:{id:4898 store_id:5 role:DemotingVoter },Remove peer:{id:22053 store_id:6 role:IncomingVoter },Add peer:{id:4898 store_id:5 role:Learner },Add peer:{id:22053 store_id:6 }"] [old-confver=44] [new-confver=46]
[2023/02/13 16:08:25.504 +08:00] [ERROR] [cluster.go:911] ["failed to save region to storage"] [region-id=4897] [region-meta="id:4897 start_key:\"7480000000000000FF545F720380000000FF00000D7700000000FB\" end_key:\"7480000000000000FF545F720380000000FF00000DDB03800000FF0000000003038000FF0000000002F00000FE\" region_epoch:<conf_ver:46 version:58 > peers:<id:4898 store_id:5 role:LearnerLearner > peers:<id:4900 store_id:1 > peers:<id:15193 store_id:3 > peers:<id:22053 store_id:6 >"] [error="[PD:leveldb:ErrLevelDBWrite]write /home/tidb/tidb-data/pd-2379/region-meta/000001.log: no space left on device: write /home/tidb/tidb-data/pd-2379/region-meta/000001.log: no space left on device"]
[2023/02/13 16:08:25.504 +08:00] [INFO] [operator_controller.go:651] ["send schedule command"] [region-id=4897] [step="leave joint state, promote learner peer 22053 on store 6 to voter, demote voter peer 4898 on store 5 to learner"] [source=heartbeat]

What version of PD are you using (pd-server -V)?

v6.5.0

[bufferflies]

The root cause: the peer-4898 is pending peer.
fixed by: #5795