Terminate tikv server instead of killing it when evict leader timeout or is skipped

[BusyJay]

Feature Request

Is your feature request related to a problem? Please describe:
Due to tikv/tikv#10353, if there are still leaders on a TiKV and it is shutdown gracefully, there is a chance to corrupt transactions.

Describe the feature you'd like:
TiKV is working on a patch to fix the problem, but for older versions, it's required to use SIGKILL to terminate TiKV immediately if evict leader timeout or is skipped to avoid potential risk. So there are two TODOs:

1. Set a larger timeouts for evicting leaders;
2. If evicting leader timeouts, use systemctl kill -s KILL service to stop TiKV.

Why the featue is needed:
See above.

Describe alternatives you've considered:
Ignore the problem, then there is a risk that upgrading cluster, especially large cluster, can corrupt data.

Teachability, Documentation, Adoption, Migration Strategy:

[AstroProfundis]

@BusyJay Could you confirm killing TiKV with SIGKILL is safe?

[BusyJay]

Yes, we have tests to random kill TiKV. Or we can enlarge evict leader timeout to infinite, which can be considered safer.

[BusyJay]

Due to tikv/tikv#9624, it may not be safe to kill -9 for all versions of TiKV, especially when lightening is used. So the best option seems to be set an infinite timeout for evicting leader.

[AstroProfundis]

Infinite timeout is not good in practice, that could block the upgrade process and the user have to interrupt the command by hand, leaving the cluster in a hybrid status with both old and new versions of components running, and can not overcome from that status (because retry of upgrade command will still be blocked at leader evicting).

We can indeed increase the default timeout for evicting leader, but that don't really solve the problem.

[BusyJay]

We can indeed increase the default timeout for evicting leader, but that don't really solve the problem.

Better than nothing.

[AstroProfundis]

OK