MPPTask destruct when holding the lock of MPPTaskManager, and cause TiFlash hang forever

[windtalker]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Consider A MPP query has 2 MPPTask in one TiFlash node, task1 and task2, and task1 reads data from task2 using local tunnel.

1. task1 and task2 both dispatched to TiFlash, and after prepare and preprocess, both of them are waiting for scheduler, so they are pushed into the waiting_tasks queue
2. The scheduler lets task2 to run, then found the thread usage exceed the hard limit, so makes the schedule state of task1 as EXCEEDED
3. Task1 found its schedule state is EXCEEDED, then it throws exception, and unregister itself from MPPTaskManager, note that unregister only moves the task1 from task_map, and waiting_tasks queue still holds the reference of task1, and after runImpl finishes, the reference in waiting_tasks is the last reference of task1 in the system.
4. TiDB found task1 is failed, then it sends CandMPPTask request to TiFlash to cancel the mpp query
5. In CancelMPPQuery , it acquires the lock of MPPTaskManager, then calls scheduler->deleteQuery, inside deleteQuery, it removes task1 from waiting_tasks, since the task1 in waiting_tasks is the last reference of the shared ptr, task1 is destructed after it is removed from waiting_tasks
6. When destruct task1, its ExchangeReceiver will wait the reading thread exit, for task1, the reading thread in ExchangeReceiver is the local read thread which tries to read data from task2, local read thread can only be exited after task 2 finishes or task1/task2 is cancelled, but
   • task2 can not finish because task 1 is not reading its output
   • task1 and task2 can not be cancelled, because the cancel thread is now waiting task1 to be deconstructed

So deadlock happens, and since the CancelMPPQuery holds the lock of MPPTaskManager, no more queries can be served.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)