Enhance distributed task execution framework

[ywqzzy]

Enhancement

We need to make Distributed task framework implementation more modular, reduce complexity of implement, decouple the implemented and future function. Moreover, the framework must be more robust and easy to use.

1. By implementing a goroutine pool to control the concurrency of running tasks in the distributed framework, we can effectively prevent background tasks from consuming excessive compute resources and improve management scalability.
2. Use one Dispatcher take care one task, simplify the complexity of Dispatcher management.
3. The task status should follow the state machine, the whole procedure should be simple and easy to reasoning.
4. When the Dispatcher is in error status and does not work, the impact scope can only be limited only to the task it takes care of. That means the dispatcher and scheduler's execution are decoupled. It will make the task run in distributed more robust.
5. Consider error handling of the task, make the task can be more robust when encountering many abnormal scenes.
6. Support cleanUp routines to do garbage collection for tasks.
7. Support subtask history table to provide more management of the framework.
8. Support metrics to maintain observability.
9. Support role to make the node selection policy more flexible.
10. Provide plan interface to make the whole execution process follow the plan, the business logic will be much clearer to trace and implement.
11. Introduce operator abstraction to make it easier to compose functionalities and provide more sophisticated task runtime statistics collect logic. Further more, the cpu resource should be easy to manage.

Dispatcher

HA

• 1. disktask: replace failure nodes with alive ones  #45935
• 2. disttask: handle left subtask in running state #47300
• 3. disttask: refine scheduler error handling #47313
• 4. disttask: persist business error to subtask table #47334
• 5. disttask/ddl: refine add index dispatcher for error handling #47886
• 6. disttask: add more retryable error #48033
• 7. disttask/ddl: refine manager error handling #48095
• 8. disttask: fix subtask finished immediately and mark success when encountering network partition  #48660

Pause And Resume

• Support pause and resume. disttask: support pause/resume for framework #47037

CleanUp routine

• 1. ddl: clean up cloud storage files after global sort #47084
• 2. disttask: Add history table for global task #47179
• 3. disttask: add cleanup routine for framework #47218

Refinement

• 1. disttask: refine dispatcher #45460
• 2. disttask: refine scheduleTask #45828
• 3. distribute framework: restrict the task state transform rule #45932
• 4. disttask: remove the option of running subtasks concurrently #45566
• 5. disttask, ddl: rename flowhandle #46497
• 6. disttask: add pending to failed state transform #46357
• 7. disttask, ddl: refine disttask_flow #46472
• 8. disttask: refactor dispatcher so we have chance to add init logics per task type #46603
• 9. disttask: refactor scheduler part #46669
• 10. disttask: increase checkTaskRunningInterval #46686
• 11. dist ddl: gc subtask history table #46722
• 12. disttask: remove mini executor abstraction #46922
• 13. disttask, ddl: move backend ctx register code to scheduler #46952
• 14. disttask: update task in framework #46913
• 15. disttask: refactor task step and batch subtask dispatching #46957
• 16. Refine onNextSubtasksBatch disttask: refactor task step and batch subtask dispatching #46957
• 17. disttask: remove scheduler/dispatcher id from logs and refinement #47257
• 18. disttask: fix too much log on cleanup #47406
• 19. disttask: persist business error to subtask table #47334
• 20. disttask: define TaskType/TaskState/Step instead of string/int64 #47611
• 21. disttask: more log #47575

role

• 1. disttask: add role to select instance #46453
• 2. disttask: cleanup framework meta when nodes shutdown #47536

dynamic dispatch subtasks

• disttask: dynamic dispatch subtasks #46593

Operator

Introduce operator

• 1. disttask: add operator abstraction #46279
• 2. workerpool refine: workerpool: generic result type for worker pool #46185

Integration

• 1. add index basic: ddl, disttask: implement add index operators #46414
• 2. add index integration: ddl: integrate add index operator to dist framework #46495
• 3. import into: importinto: convert encode and sort part into operator #46532
• 4. global sort: ddl, lightning: integrate operator for global sort #46734

Planner

Introduce planner

• 1. distribute framework: add planner #46395

Integration

1. add index
2. import into:

• distribute framework: add planner #46395
• ddl: eliminate ingest step for add index with local engine #47982

3. global sort

SubTask runtime

• 1. disttask: record subtask start/update time #46361

metrics

1. Support metric. disttask: add metrics collection for dispatcher and scheduler #47017

misc

• 1. distask: refine dispatcher test #45821
• 2. disttask: replace manually mock with gomock #46392
• 3. bootstrap: add summary field for system table tidb_background_subtask #46562

bug fix

• 1. disttask: fix dispatcher cancel #45790
• 2. disttask: fix wrongly add subtask when state transform fail #46259
• 3. disttask: fix owner change flaky test #46075
• 4. disttask: fix run one step SubtaskExecutor twice #46106
• 5. disttask, ddl: dispatch step-two tasks to proper instances #46464
• 6. disttask: refactor dispatcher so we have chance to add init logics per task type #46603
• 7.ddl: fix taskNodes temporarily #46671
• 8. disttask: persist subtask error properly #46674
• 9. import into jobs is always running after kill ddl owner #46735
• 10.import into job finished but no data exists when injecting one of tidb failure for 2mins #46951
• 11. br: stop s3 retryer when connection is refused #46938
• 12. disttask: batch insert subtasks #47207
• 13. disttask/ddl: change memory quota for writer #47251
• 14. ddl: fix can't pause add index in dist task mode #47450
• 15. disttask: fix upgrade #47316
• 16. distask: fix panic on refresh task fails or returns nil #47525
• 17. disttask: add sleep when subtask == nil #47668
• 18. disttask: fix removing meta when met network partition for so long then recover from it #48005
• 19. disttask/ddl: fix pause succeed task then run a new task #47804
• 20. ddl: fix panic in pause #47914
• 21. disttask: fix upgrade missing create framework_meta  #47744
• 22. disttask: fix cancel from grpc mark subtasks as failed #48339
• 23. disttask: fix wrong plan when filterByRole #48366
• 24. disttask: correct the usage of context #48343
• 25. disttask: fix split region batch panic #48348
• 26. disttask/ddl: fail task for panic #48135
• 27. disttask: fix subtask finished immediately and mark success when encountering network partition  #48660

[ywqzzy]

/close as all tasks completed