Unified Distributed Parallel Execution Framework

[zimulala]

Feature Request

Is your feature request related to a problem? Please describe:

In the first phase, we implemented a concise distributed DDL processing framework. In the second phase, I hope to change this framework to be more general, so that more functions (currently TTL and Import, etc.) can be connected, so as to achieve a unified distributed parallel execution framework.

Development Plan

• Identify common architectural interfaces docs
• Each module design document
• Design doc #43010

Each module Plan

Util

• Implement the simplest logical flow(run through demo normally) for proto #41990
• Change the test package to *_test #42262

Handle

• add handle to submit task #43065

Storage

• Implement the simplest logical flow(run through demo normally) for storage #41984
• guarantee guarantee idempotent of submitting a global task to disttask framework #42610
• Merge GlobalTaskManager and SubtaskManager #42785
• disttask.TaskManager use resource pool to get session #42831
• support error handling for distributed adding index #43029

Scheduler

• Implement the simplest logical flow(run through demo normally) for scheduler #42020
• add OnSubtaskFinished interface #43153
• disttask: scheduler can run subtask concurrently #43369
• using TiDB nodes' Ip and port to generate disttask execid #43628
• record start/end time for subtask #45990

Dispatcher

• Implement the simplest logical flow(run through demo normally) for dispatcher #41748
• Implement the "add index" GTaskFlowHandle. #42045 // Here 2 issues
• detectTask in parallel for dispatcher #42383
• Complete several storage pkg operations in a transaction
• Implement scheduler for the partition table. #42754
• disttask: dispatcher should update global task and add new tasks in a txn #42848
• Change random choose TiDB to round roubin choose TiDB to be available TiDB to run a subtask. #43439
• Support cancelling a global task in dist task framework #43497

Client

• Implement the "add index" handle and related task meta. #42042

Example

• add a test that can run the simplest logical flow normally

DDL

• Implement dispatcher code for DDL Add index step1 #42202
• Move the disttask codes for DDL to a standalone package #42395
• Start dispatcher and scheduler on each TiDB instance when node starts #42481
• integrate add index with the framework #42974
• Remove the old distribute framework #43016
• support distributed adding index for normal table #43290
• support basic row count stat for distributed add index #43642
• rollback distributed adding index is not handled correctly. #43915

Import

• disttask: implement proto for load data #42452
• add handle for load data
• add dispatcher for load data #42591
• add scheduler and subtask executor for load data #42880
• add subtask executor for load data
• disttask: enable load data for distribute framework #43106
• disttask: add finish flow for distribute framwork #43675
• disttask: add checksum verification for load data #43715
• add StatementContext result for load data #43868

system variables

• Change tidb_ddl_distribute_reorg to tidb_dist_task #42186

Test Plan

• Test plan document