Replace Loader with Lightning

[suzaku]

Description

Replace Loader with Lightning

Background

In DM, the loader.Loader struct implements loading data from mydumper output files into TiDB. Since v3.0.3, Lightning supports the TiDB backend, which enables Lightning to do the same thing.

Proposal

We consider the TiDB backend mode of Lightning the better implementation of the two, because:

1. It has more complete parsing support that can succeed in cases loader.Loader would fail;
2. It has more fine-grain concurrency control, allowing users to config read and write concurrency differently.

So we propose to replace Loader with Lightning.

Success Criteria

• Reimplement loader.Loader (which is an implementation of the Unit interface) with Lightning;
• Maintain backward-compatibility of config files like task.yaml

Category

Enhancement

Value

Enhance the ability of load data.

Score

1500

SIG slack channel

sig-migrate

Mentor

csuzhangxc

TODO list

• Refactor Lightning to make the loader path usabale as a library in DM
• A basic replacement of loader that implements essential methods of the Unit interface (eg. Init, Process) and leave the other methods with dummy implementations
• Implement Close
• Implement Pause and Resume
• Implement Status, Error, Type, IsFreshTask

Documentations

• DM 源码阅读系列文章（四）dump/load 全量同步的实现
  
• 如何使用 Lightning 导入大量数据到 TiDB/MySQL

Project

N/A

[csuzhangxc]

We are integrating Lighting into TiDB, after we did it we can update DM to call this embedded Lightning via SQL statements.

[Little-Wallace]

I'm add some interface for lightning and try to replace dm-loader with lightning.
See pr pingcap/tidb#27934

[lance6716]

I'll transfer this issue to pingcap/tiflow