task and subtask would be restarted automatically after we restart dm-worker

[IANTHEREAL]

Feature Request

Is your feature request related to a problem? Please describe:

I’m always frustrated when we must start task or subtask manually after we restart dm-worker.
Essential cause is that both dm-master and dm-worker don’t store task/subtask configuration in a remote storage or local disk

Describe the feature you'd like:

After we restart dm-worker, task and subtask would be restarted automatically.
DM should store task/subtask information in a remote storage or local disk

Describe alternatives you've considered:

Declare in advance that these solution is irrelevant about HA of dm-master/dm-worker processes

S1: store task configuration in dm-master

• need a remote storage
• need hearbeat mechanism between task and dm-master

S2: store subtask configuration in dm-worker’s disk

• implement a local file task management component

if keeping dm-master simple, we would like to choose S2, there are benefits:

• Don’t add a remote storage process in DM cluster
• Implementing HA of dm processes using a third party program, like k8s operator, dm-worker only need to store relay log, meta, task configuration in to a NFS in this solution

[csuzhangxc]

I think S2 is enough currently and easier to be implemented.

other things need to consider:

1. handle stop-task and update-task
2. maybe need a flag to disable this feature for debugging or other purposes?

[IANTHEREAL]

yep, we must handle stop-task and update-task. But I don't quite understand this special debug purpose

[csuzhangxc]

I haven’t thought very well about this debug purpose, maybe you can just ignore it.