Trainer save model in distributed training.

[jacquesqiao]

No description provided.

[helinwang]

Hi @jacquesqiao , please refer to this design doc for more information: checkpointing

Please note that the design doc is about checkpointing, but this issue is about save model.

The main differences are:

1. Checkpointing save additional states, such as optimizer state.
2. Checkpointing don't involve merge checkpoint files from different Pservers into a same file. But save model need to merge parameter files into a single file. Since merging is involved, some kind of synchronization is needed.

We can discuss the synchronization ideas, here is my initial idea for your consideration: maybe Pserver #0 can be the one who performs the merge process. And Pserver #0 will watch etcd notification, to be notified when saving model finished on other Pservers.

Feel free to split this issue to different smaller issues, so that the size of each PR can be small.

[helinwang]

Also please see this discussion for a reference of implementing getting ID for Pservers: #1620 (comment)

Edit: Sorry this is not very relevant. Keeping it here in case you are interested.

[dzhwinter]

get it, thanks for the clear explanation! @helinwang

[Yancey1989]

Following this discuss #2638 , It seems that we declining to save model on the trainer, so maybe we can close this issue?

[helinwang]

Following this discuss #2638 , It seems that we declining to save model on the trainer, so maybe we can close this issue?

@Yancey1989 Thanks! I have changed the title to Trainer save model in distributed training..