Add distributed training overview doc #9937
Conversation
jacquesqiao
changed the title
jacquesqiao
requested review from
Yancey1989,
helinwang,
panyx0718,
emailweixu and
typhoonzero
| @@ -0,0 +1,57 @@ | |||
| ## distributed training overview doc | |||
There was a problem hiding this comment.
Capitalize the first letter.
There was a problem hiding this comment.
I think this file can be renamed to README.md
|
|
||
| The training process of synchronous training is: | ||
|
|
||
|  |
There was a problem hiding this comment.
Better to use a "cross functional flowchart" to represent both procedure and communications.
|
|
||
| The training process of synchronous training is: | ||
|
|
||
|  |
There was a problem hiding this comment.
lookup table training => synchronous distributed training.
| 1. Pserver: | ||
| 1. Each parameter has a queue to receive its gradient from trainers. | ||
| 1. Each parameter has a thread to read data from the queue and run optimize block, using the gradient to optimize the parameter. | ||
| 1. Use a independente thread to handle RPC call `GetVariable` for trainers to get parameters back.(Maybe here we should use a thread pool to speed up the parameter fetch.) |
There was a problem hiding this comment.
Use a independente thread => Using an independent thread
There was a problem hiding this comment.
speed up the parameter fetch => speed up fetching the parameters.
| The training process of synchronous training is: | ||
|
|
||
|  | ||
|
|
There was a problem hiding this comment.
In the figure, param_1@GRAD.shard_n.trainer_0 => param_1@GRAD.block_n.trainer_0.
There was a problem hiding this comment.
fixed, thanks! @tonyyang-svail
refs: #9932
refs: https://docs.google.com/document/d/1-W3HHC0yNYolGuYL7L0Crxs3Us8rZfGjlSe0o5X7lQA/edit#heading=h.td8lf867c0rd
task list: #9941