Design doc: operator based parameter server.

[helinwang]

Here could be easier to review

[Superjomn]

implement

[helinwang]

Fixed, thanks!

[jacquesqiao]

W is also an input of parameter update op

[putcn]

if I understand correctly, for SEND op, it only sends graph, for RECEIVE op, it only receives gradient. any given worker sees only part of the whole graph, but how would the training data travel through the whole graph? would there be some part of the graph idle until data reaches it's parent?

[typhoonzero]

Question: if there are multiple parameters(or variables) to send to parameter server, are we:

1. create multiple Send operators for each variable or,
2. create one Hash operator to divide parameters equally and one Sender operator to do send.

[typhoonzero]

And also, maybe add some description about the send, recv operators like:

• Send:
  • Inputs:
  • Outputs:
  • Description:

[Yancey1989]

The same confusion with @typhoonzero , maybe we need an operator to sharded parameters.

For others, from @typhoonzero

create multiple Send operators for each variable

Maybe we only need one Sender operator. If we have too many parameters, too much Sender operator will cased too much connection to parameter server.

[helinwang]

@typhoonzero @Yancey1989 sorry, the PR could be more clear:
In short, the answer is "create multiple Send operators for each variable".

From the graph's perspective, the Send and Recv OP are one for each variable (but not one per replica: different (trainer) replicas share one Send and Recv for each variable).
In the implementation detail, we could group send implementations to a single port handler.

In this design the variable placement is done by the graph converter before the graph is sent to worker, so it's not a runtime concept like a Hash operator. I think the "Hash" solution is for the simplest element-wise optimization case. If we want the parameter server to do things more than element-wise operation, we need to decide the parameter variable and OP placement before the graph is sent to worker.

[typhoonzero]

Understood. You mean people who develop graphs, do not need to look into the implement details about how we actually send and recv variables, the graph is how the calculations flow logically. But when we build and optimize the graph, we can make the actual send operation one per trainer.

Will you add some implementation thoughts in this PR or in another one?

[helinwang]

@typhoonzero Thanks for the reminder on the implementation thoughts! That's a good idea. I will perhaps not mention implementation detail in this PR, but create a separate issue discussing it. After receiving you guys' comments, I have some point need to re-think and will update this PR and create the implementation detail issue at that time.

[helinwang]

@putcn Sorry, my PR could be more clear, the Send OP is for sending any tensor (not the graph), the Recv OP is for receiving any tensor. In this way data (tensor) can travel through the graph.
Yes, the Recv OP will be blocked until it receives the data, so the OPs that depends on Recv will be idle until then.

[helinwang]

From @Superjom : graph拼起来之后应该还是一个可以用的graph。

[typhoonzero]

LGTM!

[dzhwinter]

According to the current design, there is more concept need to clarify in this design doc.
1、How the scope should be implemented in a distributed environment? Every sub-graph must run with a scope to get variable value from it. How to partition the global scope?
here is some discussion related
2、The distributed graph need to compatible with Block design.

[dzhwinter]

LGTM