Supporting arbitrary directed acyclic networks

[aravindhm]

Backpropagation can work on arbitrary directed acyclic networks. Does the current implementation support a blob being used by two different layers. I see that each layer is initializing bottom_diff to zero and then accumulating gradient into it, this would override the gradient contributed by a second layer acting on the same blob. Or am I missing something? If not, is simply changing = to += a good way of solving the problem?

[Yangqing]

It does not yet. Decaf does it by detecting branches and inserting SplitLayers:

https://github.com/UCB-ICSI-Vision-Group/decaf-release/blob/master/decaf/base.py#L168

to the internal instantiation of the network, but caffe has not implemented such mechanism yet.

Note that simply changing = to += might have additional complexities (e.g. whether and who initializes gradients to zero, and requiring all existing and future layers to enforce +=), but it is definitely doable by either this or inserting utility layers.

[aravindhm]

The split layer method would require more memory.
The cpu_diff() / gpu_diff() can be initialized to zero by the network (net.cpp) in the Net::Backward() function before the 'for' loop over layers begins.
Do you feel its worth enforcing the += instead of a splitlayer mechanism?

[Yangqing]

I am a little bit torn, since enforcing += will require us to always to two
passes to the gradients, something that is not that kosher for most use
cases. But indeed the split layer would require more memory.

Yangqing

On Mon, Feb 10, 2014 at 3:08 PM, aravindhm notifications@github.com wrote:

The split layer method would require more memory.
The cpu_diff() / gpu_diff() can be initialized to zero by the network
(net.cpp) in the Net::Backward() function before the 'for' loop over layers
begins.
Do you feel its worth enforcing the += instead of a splitlayer mechanism?

Reply to this email directly or view it on GitHubhttps://github.com//issues/92#issuecomment-34700270
.

[aravindhm]

Since most of the network won't have splits, a second pass through the gradients seems wasteful. I'll use a split layer - manually defined for simplicity.

[shelhamer]

Split layers are in dev as of #129 and on their way to master.