Add ClipLayer #3112
Add ClipLayer #3112
Conversation
|
不太清楚commit的时候为什么python/paddle/trainer_config_helpers/layers.py里 |
paddle/gserver/layers/ClipLayer.cpp
Outdated
| Matrix::resizeOrCreate( | ||
| tmpMtx, outG->getHeight(), outG->getWidth(), false, useGpu_); | ||
| tmpMtx->clipDerivative(*inV, clipThresholdLow_, clipThresholdHigh_); | ||
| inG->addDotMul(*outG, *tmpMtx, 1, 1); |
There was a problem hiding this comment.
if(inG) {
MatrixPtr outV = getOutputValue();
MatrixPtr outG = getOutputGrad();
MatrixPtr tmpMtx;
Matrix::resizeOrCreate(
tmpMtx, outG->getHeight(), outG->getWidth(), false, useGpu_);
tmpMtx->clipDerivative(*inV, clipThresholdLow_, clipThresholdHigh_);
inG->addDotMul(*outG, *tmpMtx, 1, 1);
}
There was a problem hiding this comment.
反向的计算不对吧？梯度直接往前层传递，不需要对梯度截断。
抱歉，我犯蠢了，梯度计算是正确的。
There was a problem hiding this comment.
There was a problem hiding this comment.
- 好的，明白啦。我犯蠢了，这里计算的是如果将
clip当做一种运算，clip这运算的导数：超过上下阈值部分输出是常数，对$x$求导梯度为0；否则对$x$求导，梯度为1。
There was a problem hiding this comment.
对的，clipDerivative计算的是clip的(sub)gradient
| @@ -2169,6 +2169,23 @@ def __init__(self, name, inputs, context_length, **xargs): | |||
| self.create_input_parameter(0, psize, dims) | |||
|
|
|||
|
|
|||
| @config_layer('clip') | |||
| class ClipLayer(LayerBase): | |||
| def __init__(self, name, inputs, clip_threshold_low, clip_threshold_high): | |||
There was a problem hiding this comment.
def __init__(self, name, inputs, clip_threshold_low, clip_threshold_high, **xargs):
proto/ModelConfig.proto
Outdated
| @@ -289,6 +289,11 @@ message DetectionOutputConfig { | |||
| optional uint32 width = 9 [default = 1]; | |||
| } | |||
|
|
|||
| message ClipConfig { | |||
| required float clip_threshold_low = 1; | |||
| required float clip_threshold_high = 2; | |||
There was a problem hiding this comment.
- clip_threshold_low -->min
- clip_threshold_high-->max
There was a problem hiding this comment.
Done，已将type和name进行修改
| clip_threshold_low=clip_threshold_low, | ||
| clip_threshold_high=clip_threshold_high) | ||
| return LayerOutput( | ||
| name, LayerType.CLIP_LAYER, parents=[input], size=input.size) |
There was a problem hiding this comment.
- unit test for this interface
- add this layer in the doc: https://github.com/PaddlePaddle/Paddle/blob/develop/doc/api/v2/config/layer.rst
| * \f] | ||
| */ | ||
|
|
||
| class ClipLayer : public Layer { |
There was a problem hiding this comment.
- ClipLayer 重命名成 ValueClipLayer
There was a problem hiding this comment.
ClipLayer已经是个Layer了，forward一定是value的操作，所以其实觉得ClipLayer名字还好
There was a problem hiding this comment.
Done，使用ClipLayer命名
paddle/gserver/layers/ClipLayer.cpp
Outdated
| class ClipLayer : public Layer { | ||
| protected: | ||
| real clipThresholdLow_; | ||
| real clipThresholdHigh_; |
There was a problem hiding this comment.
- clipThresholdLow_ --> min_
- clipThresholdHigh_ --> max_
There was a problem hiding this comment.
还有一个问题，一般这种截断都是对称截断，因此不需要upper、lower 两个阈值，请问一定需要两个阈值吗？
There was a problem hiding this comment.
觉得两个阈值功能多些，caffe2的clip_op也是min,max两个阈值： https://github.com/caffe2/caffe2/blob/master/caffe2/operators/clip_op.h#L55
There was a problem hiding this comment.
Done，已修改为min_和max_
paddle/gserver/layers/ClipLayer.cpp
Outdated
| Matrix::resizeOrCreate( | ||
| tmpMtx, outG->getHeight(), outG->getWidth(), false, useGpu_); | ||
| tmpMtx->clipDerivative(*inV, clipThresholdLow_, clipThresholdHigh_); | ||
| inG->addDotMul(*outG, *tmpMtx, 1, 1); |
There was a problem hiding this comment.
反向的计算不对吧？梯度直接往前层传递，不需要对梯度截断。
抱歉，我犯蠢了，梯度计算是正确的。
proto/ModelConfig.proto
Outdated
| @@ -289,6 +289,11 @@ message DetectionOutputConfig { | |||
| optional uint32 width = 9 [default = 1]; | |||
| } | |||
|
|
|||
| message ClipConfig { | |||
| required float clip_threshold_low = 1; | |||
| required float clip_threshold_high = 2; | |||
There was a problem hiding this comment.
- clip_threshold_low -->min
- clip_threshold_high-->max
Add ClipLayer for clipping the input value by the threshold.