stride pooling for seqlastin and seqfirstin #1701

luotao1 · 2017-03-24T06:45:50Z

part one of #1608：

完成Average、Max和SequenceLastInstance的基类SequencePoolLayer，对stride参数的支持。
添加Argument::poolSequenceWithStride函数，并对应添加该函数的单测test_argument.cpp。
完成SequenceLastInstanceLayer对stride参数的支持，对应添加test_layerGrad中的单测。同时在trainer_config_helpers中修改first_seq和last_seq的接口，并完善python单测。

Average和Max的计算过程比SequenceLastInstance略复杂，会在下一个PR中进行完善。

luotao1 · 2017-03-24T09:11:45Z

考虑到average和max的计算单元接口如下，都使用IVectorPtr来传递第二个参数：

void CpuMatrix::sequenceAvgForward(Matrix& a, const IVector& startsPos, int mode) ;
void maxSequenceForward(Matrix& input, const IVector& sequence, IVector& index);

因此，将std::vector<int> stridePositions_变量改成IVectorPtr stridePositions_，符合原先的两个接口形式。

lcy-seso · 2017-03-28T03:41:12Z

paddle/gserver/layers/SequencePoolLayer.cpp

@@ -67,6 +66,14 @@ void SequencePoolLayer::forward(PassType passType) {
        << "when trans_type = seq, input must hasSubseq";
    output_.degradeSequence(input);
  }
+  if (stride_ > 0) {
+    CHECK_EQ(input.hasSubseq(), 0UL)
+        << "sequence stride pooling is not suitable for hasSubseq now";


is not suitable --> is invalid.

lcy-seso · 2017-03-28T03:45:43Z

python/paddle/trainer/config_parser.py

@@ -2495,10 +2496,11 @@ def __init__(self,
            **xargs)
        config_assert(
            len(inputs) == 1, 'SequenceLastInstanceLayer must have 1 input')
+        if trans_type == 'seq':
+            config_assert(stride == -1, 'subseq do not support stride window')


do not --> does not

lcy-seso · 2017-03-28T05:27:18Z

paddle/parameter/Argument.cpp

+      // empty sequence
+      tgtBuf[seqId + 1] = tgtBuf[seqId];
+    } else if (seqLength < stride) {
+      tgtBuf[seqId + 1] = tgtBuf[seqId] + 1;


可以不需要 else if (seqLength < stride) 这个条件分枝，计算逻辑只在 seqLength % stride 时，有差别。

已去掉该分支。

lcy-seso · 2017-03-28T05:31:47Z

paddle/parameter/Argument.cpp

+      tgtBuf[seqId + 1] = tgtBuf[seqId] + 1;
+    } else {
+      tgtBuf[seqId + 1] = tgtBuf[seqId] + ceil((float)seqLength / stride);
+      int size =


590 ~ 592 行代码可以简化，先计算size，tgtBuf[seqId + 1] = tgtBuf[seqId] + size

size_t size = (seqLength % stride) ? seqLength / stride : seqLength / stride + 1;

lcy-seso · 2017-03-28T05:53:33Z

paddle/parameter/Argument.cpp

+   * If input.sequenceStartPositions = [0, 9, 14, 17, 30] and stride = 5,
+   * then sequenceStartPositions = [0, 2, 3, 4, 7],
+   * and stridePostions = [0, 5, 9, 14, 17, 22, 27, 30]
+   */


stridePostions 的计算少了一个 “输入序列是逆序”这个分枝。

stridePostions 在序列正序和逆序时的计算结果不一样

上面的例子中：

stridePostions = [0, 5, 9, 14, 17, 22, 27, 30] # 正序 stridePostions = [0, 4, 9, 14, 17, 20, 25, 30] # 逆序

luotao1 · 2017-03-31T09:38:18Z

@lcy-seso 已根据要求进行修改

lcy-seso

almost LGTM. only need some small changes.

lcy-seso · 2017-04-06T08:38:09Z

paddle/parameter/Argument.cpp

+   */
+  CHECK(input.sequenceStartPositions);
+  CHECK_EQ(input.hasSubseq(), 0UL);
+  CHECK_GT(stride, 0) << "stride must larger than 0";


调用 poolSequenceWithStride 这个函数时，外围的逻辑分枝已经是 stride > 0，没有subSeq，所以不再需要 573 ~ 574 这两行的检查。

sequenceStartPositions 在调用这个函数之前已经使用（访问）过，572 行已经不再需要检查。

虽然这次外围的逻辑分支都已经检查过了，但以后调用时可能会出现外围没有做check的情况，因此想保留这些check。

lcy-seso · 2017-04-06T08:57:57Z

paddle/parameter/Argument.cpp

+    } else {
+      int size = ceil((float)seqLength / stride);
+      tgtBuf[seqId + 1] = tgtBuf[seqId] + size;
+      for (int i = 0; i < size - 1; i++) {


for 循环中的循环变量i 用 ++i

lcy-seso · 2017-04-06T09:37:29Z

paddle/parameter/Argument.h

+   After pooling with stride n (n is smaller than sequence length),
+   a long sequence will be shorten.
+   This function is not suitable for sequence with sub-sequence now.
+   */


This function is not suitable for sequence with sub-sequence now. --> This function is invalid for sequence having sub-sequence.

lcy-seso · 2017-04-06T09:53:49Z

python/paddle/trainer_config_helpers/layers.py

             layer_attr=None):
    """
    Get Last Timestamp Activation of a sequence.

+    If stride > 0, get last timestamp upon a stride window of sequence. 


这里的注释可以修改一下，第一句话变成下面这样。

If stride > 0, this layer slides a window whose size is determined by stride, and return the first value of the window as the output.

lcy-seso · 2017-04-06T09:57:03Z

python/paddle/trainer_config_helpers/layers.py

             layer_attr=None):
    """
    Get Last Timestamp Activation of a sequence.

+    If stride > 0, get last timestamp upon a stride window of sequence. 
+    And a long sequence will be shorten. Note that for sequence with 
+    sub-sequence, stride is default -1 now.


stride is default -1 now. --> the default value of stride is -1.

lcy-seso · 2017-04-06T09:58:26Z

python/paddle/trainer_config_helpers/layers.py

              layer_attr=None):
    """
    Get First Timestamp Activation of a sequence.

+    If stride > 0, get first timestamp upon a stride window of sequence,
+    and a long sequence will be shorten. Note that for sequence with 
+    sub-sequence, stride is default -1 now.


stride is default -1 now. --> the default value of stride is -1.

lcy-seso · 2017-04-06T10:06:21Z

paddle/gserver/layers/SequencePoolLayer.h

+  int stride_;
+  // store the start position of each stride window
+  IVectorPtr stridePositions_;
+  // Whether it is reversed sequence


whether the input sequence is reversed or not

lcy-seso · 2017-04-06T10:06:53Z

paddle/gserver/layers/SequencePoolLayer.h

@@ -42,6 +46,11 @@ class SequencePoolLayer : public Layer {
  enum SequenceLevel { kNonSeq = 0, kSeq = 1 };
  size_t newBatchSize_;
  ICpuGpuVectorPtr startPositions_;
+  int stride_;
+  // store the start position of each stride window


store the start position of each window

lcy-seso · 2017-04-06T10:07:27Z

paddle/gserver/layers/SequencePoolLayer.h

@@ -26,6 +26,10 @@ namespace paddle {
 *    Output: output size is the number of input sequences (NOT input instances)
 *    output[i] = seqlastin/average/max_{for each instance in this
 * sequence}{input[i]}
+ *    If stride_ > 0:
+ *        Check input sequence must don't have sub-sequence


Check input sequence must not have sub-sequence

lcy-seso · 2017-04-06T10:13:08Z

paddle/gserver/layers/SequenceLastInstanceLayer.cpp

@@ -25,6 +25,9 @@ namespace paddle {
 * Input: a sequence
 * If SequenceLevel = kNonseq:
 *   Output: a sequence containing only the last instance of the input sequence
+ *   If stride_ > 0:
+ *      Output: a shorten sequence containing several last instances of the
+ *              input sequence with stride window.


Output: a shorten sequence. The operation of getting last instance of a sequence is independently performed on every slice of the input sequence, that is obtained by sliding a window with the window size set to stride_.

luotao1

已按要求修改

luotao1 · 2017-04-07T02:31:05Z

paddle/gserver/layers/SequenceLastInstanceLayer.cpp

@@ -25,6 +25,9 @@ namespace paddle {
 * Input: a sequence
 * If SequenceLevel = kNonseq:
 *   Output: a sequence containing only the last instance of the input sequence
+ *   If stride_ > 0:
+ *      Output: a shorten sequence containing several last instances of the
+ *              input sequence with stride window.


luotao1 · 2017-04-07T02:32:12Z

paddle/gserver/layers/SequencePoolLayer.h

@@ -26,6 +26,10 @@ namespace paddle {
 *    Output: output size is the number of input sequences (NOT input instances)
 *    output[i] = seqlastin/average/max_{for each instance in this
 * sequence}{input[i]}
+ *    If stride_ > 0:
+ *        Check input sequence must don't have sub-sequence


luotao1 · 2017-04-07T02:32:38Z

paddle/gserver/layers/SequencePoolLayer.h

@@ -42,6 +46,11 @@ class SequencePoolLayer : public Layer {
  enum SequenceLevel { kNonSeq = 0, kSeq = 1 };
  size_t newBatchSize_;
  ICpuGpuVectorPtr startPositions_;
+  int stride_;
+  // store the start position of each stride window


luotao1 · 2017-04-07T02:33:03Z

paddle/gserver/layers/SequencePoolLayer.h

+  int stride_;
+  // store the start position of each stride window
+  IVectorPtr stridePositions_;
+  // Whether it is reversed sequence


luotao1 · 2017-04-07T02:34:59Z

paddle/parameter/Argument.cpp

+   */
+  CHECK(input.sequenceStartPositions);
+  CHECK_EQ(input.hasSubseq(), 0UL);
+  CHECK_GT(stride, 0) << "stride must larger than 0";


虽然这次外围的逻辑分支都已经检查过了，但以后调用时可能会出现外围没有做check的情况，因此想保留这些check。

luotao1 · 2017-04-07T02:53:40Z

python/paddle/trainer_config_helpers/layers.py

             layer_attr=None):
    """
    Get Last Timestamp Activation of a sequence.

+    If stride > 0, get last timestamp upon a stride window of sequence. 


luotao1 · 2017-04-07T02:54:11Z

python/paddle/trainer_config_helpers/layers.py

             layer_attr=None):
    """
    Get Last Timestamp Activation of a sequence.

+    If stride > 0, get last timestamp upon a stride window of sequence. 
+    And a long sequence will be shorten. Note that for sequence with 
+    sub-sequence, stride is default -1 now.


luotao1 · 2017-04-07T02:54:44Z

python/paddle/trainer_config_helpers/layers.py

@@ -1357,6 +1362,8 @@ def last_seq(input,
    :type name: basestring
    :param input: Input layer name.
    :type input: LayerOutput
+    :param stride: parameter of stride window.  


luotao1 · 2017-04-07T02:56:06Z

python/paddle/trainer_config_helpers/layers.py

              layer_attr=None):
    """
    Get First Timestamp Activation of a sequence.

+    If stride > 0, get first timestamp upon a stride window of sequence,


luotao1 · 2017-04-07T02:56:14Z

python/paddle/trainer_config_helpers/layers.py

              layer_attr=None):
    """
    Get First Timestamp Activation of a sequence.

+    If stride > 0, get first timestamp upon a stride window of sequence,
+    and a long sequence will be shorten. Note that for sequence with 
+    sub-sequence, stride is default -1 now.


wangkuiyi · 2017-04-10T23:20:23Z

paddle/gserver/layers/SequenceLastInstanceLayer.cpp

@@ -37,6 +42,7 @@ class SequenceLastInstanceLayer : public SequencePoolLayer {
 protected:
  MatrixPtr tmpSrc_;
  MatrixPtr tmpDest_;
+  std::vector<int> insId_;


insId_ ==> instanceIds_?

wangkuiyi · 2017-04-10T23:21:54Z

paddle/gserver/layers/SequencePoolLayer.h

@@ -42,6 +46,11 @@ class SequencePoolLayer : public Layer {
  enum SequenceLevel { kNonSeq = 0, kSeq = 1 };
  size_t newBatchSize_;
  ICpuGpuVectorPtr startPositions_;
+  int stride_;
+  // store the start position of each window


store ... window ==> Store ... window. ?

wangkuiyi · 2017-04-10T23:22:07Z

paddle/gserver/layers/SequencePoolLayer.h

+  int stride_;
+  // store the start position of each window
+  IVectorPtr stridePositions_;
+  // Whether the input sequence is reversed or not


not ==> not .?

wangkuiyi · 2017-04-10T23:24:09Z

paddle/gserver/tests/test_LayerGrad.cpp

+void testDegradeLayer(bool hasSubseq,
+                      string layer_type,
+                      string trans_type,
+                      int stride = -1) {


Default argument value 是可以用的，但是一般不用，原因在这里： https://google.github.io/styleguide/cppguide.html#Default_Arguments

wangkuiyi · 2017-04-10T23:24:44Z

paddle/parameter/Argument.cpp

+                                      size_t stride,
+                                      IVectorPtr* stridePostions,
+                                      bool reversed) {
+  /*


在.cpp文件里，用C++ style comment // 更能统一风格。

stride pooling for seqlastin and seqfirstin

9298a9e

luotao1 requested a review from lcy-seso March 24, 2017 06:46

luotao1 added 2 commits March 24, 2017 15:58

Merge branch 'develop' into stride

0291c01

adjust poolSequenceWithStride interface for average and max

cbbec59

lcy-seso requested changes Mar 28, 2017

View reviewed changes

luotao1 added 5 commits March 28, 2017 17:33

Merge branch 'develop' into stride

5262c44

Merge branch 'develop' into stride

b3b6e82

Merge branch 'develop' into stride

9a2af02

Merge branch 'develop' into stride

08d6622

add reversed poolSequenceWithStride

d369577

lcy-seso reviewed Apr 6, 2017

View reviewed changes

Merge branch 'develop' into stride

a5734f7

luotao1 commented Apr 7, 2017

View reviewed changes

refine annotation

dd61304

wangkuiyi reviewed Apr 10, 2017

View reviewed changes

luotao1 added 3 commits April 11, 2017 13:39

Merge branch 'develop' into stride

c1738e2

Merge branch 'develop' into stride

ba68ce1

update with comments

e6366e3

lcy-seso approved these changes Apr 13, 2017

View reviewed changes

lcy-seso merged commit df5a95d into PaddlePaddle:develop Apr 13, 2017

luotao1 deleted the stride branch April 13, 2017 02:44

stride pooling for seqlastin and seqfirstin #1701

stride pooling for seqlastin and seqfirstin #1701

Conversation

luotao1 commented Mar 24, 2017

luotao1 commented Mar 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Mar 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luotao1 commented Mar 31, 2017

lcy-seso left a comment

Choose a reason for hiding this comment

lcy-seso Apr 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Apr 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Apr 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luotao1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcy-seso Mar 28, 2017 •

edited

Loading

lcy-seso Apr 6, 2017 •

edited

Loading

lcy-seso Apr 6, 2017 •

edited

Loading

lcy-seso Apr 6, 2017 •

edited

Loading