BilinearUpsamplingFiller used with DeconvolutionLayer for upsampling

[tnarihi]

This is intended to be used in DeconvolutionLayer acting like UpsamplingLayer which is not implemented explicitly. You can upsample a feature map by any integer factor using the following proto.

layer {
  name: "upsample", type: "Deconvolution"
  bottom: "{{bottom_name}}" top: "{{top_name}}"
  convolution_param {
    kernel_size: {{2 * factor - factor % 2}} stride: {{factor}}
    num_output: {{channels_in}} group: {{channels_in}}
    pad: {{ceil((factor - 1) / 2.)}}
    weight_filler: { type: "bilinear_upsampling" } bias_term: false
  }
  param { lr_mult: 0 decay_mult: 0 }
}

Replace {{}} with your values. Note that the learning rate and the weight decay are set to 0 in order to keep coefficient values of bilinear interpolation unchanged during training. If you apply this to an image, this operation is equivalent to the following call in Python with Scikit.Image.

out = skimage.transform.rescale(img, factor, mode='constant', cval=0)

[tnarihi]

Here is an ipython notebook example.

[weiliu89]

I believe this is wrong. According to https://gist.githubusercontent.com/shelhamer/80667189b218ad570e82/raw/938f470ad91072929cbb9f6c739dc34488ca7d03/solve.py, it uses net.params[l][0].data[range(m), range(k), :, :] = filt. This doesn't fill filt in each channel in each kernel. It only fills the ith channel in the ith kernel, all others are 0.

Bilinear interpolation is doing channels-wise. It could be easy to fix it.

[tnarihi]

Thanks for commenting @weiliu89, but this is not wrong. Please note that I assume you specify group: {{num_in}} in convolution_param. That results channel-wise convolution, and it is more efficient for both memory and computation. We have conovolunal weights with size of (num_in, 1, kernel_size, kernel_size) instead of (num_in, num_in, kernel_size, kernel_size). I don't do diagonal convolution explicitly. I updated my IPython notebook example to add a RGB image example other than gray one. This example shows you this implementation gives you result images that are equivalent to results from a well-known imaging library in Python (Scikits.Image).

EDIT: (num_in, 1, kernel_size, kernel_size) is equivalent to (num_out, 1, kernel_size, kernel_size) here. Actually, this num_out/num_in channels for weights are still redundant for channel-wise interpolation. If we implement Convolution/Deconvolution with weight shading over channels, this redundancy could be removed, but I think this will make implmentation more complicated and unmaintenanciable.

[weiliu89]

@tnarihi Thanks for pointing out my mistake. Yours is indeed more efficient than the original python code. I thought group is not important in the setting and thus ignored it. Probably it is better to emphasize this more in the code, otherwise the filler is doing completely the wrong thing.

[shelhamer]

@tnarihi @weiliu89 setting group is more obvious and makes use of less memory. The posted semantic segmentation FCNs will likely be switched to group deconvolution at some point for this reason. However the current Python filler is correct:

[range(m), range(k), :, :]

indexes each (idx, idx) pair for channelwise interpolation while all "off-diagonal" filter coefficients are left to their zero initialization.

[shelhamer]

@tnarihi thanks for the C++ filler so interpolation can be defined instead of needing initialization by net surgery. I might call this bilinear instead of bilinear_upsampling since its the stride that results in upsampling while the bilinear kernel alone is interpolation -- although to be fair the main point of this kernel is for upsampling.

@longjon thoughts?

[tnarihi]

@weiliu89 Your point is right. I updated doc.

@shelhamer Thanks for your attention to this PR. Yeah, I once called it bilinear, but as you pointed, it is mainly dedicated for upsampling and I couldn't find other purpose of this filler besides upsamling, so I now call it bilinear_upsampling for clarity. Anyway, I don't have any strong preference about the name. Either is fine for me.

[longjon]

@shelhamer @tnarihi, I agree that "upsampling" is a bit overly specific, e.g., you can use this to initialize a convolution layer and do _down_sampling. Either name is okay with me though. Also consider bilinear_filter.

[tnarihi]

@longjon I am still unclear how we could create a downsampling layer which is equivalent to common image downsampling implementation by using this filler though, bilinear_filter also sounds good to me.
@shelhamer @longjon Please choose what you prefer. Is bilinear simpler? I don't have any preference among three now. Once naming is fixed, I think this PR is ready to merge.

[shelhamer]

Merged in 805a995. @tnarihi I changed the filler name to bilinear but kept your commit details and authorship. Thanks for the filler!

[tnarihi]

Thanks for merging, @shelhamer!

[xyy19920105]

how to use this to downsampling with bilinear?

[thigi]

This does not work for 5D blobs. Does anyone know a workaround to use the Deconvolution-Layer with bilinear filler for 5D blobs? 5056

[zmlmanly]

Hi, I have a problem that the result of tensorflow.image.resize_bilinear() is different from Deconvolution-Layer with bilinear in Caffe, could you give me a help? Thank you very much.