create learning rate variable for every parameter

[QiJune]

Fix #5525

[abhinavarora]

Are we sure that this will be multiplied? Is there a reference for this implementation or is it taken from the old Paddle code?

[QiJune]

@lcy-seso @reyoung Could you please confirm this? Is each parameter's learning rate set relatively, and calculated by multiplying global learning rate?

[lcy-seso]

This is the way old PaddlePaddle uses. The learning rate of one parameter is determined by multiplying the global learning rate to the layerwise set learning rate (if set).

[abhinavarora]

@lcy-seso Do you know how this works together with learning rate scheduler? Does the learning rate scheduler only work for the global learning rate?

[lcy-seso]

From the codes, it seems that the learning rate schedule only works for global learning rate:

In old PaddlePaddle it seems that the learning rate scheduler just calculates a scale factor to decrease the learning rate.

• calcLearningRate adjusts the learning rate:
  https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/parameter/FirstOrderOptimizer.h#L32 https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/parameter/ParameterOptimizer.h#L49
• Here takes Adam for example, how the final learning rate in update calculates:
  https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/parameter/FirstOrderOptimizer.cpp#L272

[abhinavarora]

LGTM!