What param init defaults should we use

[albertz]

Some options:

• Follow whatever RETURNN does currently
• Follow TensorFlow / Keras
• Follow PyTorch. Although the current defaults are not optimal and there is ongoing discussion to change them, how to change then, and what the new defaults should be. See Update weight initialisations to current best practices pytorch/pytorch#18182 and Update weight init to general use cases pytorch/pytorch#41638.
• Figure out the best current practice. Although this is maybe not so clear.

I collected some common options here: https://github.com/rwth-i6/returnn/wiki/Parameter-initialization

[albertz]

From other posts I read so far, it is claimed that Kaiming He (scale=2., mode="fan_in", distribution="normal") is currently the best practice.

However, I have not really seen that being used much in public setups. TF/Keras and other Google code often uses Glorot uniform. And this is also the default for the RETURNN (TensorFlow) LinearLayer.

[albertz]

From some Twitter discussion (via @lucasb-eyer):

Kaiming for CNNs with ReLUs, Xavier as a starting point for everything else.

I guess Xavier Glorot (scale=1.0, mode="fan_avg", distribution="uniform") is a good default choice for any tensors >= 2 dimensions, and just 0. for tensors <= 1 dimension.

[albertz]

Should our LSTM overwrite this?