Does xFormers support weight tying?

[erip]

❓ Questions and Help

Tying weights between encoder and decoder embedding layers is often useful for convergence and task performance. Is there a mechanism for sharing weights between the encoder and decoder?

[blefaudeux]

Good question, thanks @erip ! As of now there are basically two sides of xformers, one is a part zoo (at different altitudes), the other one is around the factories, which combine these parts programatically.

If you use the first path (build the model yourself given parts), then tying weights is certainly possible, as you would do with any other model. It's not supported with the factory right now though, but could certainly be done around that. Thoughts all ?

[erip]

Closed!