EfficientFormer

[blefaudeux]

🚀 Feature

Support this micro architecture https://arxiv.org/abs/2206.01191

Motivation

• seems to work very well for low flops
• compare with other architectures supported here
• this is fun !

Pitch

Support this piecewise, same as Visual Attention for instance

Alternatives

Not doing it and stick to AlexNet

[blefaudeux]

turns out that we already support that model, it's a subset of meta formers really :) just a small block missing, adding this and testing that out

[blefaudeux]

I've an example ready, requires small changes in the hierarchical config generation, but other than that all the parts were already there.

[blefaudeux]

some layers are present, the rest would not be too hard to add but probably not a big priority, closing this to reduce noise for the xformers team