performer

Simple implementaion of pytorch Performer.

Model

Performer approximates kernel using random feature map. The kernel expects to replace Transformer's Dot-Product Self attention.

Softmax Kernel

kernel_transformation=softmax_kernel_transformation.

Relu Kernel

kernel_transformation=relu_kernel_transformation

Test & Example

Language Model

Pertrain

pretrain masked language model.

• Pretrain file: /example/train_mlm.py.
• Config file: /example/config.json

Usage

① prepare dataset and vocab you want to train
② check configuration in config.json
③ run /example/train_mlm.py

Finetuing

TODO

• Performer performance test
• Write test example
• apply to language model
• evaluate language model

References

• Performer's Fast Attention (FAVOR+) Module
• Google Performer Blog
• teddykoker Performer