Code

Reformer#3351

Code

Merged

patrickvonplaten merged 162 commits intomasterhuggingface/transformers:masterfrom reformer_add_modelhuggingface/transformers:reformer_add_modelCopy head branch name to clipboardMay 7, 2020

+3,608-23

Commits

Commits on May 7, 2020

• first copy & past commit from Bert and morgans LSH code

  patrickvonplaten

  committed
  ee0ce08

• add easy way to compare to trax original code

  patrickvonplaten

  committed
  3259115

• translate most of function

  patrickvonplaten

  committed
  25d162e

• make trax lsh self attention deterministic with numpy seed + copy paste code

  patrickvonplaten

  committed
  dc07c08

• add same config

  patrickvonplaten

  committed
  09d4230

• add same config

  patrickvonplaten

  committed
  9386450

• fix merge conflicts

  patrickvonplaten

  committed
  3fb1182

• make layer init work

  patrickvonplaten

  committed
  ebb9d3f

• implemented hash_vectors function for lsh attention

  patrickvonplaten

  committed
  c910e03

• continue reformer translation

  patrickvonplaten

  committed
  b956933

• hf LSHSelfAttentionLayer gives same output as trax layer

  patrickvonplaten

  committed
  a449c2e

• refactor code

  patrickvonplaten

  committed
  35491d8

• refactor code

  patrickvonplaten

  committed
  c9a0919

• refactor code

  patrickvonplaten

  committed
  4ef4739

• refactor

  patrickvonplaten

  committed
  f580074

• refactor + add reformer config

  patrickvonplaten

  committed
  4176564

• delete bogus file

  patrickvonplaten

  committed
  8e02fe7

• split reformer attention layer into two layers

  patrickvonplaten

  committed
  2af8377

• fix merge conflicts

  patrickvonplaten

  committed
  6fe9478

• save intermediate step

  patrickvonplaten

  committed
  9e6e1af

• save intermediate step

  patrickvonplaten

  committed
  1855074

• make test work

  patrickvonplaten

  committed
  1a4e61a

• add complete reformer block layer

  patrickvonplaten

  committed
  da6bfe4

• finish reformer layer

  patrickvonplaten

  committed
  2825d24

• implement causal and self mask

  patrickvonplaten

  committed
  45e6635

• clean reformer test and refactor code

  patrickvonplaten

  committed
  b5ed5d4

• update init

  patrickvonplaten

  committed
  ddb2f09

• fix device for GPU

  patrickvonplaten

  committed
  cbb5ab9

• fix chunk length init for tests

  patrickvonplaten

  committed
  f17fd5b

• include morgans optimization

  patrickvonplaten

  committed
  eca8cce

• improve memory a bit

  patrickvonplaten

  committed
  db2ebb1

• improve comment

  patrickvonplaten

  committed
  04aa067

• factorize num_buckets

  patrickvonplaten

  committed
  4aec75e

• better testing parameters

  patrickvonplaten

  committed
  d030e39

• make whole model work

  patrickvonplaten

  committed
  d318089

• make lm model work

  patrickvonplaten

  committed
  4f0b114

• add t5 copy paste tokenizer

  patrickvonplaten

  committed
  6c8bad6

• add chunking feed forward

  patrickvonplaten

  committed
  b71ef16

• clean config

  patrickvonplaten

  committed
  99427c6

• add improved assert statements

  patrickvonplaten

  committed
  4ffa925

• make tokenizer work

  patrickvonplaten

  committed
  b116e3c

• improve test

  patrickvonplaten

  committed
  79a0bab

• correct typo

  patrickvonplaten

  committed
  aceb586

• extend config

  patrickvonplaten

  committed
  a4814bd

• add complexer test

  patrickvonplaten

  committed
  5eeeb25

• add new axial position embeddings

  patrickvonplaten

  committed
  0ee5db4

• add local block attention layer

  patrickvonplaten

  committed
  938aa8b

• clean tests

  patrickvonplaten

  committed
  4d7c23b

• refactor

  patrickvonplaten

  committed
  50276de

• better testing

  patrickvonplaten

  committed
  37a2b00

• save intermediate progress

  patrickvonplaten

  committed
  07c0c72

• clean test file

  patrickvonplaten

  committed
  060a691

• make shorter input length work for model

  patrickvonplaten

  committed
  ace301f

• allow variable input length

  patrickvonplaten

  committed
  80d18db

• refactor

  patrickvonplaten

  committed
  86f4ac4

• make forward pass for pretrained model work

  patrickvonplaten

  committed
  e571849

• add generation possibility

  patrickvonplaten

  committed
  d5e1363

• finish dropout and init

  patrickvonplaten

  committed
  562d530

• make style

  patrickvonplaten

  committed
  c98eafe

• refactor

  patrickvonplaten

  committed
  9c9fab9

• add first version of RevNet Layers

  patrickvonplaten

  committed
  a188a39

• make forward pass work and add convert file

  patrickvonplaten

  committed
  8047573

• make uploaded model forward pass work

  patrickvonplaten

  committed
  31a596b

• make uploaded model forward pass work

  patrickvonplaten

  committed
  bae0700

• refactor code

  patrickvonplaten

  committed
  831dcec

• add namedtuples and cache buckets

  patrickvonplaten

  committed
  57ee09c

• correct head masks

  patrickvonplaten

  committed
  2d23fad

• refactor

  patrickvonplaten

  committed
  0c35bbf

• made reformer more flexible

  patrickvonplaten

  committed
  232463e

• make style

  patrickvonplaten

  committed
  2648a94

• remove set max length

  patrickvonplaten

  committed
  902408b

• add attention masks

  patrickvonplaten

  committed
  8ed63ab

• fix up tests

  patrickvonplaten

  committed
  513bb43

• fix conflict

  patrickvonplaten

  committed
  db60c23

• fix lsh attention mask

  patrickvonplaten

  committed
  9f359af

• make random seed optional for the moment

  patrickvonplaten

  committed
  48097a0

• improve memory in reformer

  patrickvonplaten

  committed
  650e00c

• add tests

  patrickvonplaten

  committed
  ccba9ac

• make style

  patrickvonplaten

  committed
  f83721e

• make sure masks work correctly

  patrickvonplaten

  committed
  125c86d

• detach gradients

  patrickvonplaten

  committed
  2beda9c

• save intermediate

  patrickvonplaten

  committed
  12e35e1

• correct backprob through gather

  patrickvonplaten

  committed
  8b058e2

• make style

  patrickvonplaten

  committed
  69258b8

• change back num hashes

  patrickvonplaten

  committed
  44c3a7c

• rename to labels

  patrickvonplaten

  committed
  48fff07

• fix rotation shape

  patrickvonplaten

  committed
  55842be

• fix detach

  patrickvonplaten

  committed
  71426c0

• update

  patrickvonplaten

  committed
  dfbcf8f

• fix trainer

  patrickvonplaten

  committed
  0ea564c

• fix backward dropout

  patrickvonplaten

  committed
  af3456c

• make reformer more flexible

  patrickvonplaten

  committed
  002f19c

• fix

  patrickvonplaten

  committed
  7de3f4f

• fix

  patrickvonplaten

  committed
  6111bd5

• add tests for fixed seed in reformer layer

  patrickvonplaten

  committed
  0c75149

• fix trainer typo

  patrickvonplaten

  committed
  7a03bc7

• fix typo in activations

  patrickvonplaten

  committed
  37943f3

• add fp16 tests

  patrickvonplaten

  committed
  0f751f5

• add fp16 training

  patrickvonplaten

  committed
  8df5dcd

• support fp16

  patrickvonplaten

  committed
  51426b5

• correct gradient bug in reformer

  patrickvonplaten

  committed
  b37fd3b

• add fast gelu

  patrickvonplaten

  committed
  e3e05ef

• re-add dropout for embedding dropout

  patrickvonplaten

  committed
  c3e32b4

• better naming

  patrickvonplaten

  committed
  52ee5ed

• better naming

  patrickvonplaten

  committed
  ece19ee

• renaming

  patrickvonplaten

  committed
  e661832

• finalize test branch

  patrickvonplaten

  committed
  f1a6355

• finalize tests

  patrickvonplaten

  committed
  ea1126e

• add more tests

  patrickvonplaten

  committed
  d4bc3c6

• finish tests

  patrickvonplaten

  committed
  94086ac

• fix

  patrickvonplaten

  committed
  01f4074

• fix type trainer

  patrickvonplaten

  committed
  9dafbc2

• fix fp16 tests

  patrickvonplaten

  committed
  de08a57

• fix tests

  patrickvonplaten

  committed
  aa570dc

• fix tests

  patrickvonplaten

  committed
  a681d19

• fix tests

  patrickvonplaten

  committed
  320c045

• fix issue with dropout

  patrickvonplaten

  committed
  482c6cd

• fix dropout seeds

  patrickvonplaten

  committed
  d7905dd

• correct random seed on gpu

  patrickvonplaten

  committed
  764e06e

• finalize random seed for dropout

  patrickvonplaten

  committed
  a3e0f59

• finalize random seed for dropout

  patrickvonplaten

  committed
  c48f88a

• remove duplicate line

  patrickvonplaten

  committed
  ce87cb6

• correct half precision bug

  patrickvonplaten

  committed
  d418dd0

• make style

  patrickvonplaten

  committed
  3248e67

• refactor

  patrickvonplaten

  committed
  6fe0648

• refactor

  patrickvonplaten

  committed
  c3031b8

• docstring

  patrickvonplaten

  committed
  6c2be30

• remove sinusoidal position encodings for reformer

  patrickvonplaten

  committed
  3d266fb

• move chunking to modeling_utils

  patrickvonplaten

  committed
  1be343f

• make style

  patrickvonplaten

  committed
  a10eb2e

• clean config

  patrickvonplaten

  committed
  f31b570

• make style

  patrickvonplaten

  committed
  b2a660f

• fix tests

  patrickvonplaten

  committed
  dfc1f64

• fix auto tests

  patrickvonplaten

  committed
  2e95c17

• pretrained models

  patrickvonplaten

  committed
  b95f6ae

• fix docstring

  patrickvonplaten

  committed
  a6f69cb

• update conversion file

  patrickvonplaten

  committed
  59868f3

• Update pretrained_models.rst

  patrickvonplaten

  committed
  a81c3e0

• fix rst

  patrickvonplaten

  committed
  c0ddf94

• fix rst

  patrickvonplaten

  committed
  62a8eb0

• update copyright

  patrickvonplaten

  committed
  47e5fc8

• fix test path

  patrickvonplaten

  committed
  b6576c8

• fix test path

  patrickvonplaten

  committed
  a111720

• fix small issue in test

  patrickvonplaten

  committed
  ff5e783

• include reformer in generation tests

  patrickvonplaten

  committed
  f7f949b

• add docs for axial position encoding

  patrickvonplaten

  committed
  91472b8

• finish docs

  patrickvonplaten

  committed
  6ed2fa8

• Update convert_reformer_trax_checkpoint_to_pytorch.py

  patrickvonplaten

  committed
  963bb5e

• remove isort

  patrickvonplaten

  committed
  425b185

• include sams comments

  patrickvonplaten

  committed
  3336d8f

• remove wrong comment in utils

  patrickvonplaten

  committed
  54eb629

• correct typos

  patrickvonplaten

  committed
  e4e1e59

• fix typo

  patrickvonplaten

  committed
  5f5c89b

• Update reformer.rst

  patrickvonplaten

  committed
  7fdf16b

• applied morgans optimization

  patrickvonplaten

  committed
  7ccec6a

• make style

  patrickvonplaten

  committed
  3978afa

• make gpu compatible

  patrickvonplaten

  committed
  01b1006

• remove bogus file

  patrickvonplaten

  committed
  e983a69

• big test refactor

  patrickvonplaten

  committed
  9ce32f0

• add example for chunking

  patrickvonplaten

  committed
  67f02c0

• fix typo

  patrickvonplaten

  committed
  4e7252a

• add to README

  patrickvonplaten

  committed
  ca4dab3