Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/layernorm #36

Merged
merged 70 commits into from
Sep 15, 2022
Merged

Feat/layernorm #36

merged 70 commits into from
Sep 15, 2022

Commits on Aug 11, 2022

  1. feat: add attention

    @gaetansnl
    gaetansnl committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    a45997c View commit details
    Browse the repository at this point in the history

  2. fix: use tuple in triton

    @gaetansnl
    gaetansnl committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    0745dd2 View commit details
    Browse the repository at this point in the history

  3. docs: attention

    @gaetansnl
    gaetansnl committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    d06fb8a View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2022

  1. feat: add torchdynamo end to end fusion

    @gaetansnl
    gaetansnl committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    2c4f4bf View commit details
    Browse the repository at this point in the history

  2. feat: causal masked attention

    @gaetansnl
    gaetansnl committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    0644ae6 View commit details
    Browse the repository at this point in the history

  3. feat: benchmark dynamo backends

    @gaetansnl
    gaetansnl committed Aug 12, 2022
    Configuration menu
    Copy the full SHA
    e6e359b View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2022

  1. Merge branch 'main' into feat/torchdynamo-fused

    @gaetansnl
    gaetansnl authored Aug 16, 2022
    Configuration menu
    Copy the full SHA
    2ba239c View commit details
    Browse the repository at this point in the history

  2. fix: renaming

    @gaetansnl
    gaetansnl committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    189ca47 View commit details
    Browse the repository at this point in the history

  3. feat: add support for arbitrary stride

    @gaetansnl
    gaetansnl committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    0e13b3c View commit details
    Browse the repository at this point in the history

  4. fix: move output outside kernel

    @gaetansnl
    gaetansnl committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    a0f880d View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2022

  1. feat: module replacement example

    @gaetansnl
    gaetansnl committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    a6dcc70 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2022

  1. fix: missing benchmark for masked attention

    @gaetansnl
    gaetansnl committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    b142c1b View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2022

  1. feat: add pattern and fix fx bug

    @gaetansnl
    gaetansnl committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    6aceec7 View commit details
    Browse the repository at this point in the history

  2. fix: refactoring

    @gaetansnl
    gaetansnl committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    a4f920f View commit details
    Browse the repository at this point in the history

  3. feat: add layer_norm

    @gaetansnl
    gaetansnl committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    5157a8b View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2022

  1. fix: show speedup in benchmark display

    @gaetansnl
    gaetansnl committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    90f989c View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2022

  1. fix: update torchdyname and matcher

    @gaetansnl
    gaetansnl committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    f7a0ffa View commit details
    Browse the repository at this point in the history

  2. fix: update matcher

    @gaetansnl
    gaetansnl committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    bf5ecfa View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2022

  1. fix: cuda graph

    @gaetansnl
    gaetansnl committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    ce73e78 View commit details
    Browse the repository at this point in the history

  2. fix: small seq_length

    @gaetansnl
    gaetansnl committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    9590e78 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2022

  1. feat: viz server

    @gaetansnl
    gaetansnl committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    3534cb5 View commit details
    Browse the repository at this point in the history

  2. fix: bug in matcher and add complete graph report

    @gaetansnl
    gaetansnl committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    26e5720 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2022

  1. fix: compatibility with pytorch stable

    @gaetansnl
    gaetansnl committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    1576ef5 View commit details
    Browse the repository at this point in the history

  2. fix: add credit, rename variables, add doc

    @gaetansnl
    gaetansnl committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    f5729df View commit details
    Browse the repository at this point in the history

  3. fix: add test for shape change

    @gaetansnl
    gaetansnl committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    be04b88 View commit details
    Browse the repository at this point in the history

  4. fix: attention renaming

    @gaetansnl
    gaetansnl committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    7e2cb62 View commit details
    Browse the repository at this point in the history

  5. fix: add license

    @gaetansnl
    gaetansnl committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    672ea04 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2022

  1. feat: add stride management on linear layer + replace cuda graph + ne… 

    …w benchmarks
    @pommedeterresautee
    pommedeterresautee committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    f5c1ae2 View commit details
    Browse the repository at this point in the history

  2. feat: remove M, N masking

    @pommedeterresautee
    pommedeterresautee committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    c8bb059 View commit details
    Browse the repository at this point in the history

  3. feat: improve autotune

    @pommedeterresautee
    pommedeterresautee committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    3998bc9 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2022

  1. fix: command line

    @pommedeterresautee
    pommedeterresautee committed Sep 5, 2022
    Configuration menu
    Copy the full SHA
    dbf7c93 View commit details
    Browse the repository at this point in the history

  2. fix: refactoring benchmarks (fix cuda graphs API, tests as dict, add … 

    …TODO, smarter shapes, etc.)
    @pommedeterresautee
    pommedeterresautee committed Sep 5, 2022
    Configuration menu
    Copy the full SHA
    95fac33 View commit details
    Browse the repository at this point in the history

  3. fix: linear layer is now working (remove trick on max contiguous), re… 

    …move unneeded code
    @pommedeterresautee
    pommedeterresautee committed Sep 5, 2022
    Configuration menu
    Copy the full SHA
    368f3e5 View commit details
    Browse the repository at this point in the history

  4. fix: refactoring of the linear benchmark with sizes similar to bert o… 

    …nes.
    @pommedeterresautee
    pommedeterresautee committed Sep 5, 2022
    Configuration menu
    Copy the full SHA
    804335b View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2022

  1. feat: add more instructions

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    e93632b View commit details
    Browse the repository at this point in the history

  2. fix: restore cuda graphs warmup (and remove todo)

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    a392150 View commit details
    Browse the repository at this point in the history

  3. feat: change linear implementation

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    efdabc9 View commit details
    Browse the repository at this point in the history

  4. feat: change linear implementation

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    d464f6b View commit details
    Browse the repository at this point in the history

  5. feat: replace GELU + layernorm by more precise version, fix all preci… 

    …sion issues in unit tests. refactor tests
    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    e9d6b04 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/main' into fix/refactoring_bench… 

    …marks
    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    a76b710 View commit details
    Browse the repository at this point in the history

  7. fix: plural

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    b91456c View commit details
    Browse the repository at this point in the history

  8. fix: some doc

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    d20619c View commit details
    Browse the repository at this point in the history

  9. fix: layer norm unit test

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    87cf360 View commit details
    Browse the repository at this point in the history

  10. feat: add cuda graph layer norm unit test

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    b112b4b View commit details
    Browse the repository at this point in the history

  11. fix: remove TODO

    @pommedeterresautee
    pommedeterresautee committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    479254f View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2022

  1. feat: add split K support

    @pommedeterresautee
    pommedeterresautee committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    afc8d41 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2022

  1. Merge branch 'main' into feat/tools

    @pommedeterresautee
    pommedeterresautee authored Sep 8, 2022
    Configuration menu
    Copy the full SHA
    106cb24 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feat/tools' into fix/refactoring_benchmarks

    @pommedeterresautee
    pommedeterresautee authored Sep 8, 2022
    Configuration menu
    Copy the full SHA
    ed6d1a1 View commit details
    Browse the repository at this point in the history

  3. fix: remove split k

    @pommedeterresautee
    pommedeterresautee committed Sep 8, 2022
    Configuration menu
    Copy the full SHA
    df3daa8 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/fix/refactoring_benchmarks' into… 

    … fix/refactoring_benchmarks
    @pommedeterresautee
    pommedeterresautee committed Sep 8, 2022
    Configuration menu
    Copy the full SHA
    cf8e7d0 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2022

  1. feat: refactoring layernorm test

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    7d91384 View commit details
    Browse the repository at this point in the history

  2. fix: add back bias and activation tests + refactoring

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    f2130a3 View commit details
    Browse the repository at this point in the history

  3. feat: make test display understandable

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    119e776 View commit details
    Browse the repository at this point in the history

  4. fix: remove benchmark display

    @gaetansnl
    gaetansnl committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    38ee01b View commit details
    Browse the repository at this point in the history

  5. fix: remove benchmark display

    @gaetansnl
    gaetansnl committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    94dda21 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'feat/tools' into fix/refactoring_benchmarks

    @pommedeterresautee
    pommedeterresautee authored Sep 9, 2022
    Configuration menu
    Copy the full SHA
    4bffeef View commit details
    Browse the repository at this point in the history

  7. fix: avoid OOM on reference implementation

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    8f91f79 View commit details
    Browse the repository at this point in the history

  8. fix: get input

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    61cbc1f View commit details
    Browse the repository at this point in the history

  9. Merge remote-tracking branch 'origin/fix/refactoring_benchmarks' into… 

    … fix/refactoring_benchmarks
    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    f39219c View commit details
    Browse the repository at this point in the history

  10. fix: remove some OOM test for reference implementation

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    10703a9 View commit details
    Browse the repository at this point in the history

  11. fix: add tests

    @pommedeterresautee
    pommedeterresautee committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    dabf536 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2022

  1. feat: new layernorm single pass variance computation implementation

    @pommedeterresautee
    pommedeterresautee committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    bd83caa View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2022

  1. Merge branch 'main' into fix/refactoring_benchmarks 

    # Conflicts:
#	README.md
#	implementations/activation_func.py
#	implementations/attention_masked_original.py
#	implementations/layer_norm.py
#	implementations/linear_layer.py
#	optimizer/dynamo_backend.py
#	optimizer/layer_norm.py
#	optimizer/linear.py
#	test/models/bert.py
#	test/test_attention.py
#	test/test_batched_matmul.py
#	test/test_layer_norm.py
#	test/test_linear_layer.py
#	test/test_torchdynamo_bert.py
    @pommedeterresautee
    pommedeterresautee committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    85ae2d8 View commit details
    Browse the repository at this point in the history

  2. fix: rename variables

    @pommedeterresautee
    pommedeterresautee committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    f3564b0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'fix/refactoring_benchmarks' into feat/layernorm

    @pommedeterresautee
    pommedeterresautee committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    7b06621 View commit details
    Browse the repository at this point in the history

  4. feat: add naive implem of layernorm

    @pommedeterresautee
    pommedeterresautee committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    c2b8ade View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into feat/layernorm 

    # Conflicts:
#	implementations/layer_norm.py
#	test/test_layer_norm.py
#	test/test_linear_layer.py
#	test/test_torchdynamo_bert.py
    @pommedeterresautee
    pommedeterresautee committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    cca52e8 View commit details
    Browse the repository at this point in the history

  6. fix: store mean/var in layernorm (for bw pass)

    @pommedeterresautee
    pommedeterresautee committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    1100ef6 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2022

  1. fix: following review comments

    @pommedeterresautee
    pommedeterresautee committed Sep 14, 2022
    Configuration menu
    Copy the full SHA
    2cedbfa View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2022

  1. fix: add manual seed

    @pommedeterresautee
    pommedeterresautee committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    7f3aefa View commit details
    Browse the repository at this point in the history