SwiGLU further optimization in MLP bw #502

danthe3rd · 2022-11-02T15:18:31Z

Stack from ghstack (oldest at bottom):

*PERFORMANCE A100

operandfused_all <- THIS PR
SwiGLUPackedFusedOp <- previous pr
[--------------------------------------- swiglu_bw ---------------------------------------]
                                     |  operandfused_all  |  eager   |  SwiGLUPackedFusedOp
1 threads: --------------------------------------------------------------------------------
      b16    B=9456, I=1536, H=4096  |       2227.6       |  2708.3  |         2341.6      
      f16    B=9456, I=1536, H=4096  |       2337.5       |  2705.8  |         2339.1      
      f16.ac B=9456, I=1536, H=4096  |       2630.5       |  2998.5  |         2806.6      
      b16    B=4440, I=1536, H=4096  |       1177.9       |  1424.5  |         1246.4      
      f16    B=4440, I=1536, H=4096  |       1205.1       |  1418.8  |         1240.6      
      f16.ac B=4440, I=1536, H=4096  |       1409.0       |  1637.4  |         1541.7      
      b16    B=4728, I=1536, H=4096  |       1238.6       |  1493.5  |         1397.5      
      f16    B=4728, I=1536, H=4096  |       1274.8       |  1488.2  |         1392.7      
      f16.ac B=4728, I=1536, H=4096  |       1478.2       |  1710.3  |         1512.9      
      b16    B=4728, I=1536, H=1024  |        461.0       |   518.7  |          487.7      
      f16    B=4728, I=1536, H=1024  |        438.2       |   498.3  |          479.8      
      f16.ac B=4728, I=1536, H=1024  |        560.9       |   623.2  |          601.4      

Times are in microseconds (us).

[ghstack-poisoned]

ghstack-source-id: 12b56c35da4e6fb208bdc51ee146601dc5c35517 Pull Request resolved: #502

***PERFORMANCE A100** ``` operandfused_all <- THIS PR SwiGLUPackedFusedOp <- previous pr [--------------------------------------- swiglu_bw ---------------------------------------] | operandfused_all | eager | SwiGLUPackedFusedOp 1 threads: -------------------------------------------------------------------------------- b16 B=9456, I=1536, H=4096 | 2227.6 | 2708.3 | 2341.6 f16 B=9456, I=1536, H=4096 | 2337.5 | 2705.8 | 2339.1 f16.ac B=9456, I=1536, H=4096 | 2630.5 | 2998.5 | 2806.6 b16 B=4440, I=1536, H=4096 | 1177.9 | 1424.5 | 1246.4 f16 B=4440, I=1536, H=4096 | 1205.1 | 1418.8 | 1240.6 f16.ac B=4440, I=1536, H=4096 | 1409.0 | 1637.4 | 1541.7 b16 B=4728, I=1536, H=4096 | 1238.6 | 1493.5 | 1397.5 f16 B=4728, I=1536, H=4096 | 1274.8 | 1488.2 | 1392.7 f16.ac B=4728, I=1536, H=4096 | 1478.2 | 1710.3 | 1512.9 b16 B=4728, I=1536, H=1024 | 461.0 | 518.7 | 487.7 f16 B=4728, I=1536, H=1024 | 438.2 | 498.3 | 479.8 f16.ac B=4728, I=1536, H=1024 | 560.9 | 623.2 | 601.4 Times are in microseconds (us). ``` [ghstack-poisoned]

codecov-commenter · 2022-11-04T11:19:27Z

Codecov Report

Base: 88.37% // Head: 88.37% // No change to project coverage 👍

Coverage data is based on head (4a444b0) compared to base (4a444b0).
Patch has no changes to coverable lines.

❗ Current head 4a444b0 differs from pull request most recent head 1227f05. Consider uploading reports for the commit 1227f05 to get more accurate results

Additional details and impacted files

@@                  Coverage Diff                  @@
##           gh/danthe3rd/57/base     #502   +/-   ##
=====================================================
  Coverage                 88.37%   88.37%           
=====================================================
  Files                        80       80           
  Lines                      4798     4798           
=====================================================
  Hits                       4240     4240           
  Misses                      558      558

Flag	Coverage Δ
Python	`88.37% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

***PERFORMANCE A100** ``` operandfused_all <- THIS PR SwiGLUPackedFusedOp <- previous pr [--------------------------------------- swiglu_bw ---------------------------------------] | operandfused_all | eager | SwiGLUPackedFusedOp 1 threads: -------------------------------------------------------------------------------- b16 B=9456, I=1536, H=4096 | 2227.6 | 2708.3 | 2341.6 f16 B=9456, I=1536, H=4096 | 2337.5 | 2705.8 | 2339.1 f16.ac B=9456, I=1536, H=4096 | 2630.5 | 2998.5 | 2806.6 b16 B=4440, I=1536, H=4096 | 1177.9 | 1424.5 | 1246.4 f16 B=4440, I=1536, H=4096 | 1205.1 | 1418.8 | 1240.6 f16.ac B=4440, I=1536, H=4096 | 1409.0 | 1637.4 | 1541.7 b16 B=4728, I=1536, H=4096 | 1238.6 | 1493.5 | 1397.5 f16 B=4728, I=1536, H=4096 | 1274.8 | 1488.2 | 1392.7 f16.ac B=4728, I=1536, H=4096 | 1478.2 | 1710.3 | 1512.9 b16 B=4728, I=1536, H=1024 | 461.0 | 518.7 | 487.7 f16 B=4728, I=1536, H=1024 | 438.2 | 498.3 | 479.8 f16.ac B=4728, I=1536, H=1024 | 560.9 | 623.2 | 601.4 Times are in microseconds (us). ``` [ghstack-poisoned]

fmassa

Thanks!

.gitmodules

xformers/components/swiglu/cuda/gemm_fused_operand_sum.cu

***PERFORMANCE A100** ``` operandfused_all <- THIS PR SwiGLUPackedFusedOp <- previous pr [--------------------------------------- swiglu_bw ---------------------------------------] | operandfused_all | eager | SwiGLUPackedFusedOp 1 threads: -------------------------------------------------------------------------------- b16 B=9456, I=1536, H=4096 | 2227.6 | 2708.3 | 2341.6 f16 B=9456, I=1536, H=4096 | 2337.5 | 2705.8 | 2339.1 f16.ac B=9456, I=1536, H=4096 | 2630.5 | 2998.5 | 2806.6 b16 B=4440, I=1536, H=4096 | 1177.9 | 1424.5 | 1246.4 f16 B=4440, I=1536, H=4096 | 1205.1 | 1418.8 | 1240.6 f16.ac B=4440, I=1536, H=4096 | 1409.0 | 1637.4 | 1541.7 b16 B=4728, I=1536, H=4096 | 1238.6 | 1493.5 | 1397.5 f16 B=4728, I=1536, H=4096 | 1274.8 | 1488.2 | 1392.7 f16.ac B=4728, I=1536, H=4096 | 1478.2 | 1710.3 | 1512.9 b16 B=4728, I=1536, H=1024 | 461.0 | 518.7 | 487.7 f16 B=4728, I=1536, H=1024 | 438.2 | 498.3 | 479.8 f16.ac B=4728, I=1536, H=1024 | 560.9 | 623.2 | 601.4 Times are in microseconds (us). ``` [ghstack-poisoned]

ghstack-source-id: b1bead3ade9c469dc859d3576b8834488845e57c Pull Request resolved: #502

Saves memory and is slightly faster

SwiGLU further optimization in MLP bw

b5a3065

[ghstack-poisoned]

This was referenced Nov 2, 2022

Split xformers.ops #486

Merged

ops.swiglu: Tests, baseline, benchmarks #487

Merged

Should fix OOMs in CI #488

Merged

unbind: Add 'stack_or_none' #498

Merged

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 2, 2022

danthe3rd mentioned this pull request Nov 2, 2022

SwiGLU optimized fw/bw #490

Merged

danthe3rd pushed a commit that referenced this pull request Nov 2, 2022

SwiGLU further optimization in MLP bw

20219b3

ghstack-source-id: 12b56c35da4e6fb208bdc51ee146601dc5c35517 Pull Request resolved: #502

danthe3rd requested a review from fmassa November 2, 2022 15:20

This was referenced Nov 2, 2022

MemEff: Set CUDA stream properly #491

Merged

SwiGLU: Support for no bias #504

Merged

Update FlashAttention #495

Merged

danthe3rd added 3 commits November 3, 2022 08:21

danthe3rd mentioned this pull request Nov 3, 2022

Benchmarks for entire models #505

Merged

danthe3rd mentioned this pull request Nov 4, 2022

Update mypy to 0.982 #508

Merged

danthe3rd added 2 commits November 4, 2022 10:04

This was referenced Nov 4, 2022

Raise if kernel not supported on device #509

Merged

MemEff: Raise if wrong bias #510

Merged

Include git hash and install date in version #511

Merged

fmassa approved these changes Nov 10, 2022

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

xformers/components/swiglu/cuda/gemm_fused_operand_sum.cu Show resolved Hide resolved

danthe3rd mentioned this pull request Nov 10, 2022

Address nits from previous PR stack + bump version #518

Merged

danthe3rd merged commit 1227f05 into gh/danthe3rd/57/base Nov 10, 2022

danthe3rd pushed a commit that referenced this pull request Nov 10, 2022

SwiGLU further optimization in MLP bw

cd27040

ghstack-source-id: b1bead3ade9c469dc859d3576b8834488845e57c Pull Request resolved: #502

danthe3rd deleted the gh/danthe3rd/57/head branch November 10, 2022 18:11

bertmaher pushed a commit to bertmaher/xformers that referenced this pull request Dec 20, 2024

Fuse s_prime and m_prime into lse in block-sparse (facebookresearch#502)

9bc9fde

Saves memory and is slightly faster

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SwiGLU further optimization in MLP bw #502

SwiGLU further optimization in MLP bw #502

danthe3rd commented Nov 2, 2022 •

edited

Loading

codecov-commenter commented Nov 4, 2022 •

edited

Loading

fmassa left a comment

SwiGLU further optimization in MLP bw #502

SwiGLU further optimization in MLP bw #502

Conversation

danthe3rd commented Nov 2, 2022 • edited Loading

codecov-commenter commented Nov 4, 2022 • edited Loading

Codecov Report

fmassa left a comment

Choose a reason for hiding this comment

danthe3rd commented Nov 2, 2022 •

edited

Loading

codecov-commenter commented Nov 4, 2022 •

edited

Loading