[Regressions] ThunderFX is slower than 2 weeks ago for 3 models

[wprazuch]

🐛 Bug

Recently found regressions:

falcon-40b -> mentioned in #1366
falcon-180B -> mentioned in #1484
falcon-7b -> New, regression probably due to MBS difference w.r.t torch.compile
pythia-12b -> New, fp8 regression, bf16 regression probably due to MBS difference w.r.t torch.compile
Platypus-30B -> mentioned in #1366
Gemma-7b -> mentioned in #1484
vicuna-33b-v1.3 -> mentioned in #1366
pythia-14m -> New, regression probably due to MBS difference w.r.t torch.compile

To Reproduce

Tested on pjnl-20241205 (as in the Latest image date in the screenshot).

Environment

system.device_product_name DGXH100
system.gpu_driver_version 535.129.03
libraries.cuda 12.8.33
libraries.pip.lightning 2.4.0.dev20240728
libraries.pip.lightning-thunder 0.2.0.dev0
libraries.pip.lightning-utilities 0.11.9
libraries.pip.litgpt 0.4.11
libraries.pip.nvfuser 0.2.23+git1dda106
libraries.pip.pytorch-lightning 2.4.0
libraries.pip.torch 2.6.0a0+git45ed7c1
libraries.pip.torchao NaN
libraries.pip.torchmetrics 1.6.0
libraries.pip.torchvision 0.19.0a0+d23a6e1