Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate mismatch between mixtral README numbers and CI/dashboard numbers #12578

Closed
skhorasganiTT opened this issue Sep 12, 2024 · 3 comments
Assignees

Comments

@skhorasganiTT
Copy link
Contributor

Possible perf regression:
Mixtral:
README: 15.7
latest passing release (https://github.com/tenstorrent/tt-metal/actions/runs/10693502206): 13.6

@skhorasganiTT
Copy link
Contributor Author

Assigning P0 per the request of @uaydonat

@mtairum
Copy link
Contributor

mtairum commented Sep 13, 2024

The 15.7 t/s/u was an abnormal perf that lasted a few days. We weren't doing perf analysis on Mixtral at the time, so we never checked what might have caused the bump.

But 13.6 t/s/u is in-inline with that the current perf should be, so I'll update the README file.

Keep in mind that after the Llama3.1-8B perf sprint we'll give some TLC to Mixtral, both CI-wise and perf-wise.

@mtairum
Copy link
Contributor

mtairum commented Sep 13, 2024

Pushed README change here: #12629

@uaydonat Let me know if we should keep this open for later when we resume mixtral perf, or close it now and then we reopen when Mixtral is stable again and ready for perf improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants