Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#12520: Adding noc_async_writes_flushed between mcast writes and mcast semaphore sets for BH #12627

Merged
merged 1 commit into from
Sep 14, 2024

Conversation

abhullar-tt
Copy link
Contributor

@abhullar-tt abhullar-tt commented Sep 13, 2024

Ticket

#12520

Problem description

After enabling cmd buffer FIFOs (needed to stop ND hangs with this test), this test starting having ND mismatches

What's changed

Added some noc_async_writes_flushed between mcast writes and mcast semaphore sets. These use different cmd buffers but it seems like order is preserved on GS/WH but not on BH.

Needs further investigation about why this is needed. #12678

When mcast write and mcast sem set use the same cmd buffer then we don't have this issue (https://github.com/tenstorrent/tt-metal/tree/abhullar/nd-mismatch-2)

FYI @davorchap

Checklist

Copy link
Contributor

@tt-aho tt-aho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if anyone uses these for reference for MMs but should we leave a comment that it is added due to a bug/issue in the code and ifdef it only for BH?

@abhullar-tt
Copy link
Contributor Author

Not sure if anyone uses these for reference for MMs but should we leave a comment that it is added due to a bug/issue in the code and ifdef it only for BH?

Yes I will ifdef it, just put this pr up to bombard the BH post commit workflow

@abhullar-tt abhullar-tt changed the title #12520: Adding noc_async_writes_flushed after mcast writes in test kernels for MatmulMultiCoreMultiDRAMIn0MCastIn1MCast #12520: Adding noc_async_writes_flushed between mcast writes and mcast semaphore sets for BH Sep 13, 2024
@abhullar-tt abhullar-tt merged commit 7380bb1 into main Sep 14, 2024
128 of 129 checks passed
@abhullar-tt abhullar-tt deleted the abhullar/nd-mismatch branch September 14, 2024 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants