-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#6344: Update RoBERTa QA demo #8896
base: main
Are you sure you want to change the base?
Conversation
f712c92
to
519788f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking for tests with both these annotations...
@pytest.mark.models_device_performance_bare_metal
@pytest.mark.models_performance_bare_metal
Note that we document this here: https://tenstorrent.github.io/tt-metal/latest/ttnn/ttnn/demos.html
See for example:
tests/ttnn/integration_tests/resnet/test_performance.py
@eyonland, |
519788f
to
72b75e3
Compare
f718b4d
to
2a63c48
Compare
2a63c48
to
db9ed9f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be good.
Please re-run the GS branch of ttnn integration tests in FD nightly.
I see
FAILED tests/ttnn/integration_tests/roberta/test_ttnn_optimized_roberta.py::test_roberta[sequence_size=384-batch_size=8-model_name=deepset/roberta-large-squad2] - OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like deepset/roberta-large-squad2 is not the path to a directory containing a file named config.json.
WH passes, but want to make sure of GS.
db9ed9f
to
a35a87c
Compare
Great, thank you |
@esmalTT , Could you please review and give your approval for this PR? |
tt_output_start_logits = tt_output[..., :, 0] | ||
tt_output_end_logits = tt_output[..., :, 1] | ||
|
||
assert_with_pcc(torch_output_start_logits, tt_output_start_logits, 0.81 if is_grayskull else 0.89) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do these PCC bounds need to be updated?
Ticket
Problem description
What's changed
Checklist