Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#6344: Update RoBERTa QA demo #8896

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kkeerthana0573
Copy link
Contributor

@kkeerthana0573 kkeerthana0573 commented May 28, 2024

Ticket

  • Link to Github Issue

Problem description

  • Functional RoBERTa model Demo.
  • Port RoBERTa model to n300.

What's changed

  • TTNN support for RoBERTa model is done using ttnn_optimized_bert pipeline.
  • Ported RoBERTa model functionality to n300.
  • Added demo for RoBERTaForQuestionAnswering task.

Checklist

  • Post commit CI passes
  • Blackhole Post commit (if applicable)
  • Model regression CI testing passes (if applicable)
  • Device performance regression CI testing passes (if applicable)
  • New/Existing tests provide coverage for changes

Copy link
Contributor

@eyonland eyonland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking for tests with both these annotations...

@pytest.mark.models_device_performance_bare_metal
@pytest.mark.models_performance_bare_metal

Note that we document this here: https://tenstorrent.github.io/tt-metal/latest/ttnn/ttnn/demos.html
See for example:
tests/ttnn/integration_tests/resnet/test_performance.py

@kkeerthana0573
Copy link
Contributor Author

@eyonland,
Will add the test and update the PR.
Thank you.

@kkeerthana0573 kkeerthana0573 force-pushed the keerthana/functional_roberta_demo branch from 519788f to 72b75e3 Compare June 19, 2024 08:09
@kkeerthana0573 kkeerthana0573 force-pushed the keerthana/functional_roberta_demo branch 3 times, most recently from f718b4d to 2a63c48 Compare November 11, 2024 07:20
@kkeerthana0573 kkeerthana0573 force-pushed the keerthana/functional_roberta_demo branch from 2a63c48 to db9ed9f Compare November 15, 2024 11:59
Copy link
Collaborator

@tt-rkim tt-rkim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be good.

Please re-run the GS branch of ttnn integration tests in FD nightly.

I see

FAILED tests/ttnn/integration_tests/roberta/test_ttnn_optimized_roberta.py::test_roberta[sequence_size=384-batch_size=8-model_name=deepset/roberta-large-squad2] - OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like deepset/roberta-large-squad2 is not the path to a directory containing a file named config.json.

WH passes, but want to make sure of GS.

@kkeerthana0573 kkeerthana0573 force-pushed the keerthana/functional_roberta_demo branch from db9ed9f to a35a87c Compare November 19, 2024 02:01
@kkeerthana0573
Copy link
Contributor Author

@tt-rkim,
FD Nightly for GS passed. Please find the link here.

@tt-rkim
Copy link
Collaborator

tt-rkim commented Nov 19, 2024

Great, thank you

@saichandax
Copy link
Contributor

@esmalTT , Could you please review and give your approval for this PR?

tt_output_start_logits = tt_output[..., :, 0]
tt_output_end_logits = tt_output[..., :, 1]

assert_with_pcc(torch_output_start_logits, tt_output_start_logits, 0.81 if is_grayskull else 0.89)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these PCC bounds need to be updated?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants