Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Robustify speech recognition test #76652

Merged
merged 4 commits into from
Oct 6, 2022
Merged

Conversation

danmoseley
Copy link
Member

@danmoseley danmoseley commented Oct 5, 2022

Fix #76626 (?)

I can't reproduce the failure despite looping the tests for some time while keeping the machine busy doing builds. So I really have no theory why it failed.

  1. Change the test words to a couple found by trial and error to be recognized with particularly high confidence level. Used two so we can see if both fail or just one.
  2. Add diagnostic logging that is output if the tests fail.
  3. Increase a couple of relevant timeouts from defaults.

Also added an error case test with silence. (Was not able to find an input string that it would refuse to recognize. It always just had low confidence. White noise does lead to failure, but I don't have a quick way to make that in a test.)

@ghost
Copy link

ghost commented Oct 5, 2022

Tagging subscribers to this area: @danmoseley
See info in area-owners.md if you want to be subscribed.

Issue Details

Fix #76626 (?)

I can't reproduce the failure despite looping the tests for some time while keeping the machine busy doing builds. So I really have no theory why it failed.

  1. Change the test words to a couple found by trial and error to be recognized with particularly high confidence level. Used two so we can see if both fail or just one.
  2. Add diagnostic logging that is output if the tests fail.
  3. Increase a couple of relevant timeouts from defaults.

Also added an error case test with silence. (Was not able to find an input string that it would refuse to recognize. It always just had low confidence.)

Author: danmoseley
Assignees: -
Labels:

area-System.Speech

Milestone: -

@danmoseley
Copy link
Member Author

danmoseley commented Oct 5, 2022

Looks like I should use "Contains"


       Passing synthesized input 'recognize'
        Speech detected at position 00:00:00
        Received speech recognized event with result 'A recognized'
        Recognized 'A recognized' with confidence 0.0050567742
        Elapsed 00:00:02.1001459
        Alternatives included 'A recognized'' with confidence 0.0050567742
        Alternatives included 'a recognized'' with confidence 0.0050567742
        Alternatives included 'recognized'' with confidence 0.0020464011
        Alternatives included 'I recognized'' with confidence 0.0024469797

Also retries, as this should be ~0.85 confidence, somethings' all wrong in this case.

@danmoseley
Copy link
Member Author

  1. Added retries.
  2. Skip on 32 bit Windows 7, as that's all the recent failures.
  3. Change to "contains" so that "a recognized" matches "recognize"

The combination of this should do it.

@danmoseley
Copy link
Member Author

test failures are unrelated: #76681

@danmoseley
Copy link
Member Author

Fixed typo, since rerunning might not be a bad thing.

@danmoseley
Copy link
Member Author

@carlossanlop maybe you could review? I expect checks to be green/known failures

@danmoseley
Copy link
Member Author

Failures are #75354 and #76681. Should be good to merge.

Copy link
Member

@carlossanlop carlossanlop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danmoseley danmoseley merged commit 8571fcb into dotnet:main Oct 6, 2022
@danmoseley danmoseley deleted the synthtest branch October 6, 2022 00:41
@ghost ghost locked as resolved and limited conversation to collaborators Nov 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SynthesizeRecognizeTests.SpeechSynthesizerToSpeechRecognitionEngine - NullReferenceException
2 participants