Flaky Regression Tests #1415

zcorpan · 2020-05-28T12:16:09Z

test/tests/dialog-modal_dialog.js is failing locally but passes in TravisCI

Test: test/tests/dialog-modal_dialog.js
Example tested by this test: examples/dialog-modal/dialog.html

3 tests are failing locally for me in openDialog3 in master branch. They seem to not be failing in Travis CI.

error log

3 tests failed

  dialog-modal_dialog › dialog-modal/dialog.html [data-test-id="key-tab"]: tab changes focus within dialog

  test/tests/dialog-modal_dialog.js:85

   84:   const dialog = (await t.context.queryElements(t, '#dialog3'))[0];
   85:   assert(                                                          
   86:     await dialog.isDisplayed(),                                    

  Rejected promise returned by test. Reason:

  AssertionError {
    actual: false,
    code: 'ERR_ASSERTION',
    expected: true,
    generatedMessage: false,
    operator: '==',
    message: 'dialog3 should have successfullly openned',
  }

  openDialog3 (test/tests/dialog-modal_dialog.js:85:3)
  test/tests/dialog-modal_dialog.js:249:3



  dialog-modal_dialog › dialog-modal/dialog.html [data-test-id="key-shift-tab"]: shift tab changes focus within dialog

  test/tests/dialog-modal_dialog.js:85

   84:   const dialog = (await t.context.queryElements(t, '#dialog3'))[0];
   85:   assert(                                                          
   86:     await dialog.isDisplayed(),                                    

  Rejected promise returned by test. Reason:

  AssertionError {
    actual: false,
    code: 'ERR_ASSERTION',
    expected: true,
    generatedMessage: false,
    operator: '==',
    message: 'dialog3 should have successfullly openned',
  }

  openDialog3 (test/tests/dialog-modal_dialog.js:85:3)
  test/tests/dialog-modal_dialog.js:358:3



  dialog-modal_dialog › dialog-modal/dialog.html [data-test-id="key-escape"]: escape closes dialog

  test/tests/dialog-modal_dialog.js:85

   84:   const dialog = (await t.context.queryElements(t, '#dialog3'))[0];
   85:   assert(                                                          
   86:     await dialog.isDisplayed(),                                    

  Rejected promise returned by test. Reason:

  AssertionError {
    actual: false,
    code: 'ERR_ASSERTION',
    expected: true,
    generatedMessage: false,
    operator: '==',
    message: 'dialog3 should have successfullly openned',
  }

  openDialog3 (test/tests/dialog-modal_dialog.js:85:3)
  test/tests/dialog-modal_dialog.js:439:5

Originally posted by @zcorpan in #1362 (comment)

The text was updated successfully, but these errors were encountered:

jongund · 2020-07-14T16:46:55Z

Problems with my PRs:

The update landmark example pull request, is generating errors for unrelated menubar tests.
The menu button example pull request. is generating regression errors in Travis, but not locally on Windows and macOS.
The slider update pull request. the CI never finishes, does not seem to be a way to restart it.
The Date picker dialog example, a tablist example test is failing, but this pull request does not modify the tablist example or tests.

jongund · 2020-07-20T15:49:28Z

@spectranaut
@mcking65
@zcorpan

The regression tests are not reliable on Windows 10, I have discovered this with the menu button examples.
I am not sure what to do, but if anyone is running regression tests on Windows 10, they should get a warning of false positives or that they need to wait for the CI to verify tests have passed, although there seems to be some issues with some menu button tests even in the CI (e.g. aria-expanded being removed). This will make it diffuclt for people developing examples and regression tests in Windows 10. I know I will not be using Windows 10 much for development.

jongund · 2020-07-20T16:01:07Z

@spectranaut
I fixed some bugs with aria-expanded not being removed and aria-activedescendant not being properly reset when the menu closes on the menu button examples. The regression testing on Windows 10 (flaky as it is) and macOS show all tests passing, for the three examples, but the CI still is generating errors, and the code through inspection appears to be working correctly.

spectranaut · 2020-08-07T22:09:10Z

Hey @jongund I just spent most of today looking into the flaky menubutton tests in master -- and I'm sorry to say I have not gotten to the bottom of it.. I ran a lot of tests and in the CI (over on the bocoup fork, see here: bocoup#27), there are a few menu button tests that regularly break the build, like 8 out of 10 times. I can't get the tests to fail on my own computer.

I also ran a bunch of tests by forking your branch #1401 with the menubutton changes. When I ran that branch in the CI, even more menu button tests failed in the CI and even more regularly.

For now:

I'm going to open a PR to mark those tests as "failing" until I or someone later can address them. I'll write a comment referencing this issue and we will see the missing tests in th regression report -- I just don't want those failing tests to slow down anyone working on something unrelated.
I would hold off on work on Update menubutton examples: High Contrast Support and APG Coding Practices #1401 until I or someone else can spend time debugging the issue. Probably it would help to do a code review of your changes, although, I looked at the code a bit and definitely nothing stood out. For the most part I was focused on the tests, and trying to pin point where the race conditions are that lead to the failures. I'll ask Matt if I can look into this the next time I have APG time.

spectranaut · 2020-08-07T22:11:12Z

These are the errors I got running the menubar test in the CI in master (bocoup#27), one or two fails per run:

 not ok 150 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-up-arrow"]: "up arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi4',

          code: 'ERR_ASSERTION',

          expected: 'mi3',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi3 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      runMicrotasks (<anonymous>)

      test/tests/menu-button_actions-active-descendant.js:308:5

  ...

not ok 160 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-up-arrow"]: "up arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi3',

          code: 'ERR_ASSERTION',

          expected: 'mi2',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi2 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      runMicrotasks (<anonymous>)

      test/tests/menu-button_actions-active-descendant.js:308:5

  ...

not ok 163 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-character"]: "character" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi1',

          code: 'ERR_ASSERTION',

          expected: 'mi2',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi2 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      runMicrotasks (<anonymous>)

      test/tests/menu-button_actions-active-descendant.js:383:5

  ...

not ok 154 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-up-arrow"]: "up arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi1',

          code: 'ERR_ASSERTION',

          expected: 'mi4',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi4 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      test/tests/menu-button_actions-active-descendant.js:303:3

  ...

not ok 159 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-down-arrow"]: "down arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi2',

          code: 'ERR_ASSERTION',

          expected: 'mi3',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi3 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      runMicrotasks (<anonymous>)

      test/tests/menu-button_actions-active-descendant.js:289:5

  ...

not ok 152 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-up-arrow"]: "up arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi4',

          code: 'ERR_ASSERTION',

          expected: 'mi3',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi3 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      test/tests/menu-button_actions-active-descendant.js:308:5

  ...

not ok 152 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-up-arrow"]: "up arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi4',

          code: 'ERR_ASSERTION',

          expected: 'mi3',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi3 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      test/tests/menu-button_actions-active-descendant.js:308:5

  ...

ot ok 159 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-down-arrow"]: "down arrow" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi2',

          code: 'ERR_ASSERTION',

          expected: 'mi3',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi3 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      runMicrotasks (<anonymous>)

      test/tests/menu-button_actions-active-descendant.js:289:5

  ...

not ok 157 - menu-button_actions-active-descendant › menu-button/menu-button-actions-active-descendant.html [data-test-id="menu-character"]: "character" on role="menu"

  ---

    name: AssertionError

    message: Rejected promise returned by test

    values:

      'Rejected promise returned by test. Reason:': |-

        AssertionError {

          actual: 'mi1',

          code: 'ERR_ASSERTION',

          expected: 'mi2',

          generatedMessage: false,

          operator: 'strictEqual',

          message: 'aria-activedescendant should be set to mi2 for item: #ex1 [role="menu"]',

        }

    at: >-

      assertAriaSelectedAndActivedescendant

      (test/util/assertAriaActivedescendant.js:22:10)

      runMicrotasks (<anonymous>)

      test/tests/menu-button_actions-active-descendant.js:383:5

  ...

Disable a few unreliable regression tests that we don't want to impede other work while we determine the root cause of the inconsistent test behavior. This change comments out tests that fail most frequently in CI. The regression test coverage report will report them as missing. The work to resolve the root issue is tracked by issue #1415.

Infrastructure: Temporarily comment out flaky tests (pull #1488) Disable a few unreliable regression tests that we don't want to impede other work while we determine the root cause of the inconsistent test behavior. This change comments out tests that fail most frequently in CI. The regression test coverage report will report them as missing. The work to resolve the root issue is tracked by issue #1415.

spectranaut · 2020-08-22T00:37:43Z

Fixed a bunch of flaky tests in this PR: #1401 (comment)

zcorpan mentioned this issue May 28, 2020

Date Picker Dialog Example: Improve high contrast support and refactor javascript #1362

Merged

6 tasks

spectranaut changed the title ~~test/tests/dialog-modal_dialog.js is failing locally but passes in TravisCI~~ Flaky Regression Tests Jul 14, 2020

spectranaut mentioned this issue Aug 7, 2020

Temporarily comment out flaky tests #1488

Merged

spectranaut closed this as completed Aug 22, 2020

mcking65 added bug Code defects; not for inaccurate prose regression-testing Related to AVA regression tests of example pages or AVA framework implementation within repo labels Aug 24, 2020

mcking65 added this to the 1.2 Release 1 milestone Aug 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flaky Regression Tests #1415

Flaky Regression Tests #1415

zcorpan commented May 28, 2020 •

edited by spectranaut

Loading

jongund commented Jul 14, 2020 •

edited

Loading

jongund commented Jul 20, 2020

jongund commented Jul 20, 2020

spectranaut commented Aug 7, 2020

spectranaut commented Aug 7, 2020

spectranaut commented Aug 22, 2020

Flaky Regression Tests #1415

Flaky Regression Tests #1415

Comments

zcorpan commented May 28, 2020 • edited by spectranaut Loading

jongund commented Jul 14, 2020 • edited Loading

jongund commented Jul 20, 2020

jongund commented Jul 20, 2020

spectranaut commented Aug 7, 2020

spectranaut commented Aug 7, 2020

spectranaut commented Aug 22, 2020

zcorpan commented May 28, 2020 •

edited by spectranaut

Loading

jongund commented Jul 14, 2020 •

edited

Loading