Can't set timerange correctly during functional tests

[timroes]

It seems recently we're seeing a lot of test failures caused by the time range not being set correctly. I've seen a couple of test failures where the "to" date is correctly set, but the "from" date is not set correctly. Failure screenshots might then look like the following (pay attention to the dates in the timepicker):

This seems to happen across the board in different apps and different tests. I suspect the PageObjects.timePicker.setAbsoluteRange might have some problems with potentially some changes we did in EUI?
I think this needs some investigation. I mostly linked Discover failures below, but I've seen this failing across other apps (e.g. Dashboard) too.

List of affected test failures:

• Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/feature_controls/discover_spaces·ts - discover feature controls spaces space with no features disabled shows "visualize" field button #60559
  • [QA][fix failing test] Refactor to ui settings for timepicker #114182
• Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/feature_controls/discover_spaces·ts - discover feature controls spaces space with Visualize disabled Does not show the "visualize" field button #113067
• Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/saved_searches·ts - discover Discover Saved Searches Customize time range should be possible to customize time range for saved searches on dashboards #104578
• Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/visualize/reporting·ts - Visualize Visualize Reporting Screenshots Print PDF button becomes available when saved #113496
• https://github.com/elastic/kibana/pull/112262/files?diff=split&w=1#diff-ec207f85930b36123ea02f6e89f360cd695dd53e722cae1b47670114c4ea136eR48
• Failing test: X-Pack Search Sessions Integration.x-pack/test/search_sessions_integration/tests/apps/discover/sessions_in_space·ts - Discover discover in space Storing search sessions in space Saves and restores a session #112913
• Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/saved_searches·ts - discover Discover Saved Searches should unselect saved search when navigating to a 'new' #114002
• (most likely) Failing test: Chrome UI Functional Tests.test/functional/apps/discover/_indexpattern_without_timefield·ts - discover app indexpattern without timefield should switch between with and without timefield using the browser back button #107057
• Failing test: X-Pack Search Sessions Integration.x-pack/test/search_sessions_integration/tests/apps/dashboard/async_search/sessions_in_space·ts - Dashboard dashboard in space Disabled storing search sessions Doesn't allow to store a session #112732
• Failing test: X-Pack Search Sessions Integration.x-pack/test/search_sessions_integration/tests/apps/discover/sessions_in_space·ts - Discover discover in space Storing search sessions in space Saves and restores a session #112913

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[LeeDr]

Test automation on the timepicker has been a pain for quite some time. Many people including me have spent hours and hours trying to get automation on it to work reliably.

1. @dmlemeshko is looking at wrapping the setAbsoluteTime method in a retry with verification
2. @wayneseymour has it on his TODO list to start replacing many instances of setting the timepicker by setting the timepicker:defaultTimes before navigating to the page as in this example
   https://github.com/elastic/kibana/blob/master/test/functional/apps/discover/_indexpattern_without_timefield.ts#L28

before
      await kibanaServer.uiSettings.replace({
        'timepicker:timeDefaults': '{  "from": "2019-01-18T19:37:13.000Z",  "to": "now"}',
      });
after
      await kibanaServer.uiSettings.unset('timepicker:timeDefaults');

3. We should leave at least 1 instance of using the timepicker in each page (and/or make sure there are good component level tests)

[flash1293]

cc @stratoula I think we had a case of this as well, right?

[stratoula]

Yes we had this one #113496
I am not sure about the solution I provided #113535 but it seems that it stabilizes it. Of course I have ran it only 100 times.

[flash1293]

@stratoula Seems like it didn't hold: #113496

[stratoula]

:( Ok I will try to use the solution provided by Lee

[timroes]

I am not sure if it makes sense trying to fix this now for each test individually. I think we need to come up with a central more safe solution for setting the timepicker, i.e. what Lee mentioned under point 1.

I'd for now just disable the test, and add it to the list above, so we can unskip all those tests once we have fixed that globally.

[timroes]

I believe this is not yet solved via #113597. A lot of the tests that were unskipped in this PR, started immediately again to behave flaky after that PR got merged. It just seems - assuming because we're now properly test for the date being set - that instead of seeing the date not being set, we run into timeouts in all those tests (e.g. see #60559).

I know that @pheyos (#113905) and @brianseeders (#104022, #112576) have been working on trying to address the timeout issues too. As far as I understand #113905 should at least enable us getting screenshots now from those timeout failures again? (which will help us seeing if this assumption here is actually true).

[pheyos]

As far as I understand #113905 should at least enable us getting screenshots now from those timeout failures again?

@timroes yes, for the cases where retry.waitFor or retry.waitForWithTimeout are used, we should now get regular wait timeouts, which will create screenshots.
There might still be other cases that trigger the outer test runner timeout, leading to a killed test execution and no screenshot, but at this point I'm not aware of such a case.