-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flaky discover test: doc view should show oldest time first #13163
Comments
Ran into this again:
https://kibana-ci.elastic.co/job/elastic+kibana+pull-request+multijob-selenium/7244/consoleFull |
@elastic/kibana-discovery, given the recent push to get rid of known flaky tests, I wonder if we should comment this one out. While not encountered often, I still do encounter it every now and then. Actually just ran into it twice in a row in two different PRs. |
Sure, if you wanna comment it out I'll add it to our roadmap to look into. |
It just failed again here; The first ~3500 chars of the actual and expected docs look identical up until this part;
|
@weltenwort has a PR to disable this test temporarily #15412 I'm also going to see if I can fix this tomorrow during low fruit friday |
I'm thinking that the data returned by Elasticsearch has to be wrong to produce this error. Either es_archiver loaded bad data into Elasticsearch somehow, or Elasticsearch mangled the data in the response. The screenshot from the failed test doesn't help because the doc table in discover isn't showing this many chars on the screen to see the point of failure. The Selenium test is getting the entire table cell of data in one shot, so it seems really unlikely the test itself is flakey. The call here; https://github.com/elastic/kibana/blob/master/test/functional/apps/discover/_field_data.js#L116 Is a single getVisibleText call;
So I don't think that field could update in the middle of getting the text. |
I spent some time looking into this today and I don't think there's an issue with the data coming from ES but I also don't think the test itself is flaky. The dates at the end of the string are coming from doc_values. They should match the dates in the documents themselves. If you look at the dates coming from _source in the doc table they're correct, it's just the doc values that are wrong. So either ES is responding with doc_values that are out of sync with the document (which seems really unlikely to me) or we have some sort of race condition or stale state issue in our code that parses the response document. I'm going to try to investigate the latter when I've got some more time. |
Showing incorrect data seems like a very serious issue. What steps can we take to get to the bottom of this? |
I submitted a PR to add debug code to troubleshoot the issue but it seems to have fallen off the radar of the reviewers. Has this test failed recently? If so I can revive the PR. |
No the test hasn't failed recently but that's because it's being skipped. #15412 |
K, I'll try to rebase my old branch and snag a reviewer tomorrow if I've got a spare moment |
@Bargs that Discover test failed again here, so maybe there's more debug info on it now? |
Should we skip this test until we can narrow root cause ? |
@Bargs Here's the output from that Jenkins job;
|
@LeeDr thanks for grabbing the output, I was out on vacation and the build info in Jenkins was already gone by the time I got back. I'll take a look at it. |
ok, there's definitely a bug here. The raw response coming back from ES looks fine. We must be modifying the response document somewhere. |
Closing this, since this test seem to no longer exist. |
Just ran into this in #13159. The PR it's backporting passed 7 times in a row, but must not have fixed this particular one.
Looks like it's relatively rare but happening more often recently.
Looks like most of the results are the same except a few dates at the end.
cc @elastic/kibana-discovery
The text was updated successfully, but these errors were encountered: