[Logs UI] Interpret finished analysis jobs as healthy #45268

weltenwort · 2019-09-10T14:42:05Z

Summary

This changes the job health check of the log analysis page to consider finished jobs as healthy. This situation is relevant when the job is non-continuous, i.e. created with a definite end date.

At the same time it ensures that datafeeds that failed to be started due to empty indices lead to the jobs being considered unhealthy.

fixes #45180

Testing

Scenario for a non-continuous job

Navigate to the Analysis tab of the Logs UI.
Create a job with a non-"indefinite" end date.
Wait until the job has stopped normally due to having processes all data. This can be checked in the ML UI.
Reload the Analysis page in the Logs UI.
Check that the result screen is shown.

Scenario for failed datafeed start

Create an index with a correct filebeat mapping, but that is empty (e.g. filebeat-8.0.0-empty with standard index templates).
Configure a source such that it only uses that index.
Attempt to set up the analysis jobs for the "empty" source via the Analysis tab.
Check that the setup fails.
Reload the Analysis page in the Logs UI.
Check that the setup screen is shown.

Checklist

This was checked for cross-browser compatibility, including a check against IE11
~~Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~Documentation was added for features that require explanation or tutorials~~
~~Unit or functional tests were updated or added to match the most common scenarios~~
~~This was checked for keyboard-only and screenreader accessibility~~

elasticmachine · 2019-09-10T14:42:11Z

Pinging @elastic/infra-logs-ui

…ealthy

elasticmachine · 2019-09-10T18:06:43Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

Kerry350

Looks good, nice work 👍 This fixes the scenarios listed. I've left a comment regarding one of the states, but it's not critical.

Kerry350 · 2019-09-11T09:16:23Z

x-pack/legacy/plugins/infra/public/containers/logs/log_analysis/log_analysis_jobs.tsx

-        setJobStatus(getInitialJobStatuses());
+        setJobStatus(currentJobStatus => ({
+          ...currentJobStatus,
+          'log-entry-rate': 'initializing',


The only part I'm not sure about is this, and it's maybe not a big problem overall. We use jobSummary.jobState === 'opening' to determine a jobStatus state of initializing, my understanding is this would be when everything has been created successfully my ML, but is still opening up. But here we're setting that in the createPromise of setupMlModule when things haven't been created yet. Maybe unknown would be a better state at this point?

If we look at the JobStatus as an indicator of the current stage of the (abstract) job/datafeed lifecycle, wouldn't unknown be incorrect? We know the state of the job, which is that it's being created but not running yet. Do you think we need to differentiate that more or that there is a risk in decoupling our abstract job state too much from the technical ML job state?

We know the state of the job, which is that it's being created but not running yet.

Okay, put like that it makes sense 👍 No, lets not add another state if it's not needed. Thanks for the explanation.

This changes the job health check of the log analysis page to consider finished jobs as healthy. This situation is relevant when the job is non-continuous, i.e. created with a definite end date. At the same time it ensures that datafeeds that failed to be started due to empty indices lead to the jobs being considered unhealthy. fixes elastic#45180

Backports the following commits to 7.x: - [Logs UI] Interpret finished analysis jobs as healthy (#45268)

Backports the following commits to 7.4: - [Logs UI] Interpret finished analysis jobs as healthy (#45268)

…-to-np-ready * 'master' of github.com:elastic/kibana: (25 commits) [ML] Fixes display of matching modules in index data visualizer (elastic#45261) [Console] Update indentation behaviour (elastic#45249) Convert value provided to PhraseValueInput to string to catch Exception (elastic#45259) [Region Map] Fix loading default vector map and base layer setting (elastic#43858) [ML] Fixing empty time range when cloning jobs (elastic#45286) [ML] Fixing wizard validation delay (elastic#45265) [Logs UI] Interpret finished analysis jobs as healthy (elastic#45268) [Console] SQL template with triple quote in completion (elastic#45248) [ML] Data Frames: Cards as links (elastic#45254) fix(code/frontend): should show updating instead of cloning when updating (elastic#45238) fix(code/frontend): fix document search result from (elastic#45236) disable another flaky suite (elastic#45323) (elastic#45330) disable flaky suite (elastic#45105) skip flaky suite (elastic#43069) skip flaky suite (elastic#45089) disable jest suite that has no enabled tests (elastic#44250) disable flaky test (elastic#45317) disable flaky test (elastic#45315) [DOCS] Creates developer folder (elastic#45280) [SIEM] Changes ML conditional links to use tabs, fixes a small bug with null filterQuery (elastic#45218) ...

…ditor * 'master' of github.com:elastic/kibana: (76 commits) Upgrade EUI to 13.8.1 (elastic#45052) [ML] Add multi metric job wizard test (elastic#45279) [SIEM] Inject/apply KQL changed in refresh button (elastic#45065) [Graph] Type persistence (elastic#44985) Functional tests: convert more test/services to TS (elastic#45176) [ML] Fixes display of matching modules in index data visualizer (elastic#45261) [Console] Update indentation behaviour (elastic#45249) Convert value provided to PhraseValueInput to string to catch Exception (elastic#45259) [Region Map] Fix loading default vector map and base layer setting (elastic#43858) [ML] Fixing empty time range when cloning jobs (elastic#45286) [ML] Fixing wizard validation delay (elastic#45265) [Logs UI] Interpret finished analysis jobs as healthy (elastic#45268) [Console] SQL template with triple quote in completion (elastic#45248) [ML] Data Frames: Cards as links (elastic#45254) fix(code/frontend): should show updating instead of cloning when updating (elastic#45238) fix(code/frontend): fix document search result from (elastic#45236) disable another flaky suite (elastic#45323) (elastic#45330) disable flaky suite (elastic#45105) skip flaky suite (elastic#43069) skip flaky suite (elastic#45089) ...

weltenwort added 2 commits September 10, 2019 16:26

Tighten up job summary response validation

ac8fc95

Interpret finished jobs as healthy

74f785e

weltenwort added bug Fixes for quality problems that affect the customer experience v8.0.0 Feature:Logs UI Logs UI feature Team:Infra Monitoring UI - DEPRECATED DEPRECATED - Label for the Infra Monitoring UI team. Use Team:obs-ux-infra_services v7.4.0 v7.5.0 labels Sep 10, 2019

weltenwort self-assigned this Sep 10, 2019

Kerry350 self-requested a review September 10, 2019 16:23

Merge branch 'master' into logs-ui-analysis-detect-finished-jobs-as-h…

48e47b0

…ealthy

weltenwort marked this pull request as ready for review September 10, 2019 18:17

weltenwort requested a review from a team as a code owner September 10, 2019 18:17

Kerry350 approved these changes Sep 11, 2019

View reviewed changes

weltenwort added the release_note:skip Skip the PR/issue when compiling release notes label Sep 11, 2019

weltenwort merged commit 7202f51 into elastic:master Sep 11, 2019

weltenwort deleted the logs-ui-analysis-detect-finished-jobs-as-healthy branch September 11, 2019 11:13

weltenwort mentioned this pull request Sep 11, 2019

[7.x] [Logs UI] Interpret finished analysis jobs as healthy (#… #45353

Merged

weltenwort mentioned this pull request Sep 11, 2019

[7.4] [Logs UI] Interpret finished analysis jobs as healthy (#… #45354

Merged

weltenwort added a commit that referenced this pull request Sep 11, 2019

[7.x] [Logs UI] Interpret finished analysis jobs as healthy (#… (#45353)

a925b9b

Backports the following commits to 7.x: - [Logs UI] Interpret finished analysis jobs as healthy (#45268)

weltenwort added a commit that referenced this pull request Sep 11, 2019

[7.4] [Logs UI] Interpret finished analysis jobs as healthy (#… (#45354)

56d0452

Backports the following commits to 7.4: - [Logs UI] Interpret finished analysis jobs as healthy (#45268)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Logs UI] Interpret finished analysis jobs as healthy #45268

[Logs UI] Interpret finished analysis jobs as healthy #45268

weltenwort commented Sep 10, 2019

elasticmachine commented Sep 10, 2019

elasticmachine commented Sep 10, 2019

Kerry350 left a comment

Kerry350 Sep 11, 2019

weltenwort Sep 11, 2019

Kerry350 Sep 11, 2019

[Logs UI] Interpret finished analysis jobs as healthy #45268

[Logs UI] Interpret finished analysis jobs as healthy #45268

Conversation

weltenwort commented Sep 10, 2019

Summary

Testing

Scenario for a non-continuous job

Scenario for failed datafeed start

Checklist

elasticmachine commented Sep 10, 2019

elasticmachine commented Sep 10, 2019

💚 Build Succeeded

Kerry350 left a comment

Choose a reason for hiding this comment

Kerry350 Sep 11, 2019

Choose a reason for hiding this comment

weltenwort Sep 11, 2019

Choose a reason for hiding this comment

Kerry350 Sep 11, 2019

Choose a reason for hiding this comment