[Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377

chrisronline · 2021-01-05T18:55:40Z

Resolves #81020

This PR reintroduces the logic discussed in #68805 where our legacy alerts (alerts which still exist as watches) need to fetch their data from .monitoring-{stack_product}-* indices instead of .monitoring-alerts-* indices.

This change is necessary because, due to #85047, we will be removing all cluster alert watches so the current logic to read from the output of these watches (which is .monitoring-alerts-* indices) needs to change. Luckily, we already wrote most of this code so it's just a matter of porting it over and making any adjustments to account for changes in the base alerting code.

To test, we just need to ensure the alerts fire properly. I recommend using the following gists (which contain code you can copy/paste into Dev Tools) to help trigger a specific alert:

License expiration -> https://gist.github.com/chrisronline/9d4d3d740e535d3c01410cac2cc74653
ES nodes changed -> https://gist.github.com/chrisronline/d441aba1a08cb45082e59f39cc9f6687
Kibana version mismatch -> https://gist.github.com/chrisronline/34328d14738f0ce754e36ec7031e45a9
Logstash version mismatch -> https://gist.github.com/chrisronline/3b982d95710ef820d11c7443a1e49091
ES version mismatch -> https://gist.github.com/chrisronline/2191a43d0ea26c0330748202175b960e

Cluster health is fairly easy to replicate - just create a document in an index without a replica (which is the default behavior):

PUT twitter/_doc/1
{
  "name": "chris"
}

Keep in mind that legacy alerts required a gold+ license but the changes here will mean basic+ can leverage them.

…_legacy_alerts

elasticmachine · 2021-01-20T17:04:08Z

Pinging @elastic/stack-monitoring (Team:Monitoring)

chrisronline · 2021-01-25T15:32:49Z

@elasticmachine merge upstream

kibanamachine · 2021-01-25T15:32:51Z

merge conflict between base and head

…_legacy_alerts

igoristic · 2021-01-31T03:38:50Z

Alright, I'm done with the functional testing under multiple environments/conditions, and everything seems to check out. Awesome job! and Thank you for the very detailed and easy to follow instructions!

Going to continue code review now

igoristic

Great job overall! 🏆

Please review the comments in the code, and some of these other issues:

I think the following alerts:

ALERT_LICENSE_EXPIRATION
ALERT_ELASTICSEARCH_VERSION_MISMATCH
ALERT_KIBANA_VERSION_MISMATCH
ALERT_LOGSTASH_VERSION_MISMATCH

Should have the default interval of 1d instead of 1m. Since, these don't change intermittently like the other metrics do

We should remove:
x-pack/plugins/monitoring/server/lib/alerts/fetch_legacy_alerts.ts
Doesn't look like it's used anymore

Shouldn't we also add nextSteps to these alerts? Or, maybe at least in a separate issue

Not sure I understand:

This alert does not feature grouping

Is this because these alerts are only supported for a single cluster?

NIT: We should probably replace the "legacy" terminology to something like "default alerts", since this PR now makes them true Kibana Alerts

x-pack/plugins/monitoring/server/alerts/base_alert.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_elasticsearch_versions.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_cluster_health.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_licenses.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_nodes_from_cluster_stats.ts

x-pack/plugins/monitoring/server/alerts/logstash_version_mismatch_alert.ts

x-pack/plugins/monitoring/server/alerts/kibana_version_mismatch_alert.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_cluster_health.ts

x-pack/plugins/monitoring/server/lib/alerts/fetch_nodes_from_cluster_stats.ts

…_legacy_alerts

chrisronline · 2021-02-05T17:46:20Z

@igoristic

I think the following alerts:

I think they all do, like [here for example](https://github.com/elastic/kibana/pull/87377/files#diff-60cf9dc990401c054b21b18964f436f24739eb0b7667fa75e8b0897e9bfef9cbL38

We should remove:

Done!

Shouldn't we also add nextSteps to these alerts? Or, maybe at least in a separate issue

Agreed on both fronts. I didn't want to complicate this work. I'm just going for straight parity

Is this because these alerts are only supported for a single cluster?

I changed the comment and added more details. LMK if this helps

kibanamachine · 2021-02-06T00:09:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7854ae2

Metrics [docs]

✅ unchanged

History

💚 Build #104426 succeeded deaa335
💚 Build #104407 succeeded aa4df51
💔 Build #104366 failed ca6c7cf
💔 Build #104339 failed a077d7c
💚 Build #100752 succeeded daaf38f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…_legacy_alerts

igoristic

Everything looks good 👍

Thank you for addressing some of my confusion!

kibanamachine · 2021-02-08T21:22:20Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: a775ff3

Metrics [docs]

✅ unchanged

History

💛 Build #104689 was flaky 83886c4
💚 Build #104451 succeeded 7854ae2
💚 Build #104426 succeeded deaa335
💚 Build #104407 succeeded aa4df51
💔 Build #104366 failed ca6c7cf

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

… directly (#87377) (#90720) * License expiration * Fetch legacy alert data from the source * Add back in the one test file * Remove deprecated code * Fix up tests * Add test files * Fix i18n * Update tests * PR feedback * Fix types and tests * Fix license headers * Remove unused function * Fix faulty license expiration logic

…timeline-and-rollover-info * 'master' of github.com:elastic/kibana: (47 commits) [Fleet] Use TS project references (elastic#87574) before/beforeEach clean up (elastic#90663) [Vega] user should be able to set a specific tilemap service using the mapStyle property (elastic#88440) [Security Solution][Case] ServiceNow SIR Connector (elastic#88655) [Search Sessions] Enable extend from management (elastic#90558) [ILM] Delete phase redesign (rework) (elastic#90291) [APM-UI][E2E] use withGithubStatus step (elastic#90651) Add folding in kb-monaco and update some viewers (elastic#90152) [Grok Debugger] Changed test to wait for grok debugger container to exist to fix test flakiness (elastic#90543) Strongly typed EUI theme for styled-components (elastic#90106) Fix vega renovate label (elastic#90591) [Uptime] Migrate to TypeScript project references (elastic#90510) [Monitoring] Migrate data source for legacy alerts to monitoring data directly (elastic#87377) [Upgrade Assistant] Add A11y Tests (elastic#90265) [Time to Visualize] Adds functional tests for linking/unlinking panel from embeddable library (elastic#89612) [dev-utils/ship-ci-stats] fail when CI stats is down (elastic#90678) chore(NA): remove write permissions on Bazel remote cache for PRs (elastic#90652) chore(NA): move bazel workspace status from bash script into nodejs executable (elastic#90560) Use default ES distribution for functional tests (elastic#88737) [Alerts] Jira: Disallow labels with spaces (elastic#90548) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/timeline/timeline.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/lib/absolute_timing_to_relative_timing.test.ts # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/lib/absolute_timing_to_relative_timing.ts

chrisronline added 5 commits December 22, 2020 12:56

License expiration

0839350

Fetch legacy alert data from the source

3407330

Add back in the one test file

a677d88

Remove deprecated code

2c75ff2

Merge remote-tracking branch 'elastic/master' into monitoring/restore…

ebbf26b

…_legacy_alerts

chrisronline mentioned this pull request Jan 5, 2021

[Monitoring] Migrate cluster alerts from watcher to Kibana alerting #81020

Closed

3 tasks

igoristic mentioned this pull request Jan 12, 2021

[Monitoring][Doc] Update out of the box alerting page #80372

Open

9 tasks

chrisronline added 6 commits January 19, 2021 10:12

Merge in master

fb5839a

Fix up tests

ef90801

Add test files

654e73e

Fix i18n

7ab849d

Merge remote-tracking branch 'elastic/master' into monitoring/restore…

fc4f12f

…_legacy_alerts

Update tests

80eff8e

chrisronline changed the title ~~[Monitoring] Migrate legacy alerts to actual alerts~~ [Monitoring] Migrate data source for legacy alerts to monitoring data directly Jan 20, 2021

chrisronline self-assigned this Jan 20, 2021

chrisronline added 8.0.0 release_note:skip Skip the PR/issue when compiling release notes review Team:Monitoring Stack Monitoring team v7.12.0 labels Jan 20, 2021

chrisronline marked this pull request as ready for review January 20, 2021 17:04

chrisronline requested a review from a team January 20, 2021 17:04

wylieconlon added v8.0.0 and removed 8.0.0 labels Jan 20, 2021

Merge remote-tracking branch 'elastic/master' into monitoring/restore…

daaf38f

…_legacy_alerts

igoristic suggested changes Feb 4, 2021

View reviewed changes

chrisronline added 3 commits February 4, 2021 10:35

Merge in master

8f99faa

PR feedback

956f761

Merge remote-tracking branch 'elastic/master' into monitoring/restore…

a077d7c

…_legacy_alerts

chrisronline requested a review from igoristic February 5, 2021 17:46

chrisronline added 4 commits February 5, 2021 13:28

Fix types and tests

ca6c7cf

Fix license headers

aa4df51

Remove unused function

deaa335

Fix faulty license expiration logic

7854ae2

chrisronline added 2 commits February 8, 2021 08:53

Merge remote-tracking branch 'elastic/master' into monitoring/restore…

83886c4

…_legacy_alerts

Merge remote-tracking branch 'elastic/master' into monitoring/restore…

a775ff3

…_legacy_alerts

igoristic approved these changes Feb 8, 2021

View reviewed changes

chrisronline merged commit 231610c into elastic:master Feb 9, 2021

chrisronline deleted the monitoring/restore_legacy_alerts branch February 9, 2021 02:50

chrisronline mentioned this pull request Feb 9, 2021

[7.x] [Monitoring] Migrate data source for legacy alerts to monitoring data directly (#87377) #90720

Merged

igoristic mentioned this pull request Feb 16, 2021

[Stack Monitoring] Alerting Phase -1 #42960

Open

10 tasks

chrisronline mentioned this pull request Mar 16, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #93072

Closed

24 tasks

simianhacker mentioned this pull request Apr 29, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #98765

Closed

24 tasks

neptunian mentioned this pull request Jun 24, 2021

[Stack Monitoring] create alert per node, index, or cluster instead of always per cluster #102544

Merged

neptunian mentioned this pull request Jul 6, 2021

[Stack Monitoring] [Test Scenario] Out of the box alerting #104440

Closed

35 tasks

neptunian mentioned this pull request Mar 1, 2022

[Stack Monitoring] Cleanup references to deprecated config #126656

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377

[Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377

chrisronline commented Jan 5, 2021 •

edited

Loading

elasticmachine commented Jan 20, 2021

chrisronline commented Jan 25, 2021

kibanamachine commented Jan 25, 2021

igoristic commented Jan 31, 2021

igoristic left a comment •

edited

Loading

chrisronline commented Feb 5, 2021

kibanamachine commented Feb 6, 2021

igoristic left a comment

kibanamachine commented Feb 8, 2021

[Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377

[Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377

Conversation

chrisronline commented Jan 5, 2021 • edited Loading

elasticmachine commented Jan 20, 2021

chrisronline commented Jan 25, 2021

kibanamachine commented Jan 25, 2021

igoristic commented Jan 31, 2021

igoristic left a comment • edited Loading

Choose a reason for hiding this comment

chrisronline commented Feb 5, 2021

kibanamachine commented Feb 6, 2021

💚 Build Succeeded

Metrics [docs]

History

igoristic left a comment

Choose a reason for hiding this comment

kibanamachine commented Feb 8, 2021

💚 Build Succeeded

Metrics [docs]

History

chrisronline commented Jan 5, 2021 •

edited

Loading

igoristic left a comment •

edited

Loading