[Monitoring] Remove deprecated watcher-based cluster alerts

[chrisronline]

Relates to #81020

This PR adds logic to call a new API introduced by the ES team (elastic/elasticsearch#50032) to disable and delete legacy watcher-based cluster alerts.

I'm not sure if we should surface a toast to the user if this action fails (the PR currently does do this), and if we do, I'm not sure what it should say.

In a scenario where the API call fails, the user needs to review the error message and most likely manually delete the watches on the affected cluster(s). We should probably supplement this work with an update to our docs to tell users what this is and steps on removing the watches manually.

Curious to thoughts from @igoristic and @ravikesarwani

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[ravikesarwani]

I am assuming the API is called after we have successfully created replacement Kibana alerts.
If we failed to delete legacy watcher-based cluster alerts we should show a message to the user.
In the message I think 2 thing should be mentioned:

• Created replacement Kibana alerts.
• Failed to delete legacy watcher based cluster alerts and some quick glimpse of how they can delete it manually.

[chrisronline]

Thanks @ravikesarwani!

some quick glimpse of how they can delete it manually.

I don't think we have any docs for this, outside of general watch deletion docs. Should we create new ones, or link to what we have?

[ravikesarwani]

Linking to existing DELETE Watch API is fine.

[chrisronline]

@elasticmachine merge upstream

[igoristic]

@chrisronline Great job 🥇

We should make a note though to revert this PR if for some reason: #87377 does not get merged into 7.12

[chrisronline]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: e647aa1

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
monitoring	979.8KB	981.2KB	+1.5KB

History

• 💚 Build #97334 succeeded a02a7ff
• 💚 Build #96657 succeeded 08f3c59
• 💔 Build #95980 failed 830f884
• 💔 Build #92062 failed 28e096c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream