[Stack Monitoring] Stack Monitoring alerts no longer work with monitoring.ui.elasticsearch.* config settings

[simianhacker]

Kibana version:

Starting with 7.13.0

Elasticsearch version:

Starting with 7.13.0

Original install method (e.g. download page, yum, from source, etc.):

Amu

Describe the bug:

When a customer uses monitoring.ui.elasticsearch.* to configure a separate monitoring cluster for the UI, the rules (alerts) fail to see the data because the the Alerting Framework's Elasticsearch client is connected to the same cluster as Kibana which is controlled by elasticsearch.*

Steps to reproduce:

1. Setup 2 Elasticsearch Clusters, prod and mon
2. Setup 1 Kibana instance where elasticsearch.* points to prod and monitoring.ui.elasticsearch.* points to mon
3. Ship monitoring data from prod to mon
4. Ship monitoring data from mon to mon
5. Visit Stack Monitoring and install alerts
6. Alerts should fail similarly to Stack Monitoring rule types failing due to empty buckets #120111

Expected behavior:

Stack monitoring alerts should be using the Elasticsearch client provided by the Stack Monitoring plugin.

Any additional context:

This bug was introduced by: #93364

Here are the lines that introduced the issue:

21587dc#diff-18ba33bb18e63b018668abafa982855d9872797ec4a909cf5777a7c4e5d9991eL236-L253

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[sachin-frayne]

This is also affecting the monitoring.ui.elasticsearch.hosts, I cannot see monitoring data in my main Kibana from the data in the Elasticsearch Monitoring cluster.

[jasonrhodes]

I think we should officially deprecate monitoring.ui.elasticsearch.* as @matschaffer suggested here #120315 -- @ravikesarwani what do you think?

[simianhacker]

100% agree! I suspect that had CCS existed long ago, we would have never added that stuff.

[shubhu934]

@simianhacker Hope you are doing well. Did we find any other workaround apart from using 2 kibana instances for different clusters ? . One of our customers is hitting this same issue in kibana 7,17 ?.