Elastic Agent and Heartbeat Integration - Phase 1 (HTTP/ICMP/TCP Ping)

[andrewvc]

We'd like to be able to centrally manage monitors in Uptime via the UI vs using the Heartbeat YML as is done today. We can look at the Security app for inspiration, an Uptime monitor is somewhat analogous to a security policy.

UI ACs:

1. Ability to perform CRUD (Create Read Update Delete) operations on all monitors from within the Uptime app.
2. Ability to deploy "Uptime" integration to any agent
3. Add use heartbeat under the beats agent

• Add heartbeat to agent
  • Support for per monitor index [Heartbeat] Add support for per monitor instance index beats#20602
  • Support basic configs in agent: Revert "[Agent] Remove heartbeat from bundled beats in agent." beats#22695 (comment)
  • Using synthetics/* types should not change the monitor.type field: [Heartbeat] Refactor config system beats#23467
  • Add synthetics prefix and define index strategy (names of the data streams) Add new index for synthetics elasticsearch#61665
• Build IM Package for Synthetics [Uptime] synth package integrations#241
• [Design] Add custom UI to Uptime [Design] Ingest Management MVP #236
• Add synthetics index to ES Add new index for synthetics elasticsearch#61665
• Determine what to do about index patterns in kibana (at minimum update default to include synthetics-*) [Uptime] Support agent data streams by default in Uptime kibana#91031
• Add friendly names to index pattern

Key flows:

We can divide user activities here into two parts:

1. Maintaining infrastructure: Configuring IM agents and policies, deploying servers, etc.
2. Actually using Uptime: Creating uptime monitors (IM integrations)

The infrastructure maintenance part only really needs to be done once, or as needed for maintenance. This is the process of adding a new server to run agent/heartbeat on, enrolling it, and creating a policy for it. The second part is the day-to-day work of creating, editing, updating, and deleting monitors. In cloud we can probably make the first part mostly invisible by doing this work for the user.

So, a user when setting up Uptime on prem would first:

1. Provision any physical servers needed for agent/heartbeat
2. Go to IM
3. Add a policy
4. Enroll agents to be used for Uptime
5. Add these agents to the policy they created

At that point they would be able to perform CRUD actions on monitors, either with the Uptime UI or the integrations package directly in IM.

On cloud, users won't need to deal with any of the provisioning stuff, and could simply use the Uptime app directly.

[paulb-elastic]

To claify, this is Phase 1 of the Elastic Agent and Heartbeat Integration, to support HTTP Ping integration end-to-end

As an Elastic user
I want to be able to add an HTTP Ping Heartbeat monitor to an Elastic Agent via an integration
So that I can centrally deploy the monitor without having to install and configure a separate Heartbeat Agent binary and YML file

ACs:

• User can add a Heartbeat HTTP Ping Integration (in the same way they can other integrations)
• User can add a monitor, with the configuration of
  • Name
  • URL
  • Frequency
• User can define which Elastic Agent, the integration will run on
• The HTTP Ping will run on the given URL, at the given frequency, and results will appear in Uptime
• In Uptime, there is a mechanism that allows me to configure (CRUD) these monitors

Known next steps:

• Blocked by Handle agent configs beats#20913 (Heartbeat needs to take configs from Agent)
• Re add the Synthetics integration: re-adding synthetic package integrations#406 (once Handle agent configs beats#20913 done, this can be merged)
• Add monitor PR from the Uptime UI: [Uptime] Add central management kibana#75490

[paulb-elastic]

With the custom UI being put together in elastic/kibana#91062, phase 1 is being extended to include HTTP Ping, ICMP Ping and TCP Ping monitor types.