Kibana seems to be required for APM Server since 8.x

[naemono]

Our docs indicate that you can use APM Server by just specifying the elasticsearchRef. This may have been true pre 8.x, but currently if just an elasticsearchRef is configured the APM Server logs these lines:

{"log.level":"error","@timestamp":"2024-02-08T14:25:11.657Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [metrics-apm.service_summary.60m] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [metrics-apm.service_summary.60m] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}

And is not usable.

Unfortunately our readiness probe:

    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 8200
        scheme: HTTPS

returns a 200 in this scenario:

❯ curl -vk https://localhost:8200/
---cut---
> GET / HTTP/2
> Host: localhost:8200
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 200
< x-content-type-options: nosniff
< content-length: 0
< date: Thu, 08 Feb 2024 14:29:19 GMT
<
* Connection #0 to host localhost left intact

We should:

1. Update our documentation to indicate this requirement since 8.x.

We should discuss whether we should not mark APM Server as health green if this api call does not come back healthy (which appears to be the case in this scenario, but we should double check that this call works with a properly configured/healthy APM Server):

❯ curl -H "Authorization: Bearer --secret-token--" -vk https://localhost:8200/config/v1/agents\?service.name\=test-service
*   Trying [::1]:8200...
---cut---
> GET /config/v1/agents?service.name=test-service HTTP/2
> Host: localhost:8200
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 503
< cache-control: max-age=300, must-revalidate
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 86
< date: Thu, 08 Feb 2024 14:34:02 GMT
<
{
  "error": "Agent configuration infrastructure is not ready. Please retry later."
}