[Fleet] Global Output ES/Kibana default port are wrong for some environments

[kcm]

Kibana version: 7.12.0

Elasticsearch version: 7.12.0

Server OS version: Ubuntu 18.04 + ECE 2.9.0

Browser version: Firefox 88

Browser OS version: OS X 11.2.3

Original install method (e.g. download page, yum, from source, etc.): ECE Stack Pack

Describe the bug: Fleet Global Output defaults to port 443 for Elasticsearch output and specifies no port for Kibana output, but ESS/ECE use port 9243 for SSL/TLS connections to both.

Steps to reproduce:

1. Enable Fleet
2. Observe Global Output settings - may have to view the policy directly as the UI truncates the setting value
3. Agents can enroll using the correct URI and token
4. Agents then using this (default) policy will not be able to connect and are shown as Unhealthy

Expected behavior:

• Agent enrolls using correct URI and token
• Agent gets (default) policy with correct ports
• Agent is healthy and operates normally

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Default policy with incorrect ports: https://gist.github.com/kcm/2988d76eb94e7fe6b1931d7342a20551

Any additional context:

ESS defaults to port 9243, though traffic on 443 is accepted (I'm guessing it's just tunneled to port 9243), so this actually "works" there, albeit accidentally.

On both ESS and ECE I'd expect we explicitly specify port 9243 for ES and Kibana. I'm not sure how autodetection might work here, since Kibana may connect directly to an ES container in a different way than an external Agent would connect.

On prem would be 9200 and 5601, respectively.

However, since there's no way to perfectly default, nor know how external connections are preferred to connect, one part of this solution might be a UI/UX step where the user is shown and confirms the "default" way to connect to ES and Kibana, and can change it before it's used. While not all users will be technically advanced enough to correct when we guess wrong, it's still better than silently guessing for them.

One small note: it would be super helpful to be able to edit the Global Output UI setting values in the screenshot above. I'm told EUI can be used to make these editable, or at least copyable. If the user could re-edit the values, or at least copy/paste the content to a new value, it would make troubleshooting and fixing much more straightforward.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

@mostlyjason I know we've had some issues filed in the past wrt to auto-detection and population of ES host on Cloud. Can you remind me what they are and what you think about the priority for them?

[mostlyjason]

Fleet Global Output defaults to port 443 for Elasticsearch output and specifies no port for Kibana output

I thought this issue addressed it for the ECE scenario? https://github.com/elastic/cloud/issues/73898. It sounded like we were going why 9243 so I'm curious why its using 443? I can't see the rest of the screenshot since the text is truncated, but when you start a new deployment I thought we fill in these values with port 9243 automatically?

[jen-huang]

@nchaulet I wonder if we can apply a fix similar to #98492 for the ES URL too? It would be nice if we can pick it up from kibana.yml elasticsearch.hosts setting.

(Kibana URL is no longer needed from 7.13)

[AlexP-Elastic]

From discussion in #98670

ECE always (from 1.2+) specifies 9243, which is always injected into the cloud Id

Depending on the region, ESS either:

• specifies nothing - which results in the cloud Id not having a port, which should be treated a 443
• specifies 9243 - which results in the cloud Id getting an injected value of 9243 .. since both 9243 and 443 work in ESS this is fine (though I am creating an issue in Cloud for us to be consistent across all regions)

[kcm]

Thanks everyone for tackling this and especially @AlexP-Elastic for the diligence in summarizing the Cloud situation.

A little out of scope but it might be worth thinking about: what are the expectations of the dynamism here? If the user changes the Global Output settings, do any of the policies automatically change and update on the edge Agents? Maybe just the unmodified ones?

[nchaulet]

If the user change the settings all the policies are going to be updated and the change will be pushed to the agent. That is a little more clear in 7.13 we added that screen.

[jen-huang]

Hi @nchaulet, until elastic/elastic-agent#299 is fixed/agreed on, can we make a fix to always expose the Fleet Server port for Cloud? Regardless of if it is 443 or 9243. This will help us tremendously with testing.

[AlexP-Elastic]

If Cloud Id exists, shouldn't the fleet server port be treated as the value in the Cloud Id (and 443 if not present)?

[jen-huang]

@nchaulet Thanks for the work on this in #98957 and #99084. Could you double check to see if we also need to apply a port fix for ES?

[nchaulet]

Yes just tested and we have the same issue with ES, if we pass a ES host without PORT it's going to use 9200 as default, going to work on a fix.

[jen-huang]

@nchaulet Think we can close this out with #99240 now merged?

[nchaulet]

Yes I think we can close it