[Fleet] Prevent output changing types from Elasticsearch -> Logstash if a Fleet Server policy uses it

[kpollich]

Fleet Server can only output to Elasticsearch. We've made one key change in #148896 wherein a policy containing the Fleet Server integration cannot have its output set to a Logstash output. However, we need to ensure that the same validation applies when an existing output is edited to have its type changed from Elasticsearch to Output.

If a Fleet Server policy exists that's pointing at an output, prevent its type attribute from being changed off of Elasticsearch. Users should be creating a separate output for their agents that output to Logstash, and this will prevent breaking their Fleet environment.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

I think we need to go a bit further than just validation here. We allow customers to set a Logstash output as their default output, which breaks any Fleet Server policies. So I think there's two things:

• Setting the default output to Logstash should never change policies with Fleet Server, those should get switched over to the previous ES output
• We need to allow an exception in the licensing code so that basic users can set their default output to Logstash but still have the Fleet Server policy attached to an ES output.

[ceeeekay]

We need to allow an exception in the licensing code so that basic users can set their default output to Logstash but still have the Fleet Server policy attached to an ES output.

This please. My architecture requires a Logstash output for agents, but I only have a basic license, and Fleet server 8.6 is unable to start with this config.

[lduvnjak]

@elastic/fleet
Any ETA on this fix? We can't use our fleet agents at the moment on version 8.6.

[kpollich]

Hi folks, we've got this prioritized internally for our next sprint. 8.7.0 might be a bit of stretch to land the fix, but definitely expect a fix to land in 8.7.1 and 8.8.0.

@juliaElastic @criamico - Do you think we could take a look at this next week to see if we can get a backport to 8.7.0 done? We're passed feature freeze but this would be an impactful bugfix so should be acceptable to land on the release branch.

[lucabelluccini]

Is the following one a valid workaround?

• deploy the Fleet Server with ES output
• switch it over to Logstash output (both data & monitoring)

[ceeeekay]

Confirming the fix works great for me in 8.7.0. Thanks!

There's a cosmetic issue that the new dialog appears to be reporting an incorrect number of agents and policies (zero), and there's a missing warning graphic:

Basic license here, if that's relevant at all.

[jen-huang]

@ceeeekay Thanks for the report! For the incorrect count, we have a bug logged here: #152425

For the missing icon, I was able to produce this on a new 8.7 instance. I will follow up with our UI library team about this, very strange.

Edit: I put up a fix for the icon issue here: #154119

[defensivedepth]

Using 8.7.0: Creating a Fleet Server Policy via the API is successful (screencap 1). Viewing the newly created Policy in the UI, it appears it has still selected the default Output, which is Logstash. (screencap 2)

If I specify the output in the API request with: "data_output_id":"output_elasticsearch"

I get the following error:

{"statusCode":400,"error":"Bad Request","message":"Invalid licence to set per policy output, you need platinum licence"}

Is this expected behavior?

[lduvnjak]

@defensivedepth, this might not help, but try adding an agent under that fleet server while having Elasticsearch as the default output, and switch to Logstash afterwards.
I can't test rn, but I remember having the same issue as you, until I added an agent, and only then switched the default output to Logstash.
Might have just been a one-off thing and not related though

[defensivedepth]

I was able to work around by creating the Policy via the API with the Logstash output and then updating the Policy through the API with "data_output_id":"output_elasticsearch"

[ceeeekay]

Not sure if I should raise this here or open a new issue, but I'm testing the Synthetics integration via a Logstash output, and it's failing with "Output not supported", which seems like the same behaviour as the Fleet server.

Basic license so I have no option to configure the Synthetics node with Elasticsearch output.

"message": "1 or more components/units in a failed state",
    "components": [
        {
            "id": "synthetics/tcp-default",
            "name": "",
            "state": 4,
            "message": "output not supported",
            "units": [
                {
                    "unit_id": "synthetics/tcp-default-synthetics/tcp-synthetics-6efbbe9d-b8b0-4a52-bc67-de69a8398f18-4d5a3bd0-c15a-11ed-bb49-0d174a3c4683-fleet",
                    "unit_type": 0,
                    "state": 4,
                    "message": "output not supported"
                },
                {
                    "unit_id": "synthetics/tcp-default",
                    "unit_type": 1,
                    "state": 4,
                    "message": "output not supported"
                }
            ],
            "version_info": {
                "name": "",
                "version": ""
            }
        }
[...]

Aside from the Basic license output limitations, requiring an Elasticsearch output would prevent me from deploying Synthetics monitoring to various locations in our network that can't access ES directly.

Any idea whether this is the intended behaviour?

Stack version: 8.7.0
Synthetics version: 0.11.8

[paulb-elastic]

Hello @ceeeekay this is by design, Heartbeat does need a direct connection to Elasticsearch when using Synthetics.

We have an issue to make this clearer in the docs, as it's not right now. Sorry you hit this issue.

As you're using 8.7.0, I assume you're using Private Locations, and not the legacy Synthetics Integration? If not, then you need to switch over to Private Locations (any Synthetics Integration monitors will stop working in 8.8)

[ceeeekay]

Hi @paulb-elastic, thanks for the info.

I was testing with Private Locations in the Uptime/Synthetics beta section. No real problem if a direct ES connection is required. I was hoping to just be able to continue by ingesting Synthetics events to my existing Logstash architecture, but I'm sure I can work around it this be deploying dedicated nodes which have direct access to ES.

I guess the real issue now is how to get this to work with a Basic license, given the outputs limitation and the requirement to be enrolled in Fleet. Are there any plans to enforce Synthetics ES output in the same way that Fleet Server does since 8.7, regardless of licensing?

Cheers.

[paulb-elastic]

@ceeeekay there is the limitation as you've mentioned. I've raised #155725 for us to consider removing this restriction. I can't give any indication yet when that might be able to be worked on, but you could watch that issue to keep track of it.