Fleet Server listens on loopback when TLS is disabled in v8.6.0

[antoineco]

Although the host is set to 0.0.0.0 in the config, fleet-server forcibly listens on 127.0.0.1:8220 since v8.6.0 when configured without TLS.

$ ss -tlpn
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port    Process                                                                         
LISTEN    0         4096             127.0.0.1:8220             0.0.0.0:*        users:(("fleet-server",pid=167,fd=12))                                         
LISTEN    0         4096             127.0.0.1:8221             0.0.0.0:*        users:(("fleet-server",pid=167,fd=14))                                         
LISTEN    0         4096             127.0.0.1:6789             0.0.0.0:*        users:(("elastic-agent",pid=7,fd=12))                                          
LISTEN    0         4096             127.0.0.1:6791             0.0.0.0:*        users:(("elastic-agent",pid=7,fd=10))                                          
LISTEN    0         4096            127.0.0.11:40239            0.0.0.0:*

The only possibly relevant log entries I could find are:

{"log.level":"info","@timestamp":"2023-01-18T11:42:17.160Z","message":"server listening","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"ecs.version":"1.6.0","service.name":"fleet-server","bind":"localhost:8220","rdTimeout":60000,"wrTimeout":600000,"@timestamp":"2023-01-18T11:42:17.16Z","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-18T11:42:17.160Z","message":"Exposed over insecure HTTP; enablement of TLS is strongly recommended","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"ecs.version":"1.6.0","service.name":"fleet-server","@timestamp":"2023-01-18T11:42:17.16Z","ecs.version":"1.6.0"}

When TLS is enabled, the "server listening" log entry shows a binding to 0.0.0.0:8200 as expected:

{"log.level":"info","@timestamp":"2023-01-17T14:13:57.314Z","message":"server listening","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"wrTimeout":600000,"ecs.version":"1.6.0","service.name":"fleet-server","bind":"0.0.0.0:8220","rdTimeout":60000,"ecs.version":"1.6.0"}

Until v8.5 (included), the host input was respected, even if TLS wasn't enabled.

[cmacknz]

I suspect this is a consequence of #1867

[rroemer]

+1 on this. Both host and port configured by the policy is not respected.
It might also be because they have refactored a lot around process handling here:
#1701

[michel-laterman]

Yes, it looks like this issue is somewhere in the elastic-agent setup. If I start in stand alone mode with no real config it binds to 0.0.0.0:8220 as expected:

11:34:10.225 WRN Exposed over insecure HTTP; enablement of TLS is strongly recommended ecs.version=1.6.0 service.name=fleet-server
11:34:10.225 DBG Listening on 0.0.0.0:8220 ecs.version=1.6.0 service.name=fleet-server

[michel-laterman]

I think the issue is caused in the bootstrapping segment here:

elastic-agent/internal/pkg/agent/cmd/enroll_cmd.go

Lines 397 to 405 in da1db71

	if c.options.FleetServer.Insecure {
	// running insecure, force the binding to localhost (unless specified)
	if c.options.FleetServer.Host == "" {
	c.options.FleetServer.Host = defaultFleetServerInternalHost
	}
	c.options.URL = fmt.Sprintf("http://%s:%d", host, port)
	c.options.Insecure = true
	return nil
	}

[michel-laterman]

The fleet-server's behaviour in

elastic-agent/internal/pkg/agent/cmd/enroll_cmd.go

Lines 397 to 405 in da1db71

	if c.options.FleetServer.Insecure {
	// running insecure, force the binding to localhost (unless specified)
	if c.options.FleetServer.Host == "" {
	c.options.FleetServer.Host = defaultFleetServerInternalHost
	}
	c.options.URL = fmt.Sprintf("http://%s:%d", host, port)
	c.options.Insecure = true
	return nil
	}

is intentional.

If the server is started with --fleet-server-insecure-http the host is set to localhost instead of 0.0.0.0.
It must be explicitly overridden by using the --fleet-server-host flag.

The host (and port) options that appear in the fleet-server integration settings are always ignored (is this correct @blakerouse )

[antoineco]

Closing because working as intended (see discussions in the linked PR #2198).

Note: in container mode, the host can also be configured using environment variables (FLEET_SERVER_HOST=0.0.0.0).