Issue with Fleet Agent deployment after upgrade from 2.4.90 to 2.4.100

[easy13]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

8

RAM

16

Storage for /

100

Storage for /nsm

200

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi guys,

Here my issue.
I have a distributed SO installation. It was running very well with a remote fleet attached.
I decided to setup a new fleet server to an other location.
Before that, I soup the SO servers from .90 to .100.
No issue with that.

I installed the new fleet server with the last .100 ISO. The integration did work perfectly and I could see it in the grid.

Then, I downloaded the new fleet agent and the tried a setup on a linux machine.

Here come the faillure:

Installation initiated, view install log for further details.

Installing in non-interactive mode.
[==  ] Service Started  [3s] Elastic Agent successfully installed, starting enrollment.
[==  ] Waiting For Enroll...  [3s] {"log.level":"info","@timestamp":"2024-09-13T10:10:53.756+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":517},"message":"Starting enrollment to URL: https://xxxxxxxx:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-13T10:10:54.067+0200","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":523},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://xxxxxxxxx:8220/ with exponential backoff (init 1s, max 10s)","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: Not Found
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.14/fleet-troubleshooting.html
[  ==] Uninstalled  [5s] Error uninstalling. Printing logs
2024-09-13T08:10:54.639Z        DEBUG   [install]       Loaded configuration from /root/so-elastic-agent_source/elastic-agent/elastic-agent.yml
2024-09-13T08:10:54.639Z        DEBUG   [install]       Merged configuration from /root/so-elastic-agent_source/elastic-agent/elastic-agent.yml into result
2024-09-13T08:10:54.639Z        DEBUG   [install]       Merged all configuration files from [/root/so-elastic-agent_source/elastic-agent/elastic-agent.yml], no external input files
2024-09-13T08:10:54.644Z        DEBUG   [install.composable]    Starting controller for composable inputs
2024-09-13T08:10:54.644Z        DEBUG   [install.composable]    Started controller for composable inputs
2024-09-13T08:10:54.653Z        DEBUG   [install.composable.providers.kubernetes]       Kubernetes provider for resource pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2024-09-13T08:10:54.653Z        DEBUG   [install.composable.providers.kubernetes]       Kubernetes provider for resource node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2024-09-13T08:10:54.658Z        DEBUG   [install.composable]    Kubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2024-09-13T08:10:54.658Z        DEBUG   [install.composable]    kubernetes_secrets provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2024-09-13T08:10:54.659Z        DEBUG   [install.composable]    Variable state changed for composable inputs; debounce started
2024-09-13T08:10:54.663Z        INFO    [install.composable.providers.docker]   Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-09-13T08:10:54.760Z        DEBUG   [install.composable]    Computing new variable state for composable inputs
2024-09-13T08:10:54.760Z        DEBUG   [install.composable]    Stopping controller for composable inputs
2024-09-13T08:10:54.859Z        DEBUG   [install.composable]    Stopped controller for composable inputs
Error: enroll command failed for unknown reason: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.14/fleet-troubleshooting.html
: exit status 1

And in the logs in SO-agent...:

timestamp=2024-09-13T12:01:05.240240444+02:00 level=info message="Version Information" ElasticAgentVersion=8.7.0 WrapperVersion=2.4.2
timestamp=2024-09-13T12:01:05.240616945+02:00 level=info message="Runtime Data" EnrollmentToken="YjZBa1M1RUJZSjZ0OFNjjjjjjjj6NFhRWnpTWHpRcVdCcVZYTUNtdnVmdw==" FleetURL/s=https://xxxxxxxxxx11:8220,https://xxxxxxxxxxt01:8220,https://xxxxxxx:8220,https://xxxxxxxxxx:8220
timestamp=2024-09-13T12:01:05.240637073+02:00 level=info message="Installation Progress" Status="Starting Installation Precheck"
timestamp=2024-09-13T12:01:08.241572802+02:00 level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://xxxxxxxxxxxxx:8220
timestamp=2024-09-13T12:01:11.242706416+02:00 level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://xxxxxxxxx:8220
timestamp=2024-09-13T12:01:14.243210115+02:00 level=warn message="Installation Progress" FleetHostConnectivityCheck=Failed FleetHostURL=https://xxxxxxxxxx:8220
timestamp=2024-09-13T12:01:14.266927854+02:00 level=info message="Installation Progress" FleetHostConnectivityCheck=Success FleetHostURL=https://xxxxxxxxxx:8220
timestamp=2024-09-13T12:01:14.267002352+02:00 level=info message="Installation Progress" Status="Fleet Host is accessible - Continuing installation"
timestamp=2024-09-13T12:01:14.267037011+02:00 level=info message="Installation Progress" Status="Installation Precheck Complete"
timestamp=2024-09-13T12:01:14.267060928+02:00 level=info message="Installation Progress" Status="Extracting Elastic Agent files"
timestamp=2024-09-13T12:01:23.100138188+02:00 level=info message="Installation Progress" Status="Executing Elastic Agent installer"
timestamp=2024-09-13T12:01:23.100340749+02:00 level=info message="Installation Progress" Status="Executing the following: ./so-elastic-agent_source/elastic-agent/elastic-agent install --url=https://xxxxxxxx:8220 --enrollment-token=YjZBa1M1RUJZSjZ0OFNzdlZPdEg6NFhRWmpkojijHpRcVdCcVZYTUNtdnVmdw== --certificate-authorities=/opt/Elastic/SO/soca.crt -n"
timestamp=2024-09-13T12:01:31.357148078+02:00 level=error message="Installation Progress" Context="Installing in non-interactive mode.\n\r[    ] Copying install files  [0s] \r                                   \r\r[=== ] Successfully copied files  [4s] \r                                       \r\r[=== ] Installing service  [4s] \r                                       \r\r[====] Installed service  [5s] \r                                       \r\r[====] Starting Service  [5s] \r                                       \r\r[====] Service Started  [5s] Elastic Agent successfully installed, starting enrollment.\n\r                                       \r\r[====] Enrolling Elastic Agent with Fleet  [5s] \r                                                \r\r[====] Waiting For Enroll...  [5s] {\"log.level\":\"info\",\"@timestamp\":\"2024-09-13T12:01:29.693+0200\",\"log.origin\":{\"file.name\":\"cmd/enroll_cmd.go\",\"file.line\":517},\"message\":\"Starting enrollment to URL: https://xxxxxxxxxxxxxx:8220/\",\"ecs.version\":\"1.6.0\"}\n{\"log.level\":\"info\",\"@timestamp\":\"2024-09-13T12:01:30.007+0200\",\"log.origin\":{\"file.name\":\"cmd/enroll_cmd.go\",\"file.line\":523},\"message\":\"1st enrollment attempt failed, retrying enrolling to URL: https://xxxxxxx:8220/ with exponential backoff (init 1s, max 10s)\",\"ecs.version\":\"1.6.0\"}\nError: fail to enroll: fail to execute request to fleet-server: Not Found\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.14/fleet-troubleshooting.html\n\r                                                \r\r[=   ] Failed to Enroll  [6s] \r                                                \r\r[=   ] Stopping Service  [6s] \r                                                \r\r[   =] Successfully Stopped Service  [6s] \r                                                \r\r[   =] Uninstalling  [6s] \r                                                \r\r[====] Stopping service  [6s] \r                                                \r\r[====] Successfully stopped service  [6s] \r                                                \r\r[==  ] Stopping upgrade watcher; none found  [7s] \r                                                  \r\r[  ==] Removing service  [7s] \r                                                  \r\r[====] Successfully uninstalled service  [7s] \r                                                  \r\r[====] Removing install directory  [7s] \r                                                  \r\r[=== ] Removed install directory  [7s] \r                                                  \r\r[=== ] Uninstalled  [7s] Error uninstalling. Printing logs\n2024-09-13T10:01:30.699Z\tDEBUG\t[install]\tLoaded configuration from /root/so-elastic-agent_source/elastic-agent/elastic-agent.yml\n2024-09-13T10:01:30.699Z\tDEBUG\t[install]\tMerged configuration from /root/so-elastic-agent_source/elastic-agent/elastic-agent.yml into result\n2024-09-13T10:01:30.699Z\tDEBUG\t[install]\tMerged all configuration files from [/root/so-elastic-agent_source/elastic-agent/elastic-agent.yml], no external input files\n2024-09-13T10:01:30.699Z\tDEBUG\t[install.composable]\tStarting controller for composable inputs\n2024-09-13T10:01:30.700Z\tDEBUG\t[install.composable]\tStarted controller for composable inputs\n2024-09-13T10:01:30.702Z\tDEBUG\t[install.composable]\tVariable state changed for composable inputs; debounce started\n2024-09-13T10:01:30.708Z\tDEBUG\t[install.composable]\tKubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable\n2024-09-13T10:01:30.710Z\tDEBUG\t[install.composable]\tkubernetes_secrets provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable\n2024-09-13T10:01:30.710Z\tDEBUG\t[install.composable.providers.kubernetes]\tKubernetes provider for resource pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable\n2024-09-13T10:01:30.710Z\tDEBUG\t[install.composable.providers.kubernetes]\tKubernetes provider for resource node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable\n2024-09-13T10:01:30.724Z\tINFO\t[install.composable.providers.docker]\tDocker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n2024-09-13T10:01:30.803Z\tDEBUG\t[install.composable]\tComputing new variable state for composable inputs\n2024-09-13T10:01:30.803Z\tDEBUG\t[install.composable]\tStopping controller for composable inputs\n2024-09-13T10:01:30.903Z\tDEBUG\t[install.composable]\tStopped controller for composable inputs\nError: enroll command failed for unknown reason: exit status 1\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.14/fleet-troubleshooting.html\n" ErrorDetails="exit status 1"

For your information, I deleted the fleet server and reinstalled it (and clean the grid between these tasks)
No change...

Thanks a lot for your help!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

Does endpoint your trying to install the elastic agent on have access to a fleet server on the ports listed under "Elastic Agents"? https://docs.securityonion.net/en/2.4/firewall.html#node-communication

You can try connecting from your linux host using

nc -zv <fleet ip> 8220 

[easy13]

Hi Reyesj2

Yes it does. I can see it on my FW logs.
No reject at all.

nc -zv xxxx 8220
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to xxxxxxxx:8220.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

The dns entry is working too

Thank you

[reyesj2]

Is that the same for the other ports specified in the docs? 8443 & 5055?

In the elastic agent install log it shows one of the urls is accessible to the agent. Is that a hostname or ip? Can the hostname be resolved by the endpoint?

[easy13]

Hi Rejes2

The nc is ok for the 3 ports.

I think I will setup the fleet server again.

Do we have to make something before attaching a new "member" ?
I installed the fleet from the 2.4.100 iso, and had no issue to connect it...

That part when I try to install the agent seems weird.
Kubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Thanks for your help.

[reyesj2]

The only thing required before adding an agent is ensuring that the endpoints IP is in the Security Onion firewall config outlined here: https://docs.securityonion.net/en/2.4/elastic-agent.html#deployment

Is the new fleet server showing all OK when you run so-status ?

Does the new fleet server have access to the Security Onion manager on ports below? Can you verify that?

TCP/443 - Sensoroni
TCP/5000 - Docker registry
TCP/8086 - influxdb
TCP/4505 - Salt
TCP/4506 - Salt

TCP/8220 (All nodes to Manager, Fleet nodes) - Elastic Agent management
TCP/8443 (All nodes to Manager) - Elastic Agent binary updates
TCP/5055 (All nodes to Manager, Fleet nodes, Receiver nodes) - Elastic Agent data

TCP/9200 - Node-to-node for Elasticsearch
TCP/5056 - Logstash-to-Logstash for Elastic Agent data ingest

[easy13]

Hi Reyes,

Everythings seems to be ok on the fleet server:

[x@fleet ~]$ nc -zv manager 443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:443.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

[x@fleet ~]$ nc -zv manager 5000
Ncat: Version 7.92 ( https://nmap.org/ncat )
.Ncat: Connected to ip_a:5000.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

[x@fleet ~]$ nc -zv manager 8086
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:8086.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

[x@fleet ~]$ nc -zv manager 4505
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:4505.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

[x@fleet ~]$ nc -zv manager 4506
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:4506.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

[x@fleet ~]$ nc -zv manager 8220
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:8220.
Ncat: 0 bytes sent, 0 bytes received in 0.08 seconds.

[x@fleet ~]$ nc -zv manager 8443
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:8443.
Ncat: 0 bytes sent, 0 bytes received in 0.06 seconds.

[x@fleet ~]$ nc -zv manager 5055
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:5055.
Ncat: 0 bytes sent, 0 bytes received in 0.06 seconds.

[x@fleet~]$ nc -zv manager 9200
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:9200.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.

[x@fleet ~]$ nc -zv manager 5056
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to ip_a:5056.
Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.


so-status

              Security Onion Status
        Container │ Status  │             Details
──────────────────┼─────────┼─────────────────────
 so-elastic-fleet │ running │           Up 7 days
      so-logstash │ running │           Up 7 days
         so-nginx │ running │ Up 7 days (healthy)
     so-sensoroni │ running │           Up 7 days
      so-telegraf │ running │           Up 7 days

 ✔ This onion is ready to make your adversaries cry!

Thanks again

[easy13]

I removed the fleet server from the grid and reboot the manager (and the other members to be sure)
In the Fleet dashboard, the agent policy of the deleted fleet server is stil availlable.

I try to delete it, but I have a pop up saying "Policy in use: 4 agents are assigned. Unassign these agents before deleting the policy"

It's weird, as there is no agent and no integrations attached.
If I click on it, It say "add agent", so I'm pretty sure of that.

How can I force the erasure?

Thank you

[easy13]

Hi
I did install a new fleet server.
Different IP (same network)

Accepted and present in the grid
A new policy pops up.

All firewall rules are setup as asked.

fleet client (8220. 8443. 5055) >> fleet server (all) >> manager search

Same error than before from 3 differents servers (2 linux, one windows)...
The windows client says it can not reach the fleet server but:

• I can see the packet on my fw
• I can see the packet with tcpdump on the fleet server.

Thanks for you help

[JhonShell]

Hi,
1 go to your manager and fleet setting heck this:

then check that those port are open with nc -v ip:port , check tcpdump on 5055 traffic to check if the traffic is coming

[easy13]

Hi,

I got this with tcpdump on the manager.

On my firewall I have a "reseted by peer" message.

That 's weird as I have the IP of the fleet server in the fleet firewall rules on the manager_search

Thanks

[easy13]

Hi

I have found a solution for my issue...
I have a proxy on my linux client (also the same on the windows one)
As the fleet server do not have any domain name, it was not filterd out.

In my linux system, I just removed the proxy entry in my .bash_profile and it worked :)

Thanks