Nodes not showing in Grid

[shawnpersona]

Version

2.4.10

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Distributed

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

8

RAM

32 GB

Storage for /

293 GB

Storage for /nsm

628 GB

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

I am trying to re-deploying security onion using the distributed architecture. I have deployed 1 Manager, 2 Search nodes, and 1 sensor node. Under administration I see my 4 nodes, as accepted, however if I look at Grid I only see my Manager node.

On the manager when I run so-status I see the following:

                         Container │ Status  │               Details
───────────────────────────────────┼─────────┼───────────────────────
                 so-dockerregistry │ running │             Up 4 days
                     so-elastalert │ running │             Up 4 days
                  so-elastic-fleet │ running │             Up 4 days
 so-elastic-fleet-package-registry │ running │   Up 4 days (healthy)
                  so-elasticsearch │ running │             Up 4 days
                       so-idstools │ running │             Up 4 days
                       so-influxdb │ running │   Up 4 days (healthy)
                         so-kibana │ running │             Up 4 days
                         so-kratos │ running │             Up 4 days
                       so-logstash │ running │             Up 4 days
                          so-nginx │ running │ Up 18 hours (healthy)
                          so-redis │ running │             Up 4 days
                      so-sensoroni │ running │             Up 4 days
                            so-soc │ running │             Up 4 days
                       so-telegraf │ running │             Up 4 days

 ✔ This onion is ready to make your adversaries cry!

However on my other 3 nodes when I run so-status I see the following

[sudo] password for manager:
so-status not yet available

I have re-deployed the nodes a few times using so-setup iso.
I am able to ping all nodes from each other.
I feel like there is maybe one command I am just missing I did run the following commands ips have been redacted:

     sudo so-firewall-minion --role=SEARCHNODE --ip=10.x.x.x
     sudo so-firewall-minion --role=SEARCHNODE --ip=10.x.x.x
     sudo so-firewall-minion --role=SENSOR --ip=10.x.x.x

below is status salt-minion on my sensor node

-minion-sensor ~]$ sudo systemctl status salt-minion
[sudo] password for x:
● salt-minion.service - The Salt Minion
     Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-11-18 21:26:48 UTC; 19h ago
       Docs: man:salt-minion(1)
             file:///usr/share/doc/salt/html/contents.html
             https://docs.saltproject.io/en/latest/contents.html
   Main PID: 44365 (python3.10)
      Tasks: 7 (limit: 98091)
     Memory: 84.4M
        CPU: 28.622s
     CGroup: /system.slice/salt-minion.service
             ├─44365 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
             └─44460 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager"

Nov 19 12:52:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119125221087536
Nov 19 13:17:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119131721087858
Nov 19 13:42:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119134221087687
Nov 19 14:07:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119140721088065
Nov 19 14:32:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119143221087536
Nov 19 14:57:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119145721087161
Nov 19 15:22:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119152221087681
Nov 19 15:47:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119154721087621
Nov 19 16:12:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119161221087402
Nov 19 16:37:21 do-minion-sensor salt-minion[44460]: [INFO    ] Running scheduled job: __mine_interval with jid 20241119163721087754

As you can see in the screenshots below administration shows all 4 node, however Grid just shows manager

I believe I have a bit more insight into what may be happening I am wondering if I need to run a firewall allow command on the non manager nodes to allow the manager to get back to them.

[sudo] password for manager:
do-minion-sensor_sensor:
    ERROR: Unable to run command '['/usr/sbin/so-status']' with the context '{'cwd': '/root', 'shell': False, 'env': {'PWD': '/', 'SYSTEMD_EXEC_PID': '44365', 'LANG': 'en_US.UTF-8', 'INVOCATION_ID': 'a9d20fcc19ad495eac33355a5020222e', 'SHLVL': '0', 'JOURNAL_STREAM': '8:93453', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'SSL_CERT_DIR': '/etc/pki/tls/certs', 'SSL_CERT_FILE': '/etc/pki/tls/cert.pem', 'LC_CTYPE': 'C', 'LC_NUMERIC': 'C', 'LC_TIME': 'C', 'LC_COLLATE': 'C', 'LC_MONETARY': 'C', 'LC_MESSAGES': 'C', 'LC_PAPER': 'C', 'LC_NAME': 'C', 'LC_ADDRESS': 'C', 'LC_TELEPHONE': 'C', 'LC_MEASUREMENT': 'C', 'LC_IDENTIFICATION': 'C', 'LANGUAGE': 'C'}, 'stdin': None, 'stdout': -1, 'stderr': -2, 'with_communicate': True, 'timeout': None, 'bg': False, 'close_fds': True}', reason: [Errno 8] Exec format error: '/usr/sbin/so-status'
securityonionsearchnode2_searchnode:
    ERROR: Unable to run command '['/usr/sbin/so-status']' with the context '{'cwd': '/root', 'shell': False, 'env': {'PWD': '/', 'SYSTEMD_EXEC_PID': '37482', 'LANG': 'en_US.UTF-8', 'INVOCATION_ID': '18d386e8a01d46b7b67a89c62484bb9a', 'SHLVL': '0', 'JOURNAL_STREAM': '8:63569', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'SSL_CERT_DIR': '/etc/pki/tls/certs', 'SSL_CERT_FILE': '/etc/pki/tls/cert.pem', 'LC_CTYPE': 'C', 'LC_NUMERIC': 'C', 'LC_TIME': 'C', 'LC_COLLATE': 'C', 'LC_MONETARY': 'C', 'LC_MESSAGES': 'C', 'LC_PAPER': 'C', 'LC_NAME': 'C', 'LC_ADDRESS': 'C', 'LC_TELEPHONE': 'C', 'LC_MEASUREMENT': 'C', 'LC_IDENTIFICATION': 'C', 'LANGUAGE': 'C'}, 'stdin': None, 'stdout': -1, 'stderr': -2, 'with_communicate': True, 'timeout': None, 'bg': False, 'close_fds': True}', reason: [Errno 8] Exec format error: '/usr/sbin/so-status'
securityonionsearchnode1_searchnode:
    ERROR: Unable to run command '['/usr/sbin/so-status']' with the context '{'cwd': '/root', 'shell': False, 'env': {'PWD': '/', 'SYSTEMD_EXEC_PID': '37507', 'LANG': 'en_US.UTF-8', 'INVOCATION_ID': 'b510b5a0cc83476a890d9cbe37f7f7a8', 'SHLVL': '0', 'JOURNAL_STREAM': '8:62645', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin', 'SSL_CERT_DIR': '/etc/pki/tls/certs', 'SSL_CERT_FILE': '/etc/pki/tls/cert.pem', 'LC_CTYPE': 'C', 'LC_NUMERIC': 'C', 'LC_TIME': 'C', 'LC_COLLATE': 'C', 'LC_MONETARY': 'C', 'LC_MESSAGES': 'C', 'LC_PAPER': 'C', 'LC_NAME': 'C', 'LC_ADDRESS': 'C', 'LC_TELEPHONE': 'C', 'LC_MEASUREMENT': 'C', 'LC_IDENTIFICATION': 'C', 'LANGUAGE': 'C'}, 'stdin': None, 'stdout': -1, 'stderr': -2, 'with_communicate': True, 'timeout': None, 'bg': False, 'close_fds': True}', reason: [Errno 8] Exec format error: '/usr/sbin/so-status'
securityonionmanager_manager:

                            Security Onion Status
                             Container │ Status  │               Details
    ───────────────────────────────────┼─────────┼───────────────────────
                     so-dockerregistry │ running │             Up 4 days
                         so-elastalert │ running │             Up 4 days
                      so-elastic-fleet │ running │             Up 4 days
     so-elastic-fleet-package-registry │ running │   Up 4 days (healthy)
                      so-elasticsearch │ running │             Up 4 days
                           so-idstools │ running │             Up 4 days
                           so-influxdb │ running │   Up 4 days (healthy)
                             so-kibana │ running │             Up 4 days
                             so-kratos │ running │             Up 4 days
                           so-logstash │ running │             Up 4 days
                              so-nginx │ running │ Up 19 hours (healthy)
                              so-redis │ running │             Up 4 days
                          so-sensoroni │ running │             Up 4 days
                                so-soc │ running │             Up 4 days
                           so-telegraf │ running │             Up 4 days

     ✔ This onion is ready to make your adversaries cry!
ERROR: Minions returned with non-zero exit code

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.