Security onion manager getting inaccessible after 600 sigma rules added #13940

JhonShell · 2024-11-13T14:35:09Z

JhonShell
Nov 13, 2024

Version

2.4.80

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

512

Storage for /

100

Storage for /nsm

1tb

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hello,
After 5 months working with security onion, the manager seems to get inaccessible even though ssh, every day after midnight, I would like to know what could possibly be happening, all of it start happing after our analyst added 600 sigma rules in a couple of day.

Security Onion Log Check - Wed Nov 13 02:20:00 PM UTC 2024

RECENT_LOG_LINES: 200
EXCLUDE_STARTUP_ERRORS: N
EXCLUDE_FALSE_POSITIVE_ERRORS: N
EXCLUDE_KNOWN_ERRORS: N

=========================================================================
Checking container "so-kratos"

=========================================================================
Checking container "so-dockerregistry"

time="2024-11-05T23:03:13.955471066Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2024-11-06T23:03:14.077232835Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2024-11-07T23:03:14.270647486Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2024-11-08T16:15:29.407082245Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2024-11-09T16:45:39.308283809Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2024-11-11T14:27:14.851161682Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"
time="2024-11-12T15:22:57.176502635Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=0"

=========================================================================
Checking log file /opt/so/log/sostatus/status.log

=========================================================================
Checking log file /opt/so/log/kratos/kratos-migrate.log

{"audience":"application","level":"debug","migration_file":"migrations/sql/20221220124639000000_errors_index.up.sql","migration_name":"errors_index","msg":"Migration has already been applied, skipping.","service_name":"Ory Kratos","servi
ce_version":"v1.1.0","time":"2024-11-13T14:15:44.974934846Z","version":"20221220124639000000"}

Checking log file /opt/so/log/influxdb/influxdb.log

ts=2024-11-13T01:51:37.317396Z lvl=info msg="http: TLS handshake error from 10.1.2.52:43898: write tcp 172.17.1.26:8086->10.1.2.52:43898: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317409Z lvl=info msg="http: TLS handshake error from 172.17.1.1:43112: write tcp 172.17.1.26:8086->172.17.1.1:43112: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317444Z lvl=info msg="http: TLS handshake error from 10.1.2.52:39460: write tcp 172.17.1.26:8086->10.1.2.52:39460: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317461Z lvl=info msg="http: TLS handshake error from 10.1.2.55:54626: write tcp 172.17.1.26:8086->10.1.2.55:54626: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317476Z lvl=info msg="http: TLS handshake error from 10.1.2.54:56532: write tcp 172.17.1.26:8086->10.1.2.54:56532: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317488Z lvl=info msg="http: TLS handshake error from 10.1.2.52:33490: write tcp 172.17.1.26:8086->10.1.2.52:33490: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317501Z lvl=info msg="http: TLS handshake error from 10.1.2.55:53780: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317514Z lvl=info msg="http: TLS handshake error from 172.17.1.1:51820: write tcp 172.17.1.26:8086->172.17.1.1:51820: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317544Z lvl=info msg="http: TLS handshake error from 10.1.2.54:57276: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317558Z lvl=info msg="http: TLS handshake error from 10.1.2.52:55930: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317572Z lvl=info msg="http: TLS handshake error from 10.1.2.55:32808: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317586Z lvl=info msg="http: TLS handshake error from 10.1.2.54:49216: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317597Z lvl=info msg="http: TLS handshake error from 10.1.2.52:48910: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317609Z lvl=info msg="http: TLS handshake error from 10.1.2.55:44380: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317620Z lvl=info msg="http: TLS handshake error from 10.1.2.52:59664: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317631Z lvl=info msg="http: TLS handshake error from 10.1.2.55:36164: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317642Z lvl=info msg="http: TLS handshake error from 10.1.2.54:45976: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317652Z lvl=info msg="http: TLS handshake error from 10.1.2.52:60942: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317666Z lvl=info msg="http: TLS handshake error from 10.1.2.55:38610: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317678Z lvl=info msg="http: TLS handshake error from 10.1.2.54:50946: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317690Z lvl=info msg="http: TLS handshake error from 10.1.2.52:37994: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317700Z lvl=info msg="http: TLS handshake error from 10.1.2.55:51056: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317709Z lvl=info msg="http: TLS handshake error from 10.1.2.54:55414: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317723Z lvl=info msg="http: TLS handshake error from 10.1.2.52:37656: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317734Z lvl=info msg="http: TLS handshake error from 10.1.2.54:38418: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317745Z lvl=info msg="http: TLS handshake error from 10.1.2.52:58120: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.317756Z lvl=info msg="http: TLS handshake error from 10.1.2.54:51852: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:51:37.564246Z lvl=info msg="http: TLS handshake error from 10.1.2.55:60646: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:03.586266Z lvl=info msg="http: TLS handshake error from 10.1.2.52:38606: write tcp 172.17.1.26:8086->10.1.2.52:38606: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255575Z lvl=info msg="http: TLS handshake error from 10.1.2.54:36886: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255631Z lvl=info msg="http: TLS handshake error from 10.1.2.52:42170: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255645Z lvl=info msg="http: TLS handshake error from 10.1.2.55:60262: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255674Z lvl=info msg="http: TLS handshake error from 172.17.1.1:41606: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255731Z lvl=info msg="http: TLS handshake error from 10.1.2.54:46322: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255751Z lvl=info msg="http: TLS handshake error from 10.1.2.54:56456: write tcp 172.17.1.26:8086->10.1.2.54:56456: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255819Z lvl=info msg="http: TLS handshake error from 10.1.2.54:48366: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255837Z lvl=info msg="http: TLS handshake error from 10.1.2.52:53406: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255860Z lvl=info msg="http: TLS handshake error from 10.1.2.52:59494: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255872Z lvl=info msg="http: TLS handshake error from 10.1.2.55:37506: write tcp 172.17.1.26:8086->10.1.2.55:37506: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.255923Z lvl=info msg="http: TLS handshake error from 172.17.1.1:34822: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.703012Z lvl=info msg="http: TLS handshake error from 10.1.2.55:53704: write tcp 172.17.1.26:8086->10.1.2.55:53704: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:04.939318Z lvl=info msg="http: TLS handshake error from 10.1.2.55:57484: write tcp 172.17.1.26:8086->10.1.2.55:57484: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:43.873665Z lvl=info msg="http: TLS handshake error from 10.1.2.54:47868: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:56.937666Z lvl=info msg="http: TLS handshake error from 10.1.2.52:58746: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219572Z lvl=info msg="http: TLS handshake error from 10.1.2.55:43412: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219630Z lvl=info msg="http: TLS handshake error from 10.1.2.54:46664: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219644Z lvl=info msg="http: TLS handshake error from 10.1.2.52:40354: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219661Z lvl=info msg="http: TLS handshake error from 10.1.2.55:48724: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219674Z lvl=info msg="http: TLS handshake error from 172.17.1.1:51806: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219755Z lvl=info msg="http: TLS handshake error from 10.1.2.54:54536: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219776Z lvl=info msg="http: TLS handshake error from 10.1.2.52:44364: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219787Z lvl=info msg="http: TLS handshake error from 172.17.1.1:54930: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219813Z lvl=info msg="http: TLS handshake error from 172.17.1.1:54946: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219840Z lvl=info msg="http: TLS handshake error from 10.1.2.55:33924: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219864Z lvl=info msg="http: TLS handshake error from 172.17.1.1:54950: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219895Z lvl=info msg="http: TLS handshake error from 10.1.2.54:33718: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219909Z lvl=info msg="http: TLS handshake error from 10.1.2.52:51328: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219920Z lvl=info msg="http: TLS handshake error from 10.1.2.55:48426: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219939Z lvl=info msg="http: TLS handshake error from 10.1.2.54:37672: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219955Z lvl=info msg="http: TLS handshake error from 10.1.2.52:52086: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219971Z lvl=info msg="http: TLS handshake error from 10.1.2.54:36794: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219983Z lvl=info msg="http: TLS handshake error from 10.1.2.52:34740: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.219994Z lvl=info msg="http: TLS handshake error from 10.1.2.54:37234: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.220023Z lvl=info msg="http: TLS handshake error from 10.1.2.52:55386: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.220045Z lvl=info msg="http: TLS handshake error from 10.1.2.55:48630: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.220058Z lvl=info msg="http: TLS handshake error from 10.1.2.52:48726: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.220180Z lvl=info msg="http: TLS handshake error from 10.1.2.55:52354: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.220244Z lvl=info msg="http: TLS handshake error from 10.1.2.54:40824: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.220264Z lvl=info msg="http: TLS handshake error from 10.1.2.55:38018: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:53:59.967801Z lvl=warn msg="internal error not returned to client" log_id=0spDaBAl000 handler=error_logger error="context canceled"
ts=2024-11-13T01:54:02.370600Z lvl=info msg="http: TLS handshake error from 10.1.2.55:45366: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:13.712557Z lvl=info msg="http: TLS handshake error from 10.1.2.55:54894: write tcp 172.17.1.26:8086->10.1.2.55:54894: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.018535Z lvl=info msg="http: TLS handshake error from 10.1.2.54:53086: write tcp 172.17.1.26:8086->10.1.2.54:53086: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.691255Z lvl=info msg="http: TLS handshake error from 10.1.2.52:40562: write tcp 172.17.1.26:8086->10.1.2.52:40562: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.691311Z lvl=info msg="http: TLS handshake error from 10.1.2.54:37804: write tcp 172.17.1.26:8086->10.1.2.54:37804: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.691363Z lvl=info msg="http: TLS handshake error from 10.1.2.52:53766: write tcp 172.17.1.26:8086->10.1.2.52:53766: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.691419Z lvl=info msg="http: TLS handshake error from 10.1.2.55:56350: write tcp 172.17.1.26:8086->10.1.2.55:56350: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.691490Z lvl=info msg="http: TLS handshake error from 172.17.1.1:39244: write tcp 172.17.1.26:8086->172.17.1.1:39244: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:18.493346Z lvl=warn msg="internal error not returned to client" log_id=0spDaBAl000 handler=error_logger error="context canceled"
ts=2024-11-13T01:54:20.023832Z lvl=info msg="http: TLS handshake error from 10.1.2.52:40292: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:20.414799Z lvl=info msg="http: TLS handshake error from 10.1.2.54:49260: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:20.453616Z lvl=info msg="http: TLS handshake error from 10.1.2.55:42618: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:25.000465Z lvl=info msg="http: TLS handshake error from 172.17.1.1:47704: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:25.512453Z lvl=info msg="http: TLS handshake error from 10.1.2.52:44524: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:42.309675Z lvl=info msg="http: TLS handshake error from 10.1.2.54:41882: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:42.464836Z lvl=info msg="http: TLS handshake error from 10.1.2.55:56172: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:44.551853Z lvl=info msg="http: TLS handshake error from 172.17.1.1:43354: write tcp 172.17.1.26:8086->172.17.1.1:43354: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:59.363031Z lvl=info msg="http: TLS handshake error from 10.1.2.55:53216: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:59.366482Z lvl=info msg="http: TLS handshake error from 10.1.2.54:54048: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:59.366523Z lvl=info msg="http: TLS handshake error from 172.17.1.1:44846: write tcp 172.17.1.26:8086->172.17.1.1:44846: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:54:59.437003Z lvl=info msg="http: TLS handshake error from 10.1.2.52:52836: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:55:09.479882Z lvl=info msg="http: TLS handshake error from 172.17.1.1:39916: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:55:12.770802Z lvl=info msg="http: TLS handshake error from 172.17.1.1:55468: write tcp 172.17.1.26:8086->172.17.1.1:55468: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:55:14.167112Z lvl=info msg="http: TLS handshake error from 10.1.2.55:56160: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:55:22.924909Z lvl=info msg="http: TLS handshake error from 10.1.2.55:37690: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:55:52.239234Z lvl=info msg="http: TLS handshake error from 172.17.1.1:39920: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:27.158689Z lvl=info msg="http: TLS handshake error from 172.17.1.1:44432: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850260Z lvl=info msg="http: TLS handshake error from 10.1.2.54:55342: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850320Z lvl=info msg="http: TLS handshake error from 10.1.2.52:57880: write tcp 172.17.1.26:8086->10.1.2.52:57880: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850354Z lvl=info msg="http: TLS handshake error from 10.1.2.52:49166: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850368Z lvl=info msg="http: TLS handshake error from 10.1.2.55:52964: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850381Z lvl=info msg="http: TLS handshake error from 10.1.2.54:35902: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850392Z lvl=info msg="http: TLS handshake error from 10.1.2.52:57998: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850412Z lvl=info msg="http: TLS handshake error from 10.1.2.54:40558: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850425Z lvl=info msg="http: TLS handshake error from 10.1.2.54:54966: write tcp 172.17.1.26:8086->10.1.2.54:54966: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850443Z lvl=info msg="http: TLS handshake error from 10.1.2.54:57708: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850457Z lvl=info msg="http: TLS handshake error from 10.1.2.52:58488: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850470Z lvl=info msg="http: TLS handshake error from 10.1.2.55:34918: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850490Z lvl=info msg="http: TLS handshake error from 10.1.2.54:47030: write tcp 172.17.1.26:8086->10.1.2.54:47030: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850502Z lvl=info msg="http: TLS handshake error from 10.1.2.54:47676: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850622Z lvl=info msg="http: TLS handshake error from 10.1.2.52:54678: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850668Z lvl=info msg="http: TLS handshake error from 172.17.1.1:54664: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850715Z lvl=info msg="http: TLS handshake error from 10.1.2.55:42226: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850729Z lvl=info msg="http: TLS handshake error from 10.1.2.54:55834: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850740Z lvl=info msg="http: TLS handshake error from 10.1.2.52:56948: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850751Z lvl=info msg="http: TLS handshake error from 10.1.2.55:54102: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850770Z lvl=info msg="http: TLS handshake error from 10.1.2.54:35114: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850782Z lvl=info msg="http: TLS handshake error from 10.1.2.52:38752: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850801Z lvl=info msg="http: TLS handshake error from 10.1.2.55:51560: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850820Z lvl=info msg="http: TLS handshake error from 10.1.2.54:36736: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850840Z lvl=info msg="http: TLS handshake error from 10.1.2.52:58934: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850852Z lvl=info msg="http: TLS handshake error from 10.1.2.55:53260: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850865Z lvl=info msg="http: TLS handshake error from 172.17.1.1:42258: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850905Z lvl=info msg="http: TLS handshake error from 10.1.2.54:43544: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850918Z lvl=info msg="http: TLS handshake error from 10.1.2.52:59826: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850931Z lvl=info msg="http: TLS handshake error from 10.1.2.55:50144: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850943Z lvl=info msg="http: TLS handshake error from 10.1.2.54:36840: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850960Z lvl=info msg="http: TLS handshake error from 10.1.2.52:33400: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850971Z lvl=info msg="http: TLS handshake error from 10.1.2.55:43152: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.850983Z lvl=info msg="http: TLS handshake error from 10.1.2.54:54700: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.851002Z lvl=info msg="http: TLS handshake error from 10.1.2.52:44282: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.851028Z lvl=info msg="http: TLS handshake error from 10.1.2.52:50038: write tcp 172.17.1.26:8086->10.1.2.52:50038: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.851049Z lvl=info msg="http: TLS handshake error from 10.1.2.55:53290: write tcp 172.17.1.26:8086->10.1.2.55:53290: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:57:28.851063Z lvl=info msg="http: TLS handshake error from 10.1.2.55:33054: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:58:02.861582Z lvl=info msg="http: TLS handshake error from 172.17.1.1:36338: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:58:48.602702Z lvl=info msg="http: TLS handshake error from 10.1.2.55:52682: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T01:59:19.551600Z lvl=info msg="http: TLS handshake error from 10.1.2.54:60572: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.407980Z lvl=info msg="http: TLS handshake error from 10.1.2.52:35910: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419250Z lvl=info msg="http: TLS handshake error from 10.1.2.55:41428: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419297Z lvl=info msg="http: TLS handshake error from 10.1.2.54:44736: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419314Z lvl=info msg="http: TLS handshake error from 10.1.2.55:52722: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419330Z lvl=info msg="http: TLS handshake error from 10.1.2.55:54868: write tcp 172.17.1.26:8086->10.1.2.55:54868: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419346Z lvl=info msg="http: TLS handshake error from 10.1.2.52:54864: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419359Z lvl=info msg="http: TLS handshake error from 10.1.2.54:39046: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419381Z lvl=info msg="http: TLS handshake error from 10.1.2.55:38692: write tcp 172.17.1.26:8086->10.1.2.55:38692: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419399Z lvl=info msg="http: TLS handshake error from 10.1.2.55:56078: write tcp 172.17.1.26:8086->10.1.2.55:56078: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419415Z lvl=info msg="http: TLS handshake error from 172.17.1.1:36340: write tcp 172.17.1.26:8086->172.17.1.1:36340: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419502Z lvl=info msg="http: TLS handshake error from 10.1.2.52:44774: write tcp 172.17.1.26:8086->10.1.2.52:44774: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419627Z lvl=info msg="http: TLS handshake error from 10.1.2.52:47414: write tcp 172.17.1.26:8086->10.1.2.52:47414: write: broken pipe" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419651Z lvl=info msg="http: TLS handshake error from 10.1.2.52:51640: write tcp 172.17.1.26:8086->10.1.2.52:51640: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419666Z lvl=info msg="http: TLS handshake error from 10.1.2.55:37734: write tcp 172.17.1.26:8086->10.1.2.55:37734: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419682Z lvl=info msg="http: TLS handshake error from 10.1.2.54:54842: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419703Z lvl=info msg="http: TLS handshake error from 10.1.2.54:46328: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419718Z lvl=info msg="http: TLS handshake error from 10.1.2.52:47862: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419729Z lvl=info msg="http: TLS handshake error from 10.1.2.52:40786: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419743Z lvl=info msg="http: TLS handshake error from 10.1.2.55:58362: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419755Z lvl=info msg="http: TLS handshake error from 10.1.2.55:38770: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419772Z lvl=info msg="http: TLS handshake error from 10.1.2.54:39168: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419784Z lvl=info msg="http: TLS handshake error from 10.1.2.52:53246: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.419804Z lvl=info msg="http: TLS handshake error from 10.1.2.54:55732: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420311Z lvl=info msg="http: TLS handshake error from 10.1.2.52:41250: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420357Z lvl=info msg="http: TLS handshake error from 10.1.2.55:42856: write tcp 172.17.1.26:8086->10.1.2.55:42856: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420374Z lvl=info msg="http: TLS handshake error from 172.17.1.1:45410: write tcp 172.17.1.26:8086->172.17.1.1:45410: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420439Z lvl=info msg="http: TLS handshake error from 10.1.2.54:36634: write tcp 172.17.1.26:8086->10.1.2.54:36634: i/o timeout" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420457Z lvl=info msg="http: TLS handshake error from 10.1.2.54:49054: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420587Z lvl=info msg="http: TLS handshake error from 10.1.2.52:44996: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420644Z lvl=info msg="http: TLS handshake error from 10.1.2.54:51160: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420691Z lvl=info msg="http: TLS handshake error from 10.1.2.52:58344: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420741Z lvl=info msg="http: TLS handshake error from 10.1.2.55:49342: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420792Z lvl=info msg="http: TLS handshake error from 10.1.2.54:40728: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420845Z lvl=info msg="http: TLS handshake error from 10.1.2.52:35982: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420897Z lvl=info msg="http: TLS handshake error from 10.1.2.55:56256: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.420970Z lvl=info msg="http: TLS handshake error from 10.1.2.54:51768: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421024Z lvl=info msg="http: TLS handshake error from 10.1.2.52:58656: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421076Z lvl=info msg="http: TLS handshake error from 10.1.2.55:51876: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421128Z lvl=info msg="http: TLS handshake error from 10.1.2.54:51600: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421179Z lvl=info msg="http: TLS handshake error from 10.1.2.52:51032: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421223Z lvl=info msg="http: TLS handshake error from 10.1.2.55:54278: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421271Z lvl=info msg="http: TLS handshake error from 10.1.2.54:33826: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421320Z lvl=info msg="http: TLS handshake error from 10.1.2.52:34248: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421364Z lvl=info msg="http: TLS handshake error from 10.1.2.55:51014: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421415Z lvl=info msg="http: TLS handshake error from 10.1.2.54:42924: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421466Z lvl=info msg="http: TLS handshake error from 10.1.2.52:54614: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421512Z lvl=info msg="http: TLS handshake error from 10.1.2.55:37786: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421558Z lvl=info msg="http: TLS handshake error from 10.1.2.54:32898: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421624Z lvl=info msg="http: TLS handshake error from 10.1.2.52:42206: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.421670Z lvl=info msg="http: TLS handshake error from 10.1.2.55:38818: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422028Z lvl=info msg="http: TLS handshake error from 10.1.2.54:47144: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422110Z lvl=info msg="http: TLS handshake error from 10.1.2.52:56972: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422165Z lvl=info msg="http: TLS handshake error from 10.1.2.55:35308: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422221Z lvl=info msg="http: TLS handshake error from 10.1.2.54:53838: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422268Z lvl=info msg="http: TLS handshake error from 10.1.2.52:42566: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422312Z lvl=info msg="http: TLS handshake error from 10.1.2.55:34474: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422353Z lvl=info msg="http: TLS handshake error from 10.1.2.54:43738: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422404Z lvl=info msg="http: TLS handshake error from 10.1.2.52:59118: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422495Z lvl=info msg="http: TLS handshake error from 10.1.2.55:53754: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422559Z lvl=info msg="http: TLS handshake error from 10.1.2.54:45778: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422597Z lvl=info msg="http: TLS handshake error from 10.1.2.52:34824: EOF" log_id=0spDaBAl000 service=http
ts=2024-11-13T02:02:53.422712Z lvl=info msg="http: TLS handshake error from 10.1.2.55:46508: EOF" log_id=0spDaBAl000 service=http

=========================================================================
Checking log file /opt/so/log/influxdb/setup.log

=========================================================================
Checking log file /opt/so/log/nginx/error.log

2024/11/13 00:28:36 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:29:18 [error] 29#29: *33638 upstream prematurely closed connection while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:29:19 [error] 28#28: *33599 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:29:23 [error] 28#28: *33599 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:29:23 [error] 29#29: *33638 auth request unexpected status: 502 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:31:07 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:31:10 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:31:10 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:31:18 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:33:05 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:33:06 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:33:06 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:33:52 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:35:01 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:35:01 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:35:01 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:35:44 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:36:53 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:36:53 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:36:53 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:37:38 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:38:39 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:38:39 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:38:39 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:39:21 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:40:27 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:40:27 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:40:27 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:41:12 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:42:07 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:42:17 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:42:17 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:42:53 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:43:47 [error] 29#29: *33618 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:43:47 [error] 30#30: *33688 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:43:47 [error] 29#29: *33687 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:44:24 [error] 29#29: *33620 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:44:56 [error] 30#30: *33688 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:44:56 [error] 29#29: *33618 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:44:56 [error] 29#29: *33620 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:44:56 [error] 29#29: *33687 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:46:37 [error] 29#29: *33761 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:46:39 [error] 30#30: *33762 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:46:45 [error] 29#29: *33765 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:46:54 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:46:53 [error] 29#29: *33768 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.3.1.140, server: 10.1.2.13, request: "GET /api/info HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:48:09 [error] 29#29: *33768 auth request unexpected status: 504 while sending to client, client: 10.3.1.140, server: 10.1.2.13, request: "GET /api/info HTTP/1.1", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:48:09 [error] 29#29: *33761 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:48:09 [error] 29#29: *33761 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:48:09 [error] 30#30: *33762 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:48:09 [error] 30#30: *33762 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:48:16 [error] 29#29: *33765 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:48:26 [error] 29#29: *33765 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:49:39 [error] 29#29: *33768 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.3.1.140, server: 10.1.2.13, request: "GET /api/info HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:49:43 [error] 29#29: *33768 auth request unexpected status: 504 while sending to client, client: 10.3.1.140, server: 10.1.2.13, request: "GET /api/info HTTP/1.1", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:49:56 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:49:58 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:50:00 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:50:17 [error] 30#30: *33785 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:51:10 [error] 30#30: *33788 upstream prematurely closed connection while reading response header from upstream, client: 10.3.1.140, server: 10.1.2.13, request: "GET /auth/self-service/login/browser HTTP/1.1", upstream: "http://10.1.2.13:4433/self-service/login/browser", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:51:48 [error] 31#31: *33795 upstream prematurely closed connection while reading response header from upstream, client: 10.3.1.140, server: 10.1.2.13, request: "GET /ws HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:51:49 [error] 31#31: *33795 auth request unexpected status: 502 while sending to client, client: 10.3.1.140, server: 10.1.2.13, request: "GET /ws HTTP/1.1", host: "10.1.2.13"
2024/11/13 00:51:51 [error] 30#30: *33802 upstream prematurely closed connection while reading response header from upstream, client: 10.3.1.140, server: 10.1.2.13, request: "GET /api/info HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:51:56 [error] 30#30: *33802 auth request unexpected status: 502 while sending to client, client: 10.3.1.140, server: 10.1.2.13, request: "GET /api/info HTTP/1.1", host: "10.1.2.13", referrer: "https://10.1.2.13/"
2024/11/13 00:51:57 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:52:00 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:52:05 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:52:28 [error] 30#30: *33785 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:52:56 [error] 31#31: *33795 upstream prematurely closed connection while reading response header from upstream, client: 10.3.1.140, server: 10.1.2.13, request: "GET /ws HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:52:57 [error] 31#31: *33795 auth request unexpected status: 502 while sending to client, client: 10.3.1.140, server: 10.1.2.13, request: "GET /ws HTTP/1.1", host: "10.1.2.13"
2024/11/13 00:53:49 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:53:51 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:53:51 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:54:41 [error] 30#30: *33785 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:55:43 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:55:43 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:55:43 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:56:12 [error] 30#30: *33785 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 00:56:16 [error] 30#30: *33785 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:58:18 [error] 30#30: *33840 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:58:26 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:58:26 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 00:58:26 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:00:10 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:00:10 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:00:11 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:00:12 [error] 30#30: *33840 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:01:42 [error] 29#29: *33780 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:01:42 [error] 30#30: *33781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:01:42 [error] 31#31: *33763 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:01:42 [error] 30#30: *33840 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:01:45 [error] 29#29: *33780 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:01:44 [error] 30#30: *33781 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:01:45 [error] 31#31: *33763 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:01:48 [error] 30#30: *33840 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:03:34 [error] 30#30: *33862 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:03:41 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:04:03 [error] 30#30: *33866 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:04:17 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:05:34 [error] 30#30: *33862 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:05:40 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:06:04 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:06:06 [error] 30#30: *33866 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:07:27 [error] 30#30: *33862 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:07:51 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:07:51 [error] 30#30: *33866 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:07:52 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:09:21 [error] 30#30: *33862 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:10:34 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:10:46 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:10:51 [error] 30#30: *33866 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:11:12 [error] 30#30: *33862 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:12:45 [error] 30#30: *33862 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:12:51 [error] 30#30: *33862 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:13:16 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:13:16 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:14:22 [error] 30#30: *33902 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:14:43 [error] 30#30: *33904 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:15:37 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:15:54 [error] 30#30: *33902 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:15:54 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:16:14 [error] 30#30: *33904 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:16:20 [error] 30#30: *33902 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:16:20 [error] 30#30: *33904 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:18:04 [error] 31#31: *33916 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:18:04 [error] 31#31: *33918 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:18:23 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:18:32 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:19:45 [error] 31#31: *33918 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:21:23 [error] 31#31: *33916 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:21:27 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:21:50 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:22:29 [error] 31#31: *33918 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:23:00 [error] 32#32: *33863 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:23:00 [error] 31#31: *33916 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:23:22 [error] 31#31: *33861 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:25:08 [error] 31#31: *33916 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:25:08 [error] 32#32: *33863 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:25:08 [error] 31#31: *33861 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:26:56 [error] 31#31: *33939 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:39:00 [error] 31#31: *33939 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:39:15 [error] 31#31: *33944 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:39:15 [error] 32#32: *33943 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:40:55 [error] 32#32: *33943 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:40:55 [error] 31#31: *33944 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:42:37 [error] 31#31: *33944 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:42:37 [error] 32#32: *33943 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:42:46 [error] 31#31: *33939 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:44:34 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:44:34 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:44:42 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:45:44 [error] 34#34: *33952 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:46:54 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:46:54 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:47:01 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:48:07 [error] 34#34: *33952 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:48:57 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:48:57 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:49:13 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:51:10 [error] 34#34: *33952 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:51:10 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:51:10 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:51:10 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:53:18 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:53:18 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:53:18 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:54:16 [error] 34#34: *33952 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:55:43 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:55:43 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:55:43 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:55:47 [error] 34#34: *33952 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 01:56:01 [error] 34#34: *33952 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:57:50 [error] 31#31: *34001 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:58:34 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:58:41 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:58:41 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 01:59:21 [error] 31#31: *34001 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 02:00:05 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 02:00:11 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 02:00:11 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 02:04:25 [error] 31#31: *34001 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 02:04:25 [error] 31#31: *34001 auth request unexpected status: 504 while sending to client, client: 10.1.2.13, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 02:05:21 [error] 32#32: *33955 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 02:05:21 [error] 32#32: *33955 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 02:44:50 [error] 31#31: *33953 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 02:44:50 [error] 31#31: *33953 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 02:44:50 [error] 31#31: *33954 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 02:44:50 [error] 31#31: *33954 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 03:31:56 [error] 34#34: *34014 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 03:34:01 [error] 31#31: *34016 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 03:35:01 [error] 32#32: *34017 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 03:55:29 [error] 31#31: *34016 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 03:55:22 [error] 34#34: *34014 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 03:55:29 [error] 32#32: *34017 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/sessions/whoami", upstream: "http://10.1.2.13:4433/sessions/whoami", host: "10.1.2.13"
2024/11/13 04:04:48 [error] 32#32: *34017 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 04:04:48 [error] 32#32: *34017 auth request unexpected status: 504 while sending to client, client: 10.1.2.55, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 04:07:54 [error] 31#31: *34016 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 04:07:54 [error] 31#31: *34016 auth request unexpected status: 504 while sending to client, client: 10.1.2.54, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"
2024/11/13 04:08:11 [error] 34#34: *34014 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", subrequest: "/50x.html", upstream: "http://10.1.2.13:9822/50x.html", host: "10.1.2.13"
2024/11/13 04:08:11 [error] 34#34: *34014 auth request unexpected status: 504 while sending to client, client: 10.1.2.52, server: 10.1.2.13, request: "POST /sensoroniagents/api/node HTTP/1.1", upstream: "http://10.1.2.13:9822/api/node", host: "10.1.2.13"

=========================================================================
Checking log file /opt/so/log/nginx/access.log

=========================================================================
Checking log file /opt/so/log/soc/sensoroni-server.log

{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion\": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:13:50.943617191Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion\": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:13:57.390050177Z","message":"Unable to determine latest value"}
{"fields":{"error":"search_phase_execution_exception: all shards failed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ],\n "type" : "search_phase_execution_exception",\n "reason" : "all shards failed",\n "phase" : "query",\n "grouped" : true,\n "failed_shards" : [\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-detections.alerts-so-2024.10.18-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent.endpoint_security-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n "},"level":"error","timestamp":"2024-11-13T01:14:12.661097927Z","message":"Failed to get all community SIDs"}
{"fields":{"error":"search_phase_execution_exception: all shards failed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ],\n "type" : "search_phase_execution_exception",\n "reason" : "all shards failed",\n "phase" : "query",\n "grouped" : true,\n "failed_shards" : [\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-detections.alerts-so-2024.10.18-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent.endpoint_security-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n "},"level":"error","timestamp":"2024-11-13T01:14:12.665711466Z","message":"unable to sync suricata community detections"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:14:14.498546786Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:14:45.715181126Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:16:09.638197481Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:16:27.204776924Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:16:32.399805634Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:16:37.404477986Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:16:46.433826971Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:16:53.405537719Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:16:59.791551374Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:17:21.410815395Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:17:35.79396444Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:17:55.547101941Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:18:20.805895431Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:18:34.933721377Z","message":"Unable to determine latest value"}
{"fields":{"detectionEngine":"strelka","error":"search_phase_execution_exception: all shards failed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ],\n "type" : "search_phase_execution_exception",\n "reason" : "all shards failed",\n "phase" : "query",\n "grouped" : true,\n "failed_shards" : [\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-detections.alerts-so-2024.10.18-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent.endpoint_security-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ","intCheckId":"1c36995e-3299-4631-9d0c-935a57187914"},"level":"error","timestamp":"2024-11-13T01:18:53.139213816Z","message":"unable to query for enabled detections"}
{"fields":{"engineName":"strelka","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2024-11-13T01:18:53.139314017Z","message":"integrity check repeat failure, alerting user"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:19:04.922601781Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:19:27.920200075Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:19:46.089488195Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:19:54.276491088Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:20:01.222183893Z","message":"Unable to determine latest value"}
{"fields":{"error":"Get "https://github.com/Security-Onion-Solutions/securityonion-yara/info/refs?service=git-upload-pack": dial tcp: lookup github.com on 127.0.0.11:53: read udp 127.0.0.1:54983-\u003e127.0.0.11:53: i/o timeout","repoPath":"/opt/sensoroni/yara/repos/securityonion-yara"},"level":"error","timestamp":"2024-11-13T01:20:05.321192435Z","message":"failed to clone repo, doing nothing with it"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:20:06.788610578Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:20:13.238232512Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:20:20.260903249Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:20:41.9576524Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:25:08.134876688Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:42:31.543614567Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": dial tcp: lookup manager: i/o timeout"},"level":"error","timestamp":"2024-11-13T01:42:42.45769386Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:42:55.996241101Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:44:11.083289272Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:44:16.221843562Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:44:36.547255278Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:45:01.932075053Z","message":"Unable to determine latest value"}
{"fields":{"sigmaConvertCode":0,"sigmaConvertCommand":"/usr/bin/sigma convert -t eql -p /opt/sensoroni/sigma_final_pipeline.yaml -p /opt/sensoroni/sigma_so_pipeline.yaml -p windows-logsources -p ecs_windows /dev/stdin","sigmaConvertError":null,"sigmaConvertExecTime":2075.957054513,"sigmaConvertOutput":"Parsing Sigma rules\nany where (process.executable like~ ("\\Users\\Public\\", "\\$Recycle.bin", "\\Users\\All Users\\", "\\Users\\Default\\", "\\Users\\Contacts\\", "\\Users\\Searches\\", "C:\\Perflogs\\", "\\config\\systemprofile\\", "\\Windows\\Fonts\\", "\\Windows\\IME\\", "\\Windows\\addins\\")) and (process.parent.executable like~ ("\\services.exe", "\\svchost.exe"))\n"},"level":"info","timestamp":"2024-11-13T01:45:21.678822129Z","message":"executing sigma cli"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:45:22.139210796Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:45:50.093358442Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:05.296529639Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:11.00372347Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:16.562914512Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:23.357367073Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:29.804429445Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:36.285745392Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:41.763821582Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:47.814596072Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:53.034028785Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:46:58.289877699Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:03.417110637Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:17.540593955Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:23.362885616Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:28.447845818Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:34.678314223Z","message":"Unable to determine latest value"}
{"fields":{"error":"search_phase_execution_exception: all shards failed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ],\n "type" : "search_phase_execution_exception",\n "reason" : "all shards failed",\n "phase" : "query",\n "grouped" : true,\n "failed_shards" : [\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-detections.alerts-so-2024.10.18-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent.endpoint_security-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n "},"level":"error","timestamp":"2024-11-13T01:47:35.30349999Z","message":"Failed to get all community SIDs"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:40.673998661Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:47:46.219780125Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:48:14.113969958Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:48:23.998431959Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:48:29.65788342Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:48:40.761166526Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:48:46.836931202Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:48:54.043701183Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:49:01.89679121Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:49:08.030081878Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:49:26.329384555Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:49:35.566502394Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:50:19.732932126Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:50:26.91656673Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:51:13.820157744Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": dial tcp: lookup manager: i/o timeout (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:51:34.596065193Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:52:02.968305617Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:52:47.731170748Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:53:06.754603271Z","message":"Unable to determine latest value"}
{"fields":{"error":"search_phase_execution_exception: all shards failed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ],\n "type" : "search_phase_execution_exception",\n "reason" : "all shards failed",\n "phase" : "query",\n "grouped" : true,\n "failed_shards" : [\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-detections.alerts-so-2024.10.18-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent.endpoint_security-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n "},"level":"error","timestamp":"2024-11-13T01:53:08.834176288Z","message":"unable to sync suricata community detections"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:53:15.378768199Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:53:27.76033893Z","message":"Unable to determine latest value"}
{"fields":{"error":"Get "https://github.com/Security-Onion-Solutions/securityonion-yara/info/refs?service=git-upload-pack": dial tcp: lookup github.com: i/o timeout","repoPath":"/opt/sensoroni/yara/repos/securityonion-yara"},"level":"error","timestamp":"2024-11-13T01:53:43.630122626Z","message":"failed to clone repo, doing nothing with it"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:54:00.27285977Z","message":"Unable to determine latest value"}
{"fields":{"error":"search_phase_execution_exception: all shards failed -\u003e {\n "error" : {\n "root_cause" : [\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n },\n {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n ],\n "type" : "search_phase_execution_exception",\n "reason" : "all shards failed",\n "phase" : "query",\n "grouped" : true,\n "failed_shards" : [\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-detections.alerts-so-2024.10.18-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n },\n {\n "shard" : 0,\n "index" : "manager:.ds-logs-elastic_agent.endpoint_security-default-2024.11.12-000001",\n "node" : null,\n "reason" : {\n "type" : "no_shard_available_action_exception",\n "reason" : null\n }\n "},"level":"error","timestamp":"2024-11-13T01:54:03.773615876Z","message":"Failed to get all community SIDs"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:54:13.961305442Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:54:26.646254572Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:54:50.000264707Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:55:01.971272987Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:55:07.282439158Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:55:22.046217785Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:56:04.891448608Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:56:54.295360013Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:57:23.513955586Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:57:56.242518336Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": net/http: TLS handshake timeout"},"level":"error","timestamp":"2024-11-13T01:58:08.285297337Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:58:39.887928786Z","message":"Unable to determine latest value"}
{"fields":{"error":"Post "https://manager:8086/api/v2/query?org=Security+Onion": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"},"level":"error","timestamp":"2024-11-13T01:59:21.895406637Z","message":"Unable to determine latest value"}

=========================================================================
Checking log file /opt/so/log/soc/salt-relay.log

=========================================================================
Checking log file /opt/so/log/soc/detections_runtime-status_sigma.log

=========================================================================
Checking log file /opt/so/log/soc/sync.log

=========================================================================
Checking log file /opt/so/log/soc/detections-backup.log

=========================================================================
Checking log file /opt/so/log/sensoroni/sensoroni.log

{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T00:09:33.802244771Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T00:09:33.804637192Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T00:12:53.665592572Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T00:12:53.693559112Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:15:31.750756542Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:18:18.327360111Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:20:47.475415068Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:22:51.43297906Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:24:53.580937015Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:26:46.661512418Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:29:33.469398948Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:32:12.451410392Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:34:01.609536473Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:35:52.167327841Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:37:41.743181031Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:39:29.085090569Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:41:13.522622678Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T00:44:56.625610391Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T00:44:56.756137802Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T00:48:27.780835569Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T00:48:28.044797016Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:50:46.762610164Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:52:58.308274059Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T00:56:17.498869345Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T00:56:37.02489526Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T00:58:27.358243306Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T01:01:50.585945963Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T01:02:05.91513054Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:04:25.007275377Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:06:11.199162274Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:08:58.178414772Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:12:34.788903515Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T01:16:20.370859404Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T01:16:20.506491214Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T01:22:30.038633535Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T01:22:30.207307119Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:46:25.580211623Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:49:29.403599363Z","message":"Failed to poll for pending jobs"}
{"fields":{"error":"Request did not complete successfully (401): 401 Unauthorized"},"level":"warn","timestamp":"2024-11-13T01:51:39.220373342Z","message":"Failed to poll for pending jobs"}
{"fields":{"contentLength":170,"method":"POST","status":"500 Internal Server Error","statusCode":500,"url":"https://10.1.2.13/sensoroniagents/api/node"},"level":"info","timestamp":"2024-11-13T01:56:01.700296294Z","message":"HTTP request finished"}
{"fields":{"error":"Request did not complete successfully (500): 500 Internal Server Error"},"level":"warn","timestamp":"2024-11-13T01:56:01.811680798Z","message":"Failed to poll for pending jobs"}

=========================================================================
Checking log file /opt/so/log/telegraf/telegraf.log

2024-11-13T01:48:57Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:49:01Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:49:02Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:49:07Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:49:15Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:49:15Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/_local/name": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:49:30Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:49:30Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:49:30Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/redis.sh":
2024-11-13T01:49:30Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:49:32Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:17: "sostatus status=,json="""
2024-11-13T01:49:32Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/lasthighstate.sh":
2024-11-13T01:49:57Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:49:57Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:50:04Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:50:08Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:50:08Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:17: "sostatus status=,json="""
2024-11-13T01:50:09Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:50:18Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:50:57Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:50:57Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:51:47Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:17: "sostatus status=,json="""
2024-11-13T01:51:47Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:51:49Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:51:49Z E! [agent] Error terminating process children: no such process
2024-11-13T01:51:49Z E! [agent] Error terminating process: os: process already finished
2024-11-13T01:51:49Z E! [agent] Error terminating process children: no such process
2024-11-13T01:51:49Z E! [agent] Error terminating process: os: process already finished
2024-11-13T01:51:54Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/_local/name": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:51:55Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:52:16Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:52:23Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:52:32Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:52:34Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:52:35Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:52:36Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:52:51Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_cluster/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:52:58Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:53:04Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:17: "sostatus status=,json="""
2024-11-13T01:53:08Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:53:09Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/_local/name": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:53:09Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:53:15Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:53:15Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:53:30Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/agentstatus.sh":
2024-11-13T01:53:31Z E! [agent] Error killing process: os: process already finished
2024-11-13T01:53:35Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/os.sh":
2024-11-13T01:53:55Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:53:56Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:54:35Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:54:35Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:54:35Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:55:01Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:55:04Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:55:04Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:55:18Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:55:24Z E! [agent] Error terminating process children: no such process
2024-11-13T01:55:25Z E! [agent] Error killing process: os: process already finished
2024-11-13T01:55:25Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_cluster/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:55:26Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/redis.sh":
2024-11-13T01:55:26Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:55:32Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:55:33Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:55:42Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:55:57Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/redis.sh":
2024-11-13T01:56:01Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/agentstatus.sh":
2024-11-13T01:56:02Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/_local/name": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:56:15Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:56:15Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/lasthighstate.sh":
2024-11-13T01:56:16Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:56:25Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/os.sh":
2024-11-13T01:56:25Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:56:25Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/sostatus.sh":
2024-11-13T01:56:31Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:56:35Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:56:49Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/redis.sh":
2024-11-13T01:56:53Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/features.sh":
2024-11-13T01:56:53Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/os.sh":
2024-11-13T01:56:53Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/raid.sh":
2024-11-13T01:56:55Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:57:01Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:19: "influxsize kbytes="
2024-11-13T01:57:14Z E! [inputs.elasticsearch] Error in plugin: context deadline exceeded (Client.Timeout or context cancellation while reading body)
2024-11-13T01:57:24Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:57:30Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:57:36Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:17: "sostatus status=,json="""
2024-11-13T01:57:38Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:19: "influxsize kbytes="
2024-11-13T01:57:38Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 1:20: "agentstatus online=,error=,inactive=,offline=,updating=,unenrolled=,other=,events=,total=,all=,active="
2024-11-13T01:57:49Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/_local/name": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:58:03Z E! [agent] Error killing process: os: process already finished
2024-11-13T01:58:19Z E! [inputs.exec] Error in plugin: exec: command timed out for command "/scripts/redis.sh":
2024-11-13T01:58:19Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/stats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:59:34Z E! [inputs.docker] Error in plugin: timeout retrieving docker engine info
2024-11-13T01:59:38Z E! [agent] Error writing to outputs.influxdb_v2: failed to send metrics to any configured server(s)
2024-11-13T01:59:40Z E! [inputs.elasticsearch] Error in plugin: Get "https://10.1.2.13:9200/_nodes/_local/name": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-11-13T01:59:40Z E! [inputs.docker] Error in plugin: timeout retrieving container list
2024-11-13T01:59:41Z E! [inputs.logstash] Error in plugin: Get "http://localhost:9600/_node/stats/pipelines": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

=========================================================================
Checking log file /opt/so/log/telegraf/influxdb_size.log

=========================================================================
Checking log file /opt/so/log/idstools/download_idstools_state.log

=========================================================================
Checking log file /opt/so/log/idstools/download_cron.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/cron-elasticsearch-indices-delete.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/so-elasticsearch-indices-delete.log

{"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (remove-data-stream [logs-endpoint.events.security-default]) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (remove-data-stream [logs-endpoint.events.security-default]) within 30s"},"status":503}
{"error":{"root_cause":[{"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (remove-data-stream [logs-fortinet_fortigate.log-default]) within 30s"}],"type":"process_cluster_event_timeout_exception","reason":"failed to process cluster event (remove-data-stream [logs-fortinet_fortigate.log-default]) within 30s"},"status":503}

=========================================================================
Checking log file /opt/so/log/elasticsearch/securityonion.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/indices-delete-alert.log

=========================================================================
Checking log file /opt/so/log/logstash/logstash-2024-09-24.log

=========================================================================
Checking log file /opt/so/log/logstash/logstash.log

=========================================================================
Checking log file /opt/so/log/redis/redis-server.log

1:M 13 Nov 2024 01:47:23.686 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.686 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.687 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.688 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.689 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:47:23.690 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:03.681 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.068 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.797 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.797 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.797 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.797 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.797 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:50:04.797 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.237 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.638 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 01:58:26.639 # Error accepting a client connection: Connection reset by peer
1:M 13 Nov 2024 02:02:18.247 # Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (conn: fd=8)

=========================================================================
Checking log file /opt/so/log/kibana/misc.log

=========================================================================
Checking log file /opt/so/log/elastalert/stdout.log

=========================================================================
Checking log file /opt/so/log/elastalert/stderr.log

ValueError: sleep length must be non-negative

=========================================================================
Checking log file /opt/so/log/elastalert/elastalert.log

raise RuntimeError('cannot schedule new futures after shutdown')

RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,527 ERROR apscheduler.scheduler Error submitting job "Rule: Copy .DMP/.DUMP Files From Remote Share Via Cmd.EXE -- 044ba588-dff4-4918-9808-3f95e8160606 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,527 ERROR apscheduler.scheduler Error submitting job "Rule: Potential ReflectDebugger Content Execution Via WerFault.EXE -- fabfb3a7-3ce1-4445-9c7c-3c27f1051cdd (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,528 ERROR apscheduler.scheduler Error submitting job "Rule: Dumping of Sensitive Hives Via Reg.EXE -- fd877b94-9bb5-4191-bb25-d79cbd93c167 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,528 ERROR apscheduler.scheduler Error submitting job "Rule: Diamond Sleet APT DNS Communication Indicators -- fba38e0f-4607-4344-bb8f-a4b50cdeef7f (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,528 ERROR apscheduler.scheduler Error submitting job "Rule: Default Cobalt Strike Certificate -- 7100f7e3-92ce-4584-b7b7-01b40d3d4118 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,529 ERROR apscheduler.scheduler Error submitting job "Rule: Okta Admin Role Assignment Created -- 139bdd4b-9cd7-49ba-a2f4-744d0a8f5d8c (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,529 ERROR apscheduler.scheduler Error submitting job "Rule: Invoke-Obfuscation Via Use Clip - PowerShell Module -- ebdf49d8-b89c-46c9-8fdf-2c308406f6bd (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,530 ERROR apscheduler.scheduler Error submitting job "Rule: Remote PowerShell Session (PS Module) -- 96b9f619-aa91-478f-bacb-c3e50f8df575 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,530 ERROR apscheduler.scheduler Error submitting job "Rule: Disable Administrative Share Creation at Startup -- c7dcacd0-cc59-4004-b0a4-1d6cdebe6f3e (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,531 ERROR apscheduler.scheduler Error submitting job "Rule: Transferring Files with Credential Data via Network Shares - Zeek -- 2e69f167-47b5-4ae7-a390-47764529eff5 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,531 ERROR apscheduler.scheduler Error submitting job "Rule: Email Exifiltration Via Powershell -- 312d0384-401c-4b8b-abdf-685ffba9a332 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,531 ERROR apscheduler.scheduler Error submitting job "Rule: Curl Download And Execute Combination -- 21dd6d38-2b18-4453-9404-a0fe4a0cc288 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,532 ERROR apscheduler.scheduler Error submitting job "Rule: Suspicious File Downloaded From Direct IP Via Certutil.EXE -- 13e6fe51-d478-4c7e-b0f2-6da9b400a829 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,532 ERROR apscheduler.scheduler Error submitting job "Rule: Antivirus Filter Driver Disallowed On Dev Drive - Registry -- 31e124fb-5dc4-42a0-83b3-44a69c77b271 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,533 ERROR apscheduler.scheduler Error submitting job "Rule: Potential Information Disclosure CVE-2023-43261 Exploitation - Web -- a2bcca38-9f3a-4d5e-b603-0c587e8569d7 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-11-13 01:59:19,533 ERROR apscheduler.scheduler Error submitting job "Rule: Renamed BrowserCore.EXE Execution -- 8a4519e8-e64a-40b6-ae85-ba8ad2177559 (trigger: interval[0:03:00], next run at: 2024-11-13 01:59:17 UTC)" to executor "default"
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

=========================================================================
Checking log file /opt/so/log/elasticfleet/so-elastic-agent-gen-installers.log

=========================================================================
Checking log file /var/log/cron

Result: One or more errors found

Guidelines

I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

cm-ops · 2024-11-14T13:38:47Z

cm-ops
Nov 14, 2024
Maintainer

From all of that above, it looks like you are having an issue with Elasticsearch. Do you have access to the CLI? If so, what is sudo so-elasticsearch-query _cluster/health?pretty showing? If it shows nothing, check the so-elasticsearch container to see if it is running, if it returns and is anything but Green, validate all the data nodes are acconted for and check sudo so-elasticsearch-query _cluster/allocation/explain?pretty for a reason why it is not Green.

0 replies

JhonShell · 2024-11-15T13:33:13Z

JhonShell
Nov 15, 2024
Author

Hi cm-ops,
It seems to be that after 2 days with the server getting inaccessible, I decide to disable all the custom rule that the analyst spend almost a month creating and it stop crashing every day at midnight, please help me without those rules we are not be able to detect most of the attack.

2 replies

cm-ops Nov 18, 2024
Maintainer

How were the rules added? What types of rules (Elastalert, Suricata, Yara, Sigma)?

JhonShell Nov 18, 2024
Author

We added the rule from the GUI, sigma rule, the Weido thing is that is working until midnight and then its stop working.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Security onion manager getting inaccessible after 600 sigma rules added #13940

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Security onion manager getting inaccessible after 600 sigma rules added #13940

JhonShell Nov 13, 2024

Version

Installation Method

Description

Installation Type

Location

Hardware Specs

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

Network Traffic Speeds

Status

Salt Status

Logs

Detail

Security Onion Log Check - Wed Nov 13 02:20:00 PM UTC 2024

========================================================================= Checking container "so-kratos"

========================================================================= Checking container "so-dockerregistry"

========================================================================= Checking log file /opt/so/log/sostatus/status.log

========================================================================= Checking log file /opt/so/log/kratos/kratos-migrate.log

Checking log file /opt/so/log/influxdb/influxdb.log

========================================================================= Checking log file /opt/so/log/influxdb/setup.log

========================================================================= Checking log file /opt/so/log/nginx/error.log

========================================================================= Checking log file /opt/so/log/nginx/access.log

========================================================================= Checking log file /opt/so/log/soc/sensoroni-server.log

========================================================================= Checking log file /opt/so/log/soc/salt-relay.log

========================================================================= Checking log file /opt/so/log/soc/detections_runtime-status_sigma.log

========================================================================= Checking log file /opt/so/log/soc/sync.log

========================================================================= Checking log file /opt/so/log/soc/detections-backup.log

========================================================================= Checking log file /opt/so/log/sensoroni/sensoroni.log

========================================================================= Checking log file /opt/so/log/telegraf/telegraf.log

========================================================================= Checking log file /opt/so/log/telegraf/influxdb_size.log

========================================================================= Checking log file /opt/so/log/idstools/download_idstools_state.log

========================================================================= Checking log file /opt/so/log/idstools/download_cron.log

========================================================================= Checking log file /opt/so/log/elasticsearch/cron-elasticsearch-indices-delete.log

========================================================================= Checking log file /opt/so/log/elasticsearch/so-elasticsearch-indices-delete.log

========================================================================= Checking log file /opt/so/log/elasticsearch/securityonion.log

========================================================================= Checking log file /opt/so/log/elasticsearch/indices-delete-alert.log

========================================================================= Checking log file /opt/so/log/logstash/logstash-2024-09-24.log

========================================================================= Checking log file /opt/so/log/logstash/logstash.log

========================================================================= Checking log file /opt/so/log/redis/redis-server.log

========================================================================= Checking log file /opt/so/log/kibana/misc.log

========================================================================= Checking log file /opt/so/log/elastalert/stdout.log

========================================================================= Checking log file /opt/so/log/elastalert/stderr.log

========================================================================= Checking log file /opt/so/log/elastalert/elastalert.log

========================================================================= Checking log file /opt/so/log/elasticfleet/so-elastic-agent-gen-installers.log

========================================================================= Checking log file /var/log/cron

Guidelines

Replies: 2 comments · 2 replies

cm-ops Nov 14, 2024 Maintainer

JhonShell Nov 15, 2024 Author

cm-ops Nov 18, 2024 Maintainer

JhonShell Nov 18, 2024 Author

JhonShell
Nov 13, 2024

=========================================================================
Checking container "so-kratos"

=========================================================================
Checking container "so-dockerregistry"

=========================================================================
Checking log file /opt/so/log/sostatus/status.log

=========================================================================
Checking log file /opt/so/log/kratos/kratos-migrate.log

=========================================================================
Checking log file /opt/so/log/influxdb/setup.log

=========================================================================
Checking log file /opt/so/log/nginx/error.log

=========================================================================
Checking log file /opt/so/log/nginx/access.log

=========================================================================
Checking log file /opt/so/log/soc/sensoroni-server.log

=========================================================================
Checking log file /opt/so/log/soc/salt-relay.log

=========================================================================
Checking log file /opt/so/log/soc/detections_runtime-status_sigma.log

=========================================================================
Checking log file /opt/so/log/soc/sync.log

=========================================================================
Checking log file /opt/so/log/soc/detections-backup.log

=========================================================================
Checking log file /opt/so/log/sensoroni/sensoroni.log

=========================================================================
Checking log file /opt/so/log/telegraf/telegraf.log

=========================================================================
Checking log file /opt/so/log/telegraf/influxdb_size.log

=========================================================================
Checking log file /opt/so/log/idstools/download_idstools_state.log

=========================================================================
Checking log file /opt/so/log/idstools/download_cron.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/cron-elasticsearch-indices-delete.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/so-elasticsearch-indices-delete.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/securityonion.log

=========================================================================
Checking log file /opt/so/log/elasticsearch/indices-delete-alert.log

=========================================================================
Checking log file /opt/so/log/logstash/logstash-2024-09-24.log

=========================================================================
Checking log file /opt/so/log/logstash/logstash.log

=========================================================================
Checking log file /opt/so/log/redis/redis-server.log

=========================================================================
Checking log file /opt/so/log/kibana/misc.log

=========================================================================
Checking log file /opt/so/log/elastalert/stdout.log

=========================================================================
Checking log file /opt/so/log/elastalert/stderr.log

=========================================================================
Checking log file /opt/so/log/elastalert/elastalert.log

=========================================================================
Checking log file /opt/so/log/elasticfleet/so-elastic-agent-gen-installers.log

=========================================================================
Checking log file /var/log/cron

Replies: 2 comments 2 replies

cm-ops
Nov 14, 2024
Maintainer

JhonShell
Nov 15, 2024
Author

cm-ops Nov 18, 2024
Maintainer

JhonShell Nov 18, 2024
Author