Wierd data in source.ip - Dashboards failing 500

[kawaiipantsu]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

installation

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

RAM

Storage for /

Storage for /nsm

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

UTF8/Wierd data in "type": "ip" field for source.ip

Hi, if i look at the "Overview" dashboard in SO and scroll down to source.ip i can see something like the following:

You can clearly see that something is wrong, as it's showing what should be IP addresses as wierd characters.

Also when i then try to use a dashboard like sysmon/dns or anything with source.ip then i get a 500 notification:

But if i remove groupby source.ip | groupby -sankey source.ip destination.ip from the part of the dashboard query then it works as expected. So clearly it's effected by those wierd source.ip fields is my guess.

To dig further this is the log from elasticsearch:

[2024-09-23T12:07:52,013][ERROR][org.elasticsearch.rest.ChunkedRestResponseBody] failure encoding chunk
java.lang.IllegalArgumentException: Failed trying to format bytes as UTF8.  Possibly caused by a mapping mismatch
(( stack trace ))
[2024-09-23T12:07:52,014][ERROR][org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler] caught exception while encoding response chunk, closing connection [id: 0x4a2bc133, L:/172.17.1.22:9200 - R:/172.17.1.1:34016]
java.lang.IllegalArgumentException: Failed trying to format bytes as UTF8.  Possibly caused by a mapping mismatch
(( stack trace ))

I have no idea how to debug this further ??

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

Did you perform the steps in this blog post?
https://blog.securityonion.net/2024/09/security-onion-24100-hotfix-20240903.html

[kawaiipantsu]

Did you perform the steps in this blog post? https://blog.securityonion.net/2024/09/security-onion-24100-hotfix-20240903.html

Hi,

Did not think about hotfixes, thanks for pointing me onto that. But it did however now work.
I ran the soup and made sure it was applied, checking /etc/sohotfix showed 20240903 or higher.
Then i ran the 2 clean-up loops as root and checked the dashboards, same deal.

I then for good measure also did a full so-elastic-clear just to be sure. But still the same, all looked normal for a few sec then things began to populate and we where back to this:

My guess is that your hotfix does not cover all the effected indexes ?
But should my so-elastic-clear not have delt with that?

[kawaiipantsu]

It might be that the 2 for loops are not working propperly. Or it seems to timeout ?
But again, should my so-elastic-clear not do the same even more effective ?

[root@soc ~]# for i in logs-system.application-default logs-system.security-default logs-system.system-default; do

    so-elasticsearch-query $i/_rollover -XPOST

done

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-system.application-default-2024.09.26-000001",
  "new_index": "                                             .ds-logs-system.application-default-2024.09.26-000002",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {}
}
{
  "acknowle                                             dged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-system.security-default-2024.09.26-000001",
  "new_index": ".ds-logs-syst                                             em.security-default-2024.09.26-000002",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {}
}
{
  "error": {
    "root_cause": [
      {
        "                                             type": "process_cluster_event_timeout_exception",
        "reason": "failed to process cluster event (rollover_index source [.ds-logs-system                                             .system-default-2024.09.26-000002] to target [.ds-logs-system.system-default-2024.09.26-000002]) within 30s"
      }
    ],
    "type": "process_cl                                             uster_event_timeout_exception",
    "reason": "failed to process cluster event (rollover_index source [.ds-logs-system.system-default-2                                             024.09.26-000002] to target [.ds-logs-system.system-default-2024.09.26-000002]) within 30s"
  },
  "status": 503
}

[root@soc ~]# for i in logs-system.application-default logs-system.security-default logs-system.system-default; do

    INDEX_TO_DELETE=$(so-elasticsearch-query $i | jq -r 'keys[]' | tail -2 | head -1); so-elasticsearch-query $INDEX_TO_DELETE -XDELETE

done

{
  "acknowledged": true
}
{
  "error": {
    "root_cause": [
      {
        "type": "process_cluster_event_timeout_exception",
        "reason": "failed to process cluster event (delete-index [[.ds-logs-system.security-default-2024.09.26-000001/rE5JUHwzT62                                             ZRyyaImOccQ]]) within 30s"
      }
    ],
    "type": "process_cluster_event_timeout_exception",
    "reason": "failed to process cluster event (delete-index [[.ds-logs-system.security-default-2024.09.26-000001/rE5JUHwzT62ZRyy                                             aImOccQ]]) within 30s"
  },
  "status": 503
}
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "index [.ds-logs-system.system-default-2024.09.26-000001] is the write index for data stream [logs-system.syste                                             m-default] and cannot be deleted"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "index [.ds-logs-system.system-default-2024.09.26-000001] is the write index for data stream [logs-system.system-de                                             fault] and cannot be deleted"
  },
  "status": 400
}

Please note i have formatted the json output via jq.

[kawaiipantsu]

I wanted to make sure that the script where run propperly. So i edited a file on the server and made a shell script propperly with the lines.
Here is the output, looks lot better. However the wierd data on source.ip is still present on the dashboard.

[root@soc ~]# ./fix.sh
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-system.application-default-2024.09.26-000002",
  "new_index": ".ds-logs-system.application-default-2024.09.26-000004",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {}
}
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-system.security-default-2024.09.26-000002",
  "new_index": ".ds-logs-system.security-default-2024.09.26-000003",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {}
}
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-logs-system.system-default-2024.09.26-000001",
  "new_index": ".ds-logs-system.system-default-2024.09.26-000002",
  "rolled_over": true,
  "dry_run": false,
  "lazy": false,
  "conditions": {}
}
{
  "acknowledged": true
}
{
  "acknowledged": true
}
{
  "acknowledged": true
}

[kawaiipantsu]

I am however now able to load up the dashboards that previously would generate a 500 error.
But the data still looks very much wierd!

EDIT: Nope, after a while i was no longer able to load etc DNS or Sysmon dashboard anymore and it gave a 500 the main dashboard still shows source.ip with wierd data.

[dougburks]

Seems like you might have something that is sending strange data into the system.

Are you collecting logs from endpoints via Elastic Agent?

Are you collecting logs via syslog?

Are you collecting some other kind of data like netflow?

If you filter one of these dashboards to show ONLY Zeek data, do the source IP addresses look correct?

[kawaiipantsu]

Seems like you might have something that is sending strange data into the system.
Are you collecting logs from endpoints via Elastic Agent?
Are you collecting logs via syslog?
Are you collecting some other kind of data like netflow?
If you filter one of these dashboards to show ONLY Zeek data, do the source IP addresses look correct?

Hi,

Yes my thought to - That (im the cause), always look inwards before outwards hehe.
But no I have special policies for AWS/ec2 where i also collect Apache HTTP Server logs, but I tried to remove those from the policy completely and then do so-elastic-clear but they where still there. Syslog is not in use, as i use the "system" integration. I also have a pfsense integration (but also tested with that off)

As for your test, i choose the "Overview" dashboard from Dashboards.
Then down under Event.module i choose "Only" on Zeek.

PS. Also tried to explicitly target Zeek module via * AND event.module:"zeek"

Sadly they are still present

(And yes, i make enormous amounts of DNS traffic, as i run various scanning/status and crawling services related to security research)

[dougburks]

This seems like a strange issue.

Have you modified your Zeek configuration in any way?

Do the raw Zeek logs in /nsm/zeek/logs/current/ show correct source IP addresses?

Are you able to duplicate this issue on a fresh installation?

[kawaiipantsu]

This seems like a strange issue.
Have you modified your Zeek configuration in any way?
Do the raw Zeek logs in /nsm/zeek/logs/current/ show correct source IP addresses?
Are you able to duplicate this issue on a fresh installation?

So i still think you are right about it being something else that I send to it, I'm just not sure how to debug it.
Is there any way to view it via Kibana Dev tools and some sort of query to show me what indices/data streams these weird documents belong to ?

The raw Zeek logs looked fine, each and everyone of them.
Ran all the files via jq to both validate the json but also to explicitly print out the orgi_h field or what it's name was.

This is a completely fresh install, the most customization i have done is the Elastic Agent setup where i have 5 agent policies.
Also activated a bunch of sigma and yara detections.

So again, i'm sure you are right - It's an integration, just don't know how to find it.