Issue with Missing Data on Storage Node

[mckenziemmack]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

64

RAM

251G

Storage for /

558G

Storage for /nsm

38T

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,

I'm currently facing an issue where I believe I have lost all data on my storage node. The problem appears to be related to my manager storing indices that it shouldn't, which is causing the manager to hit its watermark. This seems to lead to this situation where I can no longer access or view the data on the storage node in Kibana, even though the manager still retains a few of the incorrectly stored indices.

The storage node itself is showing only 551GB of usage out of 38TB available (about 2% utilization), which makes me think that indices may have been deleted. I'm worried that the data has been lost, but I'm unsure why the storage node would automatically delete all of its storage data like this. Would it be possible that the data is still there on my storage nodes but just inaccessible?

Has anyone experienced something similar or have any insights? Any help or guidance would be greatly appreciated.

Thanks!

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What returns when you run these commands:

sudo so-elasticsearch-query _cat/allocation?v
sudo so-elasticsearch-query _cat/nodes?v

If your search node has a larger /nsm partition than your manager, then it is possible the clean up script deleted some data.

Have you done any Elasticsearch ILM tuning?

[mckenziemmack]

Thank you for your response!

This is returns when those commands are ran:

[~]$ sudo so-elasticsearch-query _cat/allocation?v
shards shards.undesired write_load.forecast disk.indices.forecast disk.indices disk.used disk.avail disk.total disk.percent host        ip          node        node.role
   137               24                 0.0                 4.2tb        4.2tb     4.5tb     31.8tb     36.3tb           12 storage1    storage1 ip storage1    dhit
    76                0                 0.0               421.2gb      300.2gb   377.9gb      146gb      524gb           72 manager     manager ip  manager dmrt

[~]$ sudo so-elasticsearch-query _cat/nodes?v
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
storage1 ip          32          99  89   21.48   18.16    17.71 dhit      -      storage1
storage2 ip          60          80  31   21.18   24.07    24.57 dhit      -      storage2
manager ip           56          98  81   10.90   11.20    11.84 dmrt      *      manager

As a current mitigation, I have adjusted the manager's watermark settings to a low of 45%, and a high of 50%, with the flood stage staying at 90%, and this has seemed to stop the search node reallocating the large shards to the manager as moving any of the complete 50 GB shards would make the manager node over the high watermark. I'm just worried I'm maybe going down the wrong rabbit hole and this mass deletion event will happen again.