Database disk image is malformed

[oliverhavrila]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

2 x Intel Xeon E5-2680 v4 14-Core 2.40 GHz 35 MB Cache

RAM

128 GB DDR4

Storage for /

315G

Storage for /nsm

7.4T

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Services failed:

                       Security Onion Status
                         Container │ Status  │              Details
───────────────────────────────────┼─────────┼──────────────────────
                 so-dockerregistry │ running │           Up 2 weeks
                     so-elastalert │ running │           Up 2 weeks
                  so-elastic-fleet │ running │           Up 2 weeks
 so-elastic-fleet-package-registry │ running │ Up 2 weeks (healthy)
                  so-elasticsearch │ running │           Up 2 weeks
                       so-idstools │ running │            Up 6 days
                       **so-influxdb │ missing │**
                         so-kibana │ running │           Up 2 weeks
                         so-kratos │ running │           Up 2 weeks
                       so-logstash │ running │           Up 2 weeks
                          so-nginx │ running │ Up 2 weeks (healthy)
                          so-redis │ running │           Up 2 weeks
                      so-sensoroni │ running │           Up 2 weeks
                            so-soc │ running │           Up 2 weeks
                          **so-steno │ missing │**
                so-strelka-backend │ running │           Up 2 weeks
            so-strelka-coordinator │ running │           Up 2 weeks
             so-strelka-filestream │ running │           Up 2 weeks
               so-strelka-frontend │ running │           Up 2 weeks
             so-strelka-gatekeeper │ running │           Up 2 weeks
                so-strelka-manager │ running │           Up 2 weeks
                       so-suricata │ running │            Up 6 days
                       so-telegraf │ running │           Up 2 weeks
                           **so-zeek │ missing │**

Salt status:

local:
    Data failed to compile:
----------
    The function "state.apply" is running as PID 3699008 and was started at 2024, Sep 19 18:34:17.857681 with jid 20240919183417857681

Disk status:

df -H
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 4.2M     0  4.2M   0% /dev
tmpfs                     68G   41k   68G   1% /dev/shm
tmpfs                     27G  2.8G   25G  11% /run
/dev/mapper/system-root  315G   70G  245G  23% /
/dev/sda2                1.1G  431M  633M  41% /boot
/dev/sda1                1.1G  6.6M  1.1G   1% /boot/efi
/dev/mapper/system-nsm   7.4T  7.4T   82M 100% /nsm
/dev/mapper/system-tmp   2.1G  189M  1.9G  10% /tmp
tmpfs                     14G     0   14G   0% /run/user/0
tmpfs                     14G     0   14G   0% /run/user/939
tmpfs                     14G     0   14G   0% /run/user/1000

Dear,
After upgrading to version 2.4.100, the system was running for about 2 weeks and suddenly this message appeared:
{"error":{"code":500,"status":"Internal Server Error","message":"sqlite create: named insert: database disk image is malformed"}}

Is this due to it being /nsm at 100%? When I checked the status of the disk after the upgrade it had about 40%, there is no way that it will fill up in two weeks. Is there any way to fix this, or what can be deleted?

Thank you

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

Is this due to it being /nsm at 100%?

Most likely.

Is there any way to fix this, or what can be deleted?

I'd start with determining what is using the majority of the disk space:

du --max-depth=1 -h /nsm

[oliverhavrila]

Unfortunately, whatever I want to do with the nsm folder, I keep getting this message:

$ sudo du --max-depth=1 -h /nsm
du: cannot access '/nsm': Input/output error

[dougburks]

Have you tried rebooting?

Have you tried running full hardware diagnostics on your system?

[oliverhavrila]

this is update after reboot:

I tried to xfs repair:
sudo xfs_repair /dev/mapper/system-nsm

mount again nsm:
sudo mount /dev/mapper/system-nsm

nsm has still 100% disk capacity so I started with determining what is using the majority of the disk space:

$ sudo du --max-depth=1 -h /nsm
32G     /nsm/docker-registry
3.3G    /nsm/repo
4.5G    /nsm/elastic-fleet
792K    /nsm/kratos
215M    /nsm/rules
2.0G    /nsm/influxdb
12G     /nsm/backup
21M     /nsm/soc
3.1T    /nsm/pcap
0       /nsm/import
0       /nsm/pcapout
243M    /nsm/mysql
3.5T    /nsm/elasticsearch
4.0K    /nsm/logstash
246M    /nsm/redis
4.0K    /nsm/pcaptmp
68G     /nsm/pcapindex
12K     /nsm/suricata
20G     /nsm/zeek
4.6M    /nsm/strelka
0       /nsm/suripcap
729M    /nsm/lost+found
6.7T    /nsm

[oliverhavrila]

update

The stengrapher function was always disabled, so I deleted the files in the /pcap folder, then freed up quite a bit of space.

However, the so-influxdb service was still not working, fortunately a restart of the service solved that.

Now I have the problem that I can't log in to SO after entering the name and password that was set there, and it doesn't work even when I add a new user: sudo so-user-add $user

[dougburks]

It sounds like the user database got corrupted due to the filesystem corruption. If I were you, I'd perform full hardware diagnostics to make sure it's not a problem with the underlying hardware. If the hardware is OK, then I'd wipe the drive and perform a fresh installation of Security Onion.

[oliverhavrila]

I did a full hardware diagnostic and everything is fine. Finally, I ran a software check of the disks, rebooted the server and finally everything is working.

But I still think it may be some kind of upgrade bug. At the very least, it turned on the stenographer function by itself.