Nginx "core dumped"

[tacerus]

Hello,

This happens upon loading the sample rules suggested in the CRS documentation.

Nginx is configured like this:

server {
        server_name georg.systems;
        modsecurity on;
        modsecurity_rules_file /etc/nginx/security/modsecurity_includes.conf;

        location / {
           <something>
        }
}

The includes file is filled according to to https://coreruleset.org/docs/deployment/install/#includes-for-nginx, except for the BEFORE/AFTER rules.

Upon trying to connect to the webserver as configured above the connection is aborted:

$ curl -I https://georg.systems/
curl: (92) HTTP/2 stream 1 was not closed cleanly before end of the underlying stream

And nginx's error.log reports the worker to have crashed:

2022/08/13 09:53:09 [notice] 6173#6173: signal 17 (SIGCHLD) received from 6940
2022/08/13 09:53:09 [alert] 6173#6173: worker process 6940 exited on signal 11 (core dumped)
2022/08/13 09:53:09 [notice] 6173#6173: start worker process 6950

ModSecurity's SecDebugLog stays empty, even with SecDebugLogLevel 9. I tested it with SecRuleEngine On as well as SecRuleEngine DetectionOnly.

The site is being served properly if I leave only the modsecurity.conf in the includes file.

Most setup options have been left at their defaults, the most I changed were the log and data storage paths.

Versions:
CRS: 3.3.2
ModSecurity: 3.0.7
ModSecurity-nginx: 1.0.3
Nginx: 1.23.1
OS: openSUSE Leap 15.4

nginx -V

nginx version: nginx/1.23.1
built by gcc 7.5.0 (SUSE Linux) 
built with OpenSSL 1.1.1d  10 Sep 2019 (running with OpenSSL 1.1.1l  24 Aug 2021 SUSE release 150400.7.7.1)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

The module was loaded dynamically.

[martinhsv]

Hello @tacerus ,

Are you using lmdb? If so, there is a known crash scenario if the file location is one that the process does not have sufficient permissions to write to. See owasp-modsecurity/ModSecurity#2755. If that sounds like your situation, likely the simplest resolution is to use the nginx configuration item 'working_directory' (owasp-modsecurity/ModSecurity#2755 (comment)).

If that is not your issue, then ...

The message about 'HTTP/2 stream 1' might be an indication that HTTP/2 is a factor in causing the problem. If so, you could try experimentally allowing only HTTP/1.1. On the other hand the HTTP/2 could be a merely consequence of the crash. Are you able to supply a stack trace from the core dump?

[tacerus]

Hi @martinhsv, thank you for getting back!

I am not using lmdb, however I have compiled ModSecurity with lmdb support.

As for HTTP/2, I actually thought of that as well and did try to connect with curl's --http1.1 and --http1.0, which resulted in a slightly different client side error message (which I unfortunately not saved) but the same server side crash.

I now recompiled ModSecurity as well as the dynamic module without lmdb support - and I can no longer reproduce the worker crashes! I assume we can conclude it to #2755, which you linked, then? But if you'd like, I can try it again with lmdb and with nginx's --debug configure option to supply a stack trace!

Whilst my core dump issue is presumably solved now, I still have the issue with it not printing any log output:

SecRuleEngine On
SecDebugLog /var/log/nginx/modsecurity/debug.log
SecDebugLogLevel 9
SecAuditEngine On
SecAuditLogType Serial
SecAuditLogStorageDir /var/log/nginx/modsecurity/audit/

$ ls -al /var/log/nginx/modsecurity/
total 0
drwxr-xr-x 1 nginx root    28 Aug 13 09:42 .
drwxr-xr-x 1 root  root 10028 Aug 13 00:00 ..
drwxr-xr-x 1 nginx root     0 Aug 13 09:42 audit
-rw-r--r-- 1 nginx root     0 Aug 12 17:20 debug.log

Should I move this to a separate issue? I assume it's not related.

[martinhsv]

Hello @tacerus,

Given your use of italics ('using'), I'm assuming you mean that you don't care about using the functionality provided by lmdb. But of course if ModSecurity has been built with '--with-lmdb' and one or more relevant rules are triggered (which, as I recall, would likely be the case based on the rule set that you cited), then ModSecurity will use lmdb. (That's probably understood by you but I thought I'd be explicit for anyone else reading this item.)

Note that for audit log, you have specified SecAuditLogStorageDir. However, that configuration item is relevant only if using SecAuditLogType Concurrent. Since you are using SecAuditLogType Serial , you need to use SecAuditLog for the audit log file location.

I have only ever seen failures to write to the debug log when there are permissions issues. If you want those particular locations, that's something you can check. (Incidentally, there is no need to pre-allocate the files yourself; ModSecurity will create them at the specified locations on startup if it has permissions to do so.)

I personally always use:

SecAuditLog /var/log/modsec_audit.log

and

SecDebugLog /var/log/modsec_debug.log

[tacerus]

Actually, I did not realize some of my rules were utilizing lmdb! and I missed that SecAuditLogStorageDir was only compatible with SecAuditLogType Concurrent. Very helpful - thank you very much!!