-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SIGSEGV writing log on FreeBSD #3255
Comments
@einsibjarni thanks for reporting. Could you share your stacktrace output? Eg. @marcstern could you take a look at this? I assume this can be related to PR #3191. |
The core dump is attached to the post as httpd.zip, but the output from
|
I've tested building 2.9.7 and the error does not occur there, but if I build 2.9.7 with pcre2, ldd shows it links with both pcre and pcre2. I've tried 2.9.8 with both pcre and pcre2 and both show this error. |
I think it's a side-effect of an already existing problem: when a mutex creation fails (for any reason), we try to use later, although it's null. |
I applied the patch from #3257 and I can confirm that it fixes the crashes. |
It's good to see that the segfault has gone away - but I still wonder why the mutex cannot be created. @einsibjarni do you think is there any special settings? I tried to set up the I'm worrying about that - as you described - this didn't happen in 2.9.7, right? So why happens this? |
It only happens with ‘SecAuditLogType Serial’ |
Ah, sorry, my bad... Then it's more weird: this is the default setting, and we also check this behavior in our CI. I also use this setting on many servers without any problem - it would be highly recommended if we could find out the cause of the error (I mean the team - with your help). Could you give some details about your FreeBSD system (architecture, version, special settings, etc...)? I'm going to install a VM and try to reproduce it. |
I've tested on FreeBSD 14.1-RELEASE-p4 with apache24 from binary pkg and I get the error. So just install base FreeBSD 14.1-RELEASE and then |
I'm building mod_security from ports, but with patches to update it to 2.9.8. Resulting configure cmd is: |
@einsibjarni : can you check if #3257 solves the problem? |
@marcstern I did try it, and it does fix the segfault. It does write "Global mutex was not created" in httpd error log, but the audit log does get written |
That's the expected behaviour, indeed. |
I need some help to bring up ModSecurity2 on FreeBSD. I already installed the VM, Apache 2.4 is running, and compiled the module from source with your After I added the module I got an error message:
Would you mind to come to Slack and ping me in private? You can join here and look at the #project-modsecurity channel. |
I can't seem to join the slack server unless I have a owasp email address. I'm building ModSecurity2 through ports, patched to update it to 2.9.8. https://docs.freebsd.org/en/books/handbook/ports/#ports-using Attached is my proposed patch to the port to update it to 2.9.8. |
hmm... I registered on OWASP Slack with my gmail.com address. I don't think there is any restriction.
I followed your instructions, and installed Apache from ports, then I built the module. After that I got the result above. |
What's the contents of I don't have |
Ah, I see. Please share your e-mail address (you can send me in private - see my e-mail address on my GH page), and I can send you an invitation. |
|
I could review the issue on FreeBSD, and found an interesting behavior. mod_security2 uses files for mutexes, see this part: rc = apr_file_mktemp(&lock_name, path, 0, mp);
if (rc != APR_SUCCESS) {
ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not create temporary file for global lock");
return -1;
}
// below func always return APR_SUCCESS
apr_file_name_get(&filename, lock_name); This function ( I tried the locking mechanism without file - simple pass a rc = apr_global_mutex_create(lock, NULL, APR_LOCK_DEFAULT, mp); and this solved the problem on FreeBSD. I haven't tested it on other systems, but my question is: do we want to insist the files? |
|
The problem is not this. When the See this summary: Breakpoint 1, acquire_global_lock (lock=0x8014a1e98, mp=0x801427028) at modsecurity.c:150
150 rc = apr_global_mutex_create(lock, filename, APR_LOCK_DEFAULT, mp);
(gdb) p filename
$1 = 0x801458568 "/tmp/modsec-lock-tmp.fO33bJ" check the file: # ls -la /tmp/modsec-lock-tmp.fO33bJ
-rw------- 1 root wheel 0 Sep 30 13:42 modsec-lock-tmp.fO33bJ continue the investigation Breakpoint 2, acquire_global_lock (lock=0x8014a1e98, mp=0x801427028) at modsecurity.c:152
152 if (rc != APR_SUCCESS) {
(gdb) p rc
$2 = 17 check the file again: # ls -la /tmp/modsec-lock-tmp.fO33bJ
ls: /tmp/modsec-lock-tmp.fO33bJ: No such file or directory I think this is because of the OS or FS (ZFS vs Ext4) has different behavior. Initialize the In other words: see the
You can see that the lock file was created with
I'm almost sure we use this locking mechanism completely wrong. |
In Apache, they use a wrapper "ap_global_mutex_create" everywhere ... |
Yes, this works for me (on FreeBSD) int acquire_global_lock(apr_global_mutex_t **lock, apr_pool_t *mp) {
apr_status_t rc;
const char *filename = NULL;
rc = apr_global_mutex_create(lock, filename, APR_LOCK_DEFAULT, mp);
if (rc != APR_SUCCESS) {
ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not create global mutex");
return -1;
}
#if !defined(MSC_TEST)
#ifdef __SET_MUTEX_PERMS
#if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2
rc = ap_unixd_set_global_mutex_perms(*lock);
#else
rc = unixd_set_global_mutex_perms(*lock);
#endif
if (rc != APR_SUCCESS) {
ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not set permissions on global mutex");
return -1;
}
#endif /* SET_MUTEX_PERMS */
#endif /* MSC_TEST */
return APR_SUCCESS;
}
I'm not sure stress tests are relevant here, because - if I'm right - the lock is created only once, when the engine starts. But it's good to know that on Windows it works as well too. I'll check that on Linux too, and will notice you here. If it works, then we should modify that wrapper function ( |
After looking at the code in APR, APR_LOCK_DEFAULT seems the right choice, no need to hard-code one. |
I asked this on Apache's developer mailing list - see the short thread here. |
PR with the fix: #3269 |
We don't have to generate a temp name ourselves, it'll be done in apr_global_mutex_create(). We don't have to provide a filename, apr_global_mutex_create() generates one automatically. Moreover, under Unix & Windows, the preferred mechanism won't use a file at all. apr_file_mktemp() cannot be used as it creates the file (at least on FreeBSD). Discussion in Apache mailing list: https://lists.apache.org/thread/ykb26kg4lgcqnldvxwd9p6hv16fy4z9l
We don't have to generate a temp name ourselves, it'll be done in apr_global_mutex_create(). We don't have to provide a filename, apr_global_mutex_create() generates one automatically. Moreover, under Unix & Windows, the preferred mechanism won't use a file at all. apr_file_mktemp() cannot be used as it creates the file (at least on FreeBSD). Discussion in Apache mailing list: https://lists.apache.org/thread/ykb26kg4lgcqnldvxwd9p6hv16fy4z9l
We don't have to generate a temp name ourselves, it'll be done in apr_global_mutex_create(). We don't have to provide a filename, apr_global_mutex_create() generates one automatically. Moreover, under Unix & Windows, the preferred mechanism won't use a file at all. apr_file_mktemp() cannot be used as it creates the file (at least on FreeBSD). Discussion in Apache mailing list: https://lists.apache.org/thread/ykb26kg4lgcqnldvxwd9p6hv16fy4z9l
Closing as solved with #3269. |
Describe the bug
I'm trying to update the FreeBSD port to 2.9.8 (current version in ports is 2.9.6) and to link with pcre2 instead of pcre.
I can build it and apache runs, but when modsecurity tries to write an audit log, the httpd process dies with SIGSEGV.
Setting
SecAuditLogType Concurrent
stops httpd from dying.Logs and dumps
Output of:
httpd.zip
Notice: Be carefully to not leak any confidential information.
To Reproduce
Steps to reproduce the behavior:
Anything that causes mod_security to write an audit log.
Expected behavior
A clear and concise description of what you expected to happen.
Server (please complete the following information):
Rule Set (please complete the following information):
Additional context
The stacktrace indicates that the problem is in apr_global_mutex_lock().
The configure command that the ports build generates and uses to build is:
./configure --with-apxs=/usr/local/sbin/apxs --with-pcre2=/usr/local --with-yajl=/usr/local --with-curl=/usr/local --without-ssdeep --without-lua --disable-mlogc --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/share/man --disable-silent-rules --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd14.1
I'm trying to get the port in FreeBSD ports updated, so even though I can workaround the problem, we need to address this issue if it is to be merged upstream.
The text was updated successfully, but these errors were encountered: