Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIGSEGV writing log on FreeBSD #3255

Closed
einsibjarni opened this issue Sep 11, 2024 · 29 comments
Closed

SIGSEGV writing log on FreeBSD #3255

einsibjarni opened this issue Sep 11, 2024 · 29 comments
Assignees
Labels
2.x Related to ModSecurity version 2.x

Comments

@einsibjarni
Copy link

einsibjarni commented Sep 11, 2024

Describe the bug

I'm trying to update the FreeBSD port to 2.9.8 (current version in ports is 2.9.6) and to link with pcre2 instead of pcre.

I can build it and apache runs, but when modsecurity tries to write an audit log, the httpd process dies with SIGSEGV.
Setting SecAuditLogType Concurrent stops httpd from dying.

Logs and dumps

Output of:

httpd.zip

Notice: Be carefully to not leak any confidential information.

To Reproduce

Steps to reproduce the behavior:

Anything that causes mod_security to write an audit log.

Expected behavior

A clear and concise description of what you expected to happen.

Server (please complete the following information):

  • ModSecurity version (and connector): [e.g. ModSecurity v3.0.1 with nginx-connector v1.0.0]
  • 2.9.8
  • WebServer: [e.g. nginx-1.15.5]
  • apache 2.4.62
  • OS (and distro): [e.g. Linux, archlinux]
  • FreeBSD 14.1-RELEASE

Rule Set (please complete the following information):

  • Running any public or commercial rule set? [e.g. SpiderLabs commercial rules]
  • What is the version number? [e.g. 2018-08-11]

Additional context

The stacktrace indicates that the problem is in apr_global_mutex_lock().

The configure command that the ports build generates and uses to build is:
./configure --with-apxs=/usr/local/sbin/apxs --with-pcre2=/usr/local --with-yajl=/usr/local --with-curl=/usr/local --without-ssdeep --without-lua --disable-mlogc --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/share/man --disable-silent-rules --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd14.1

I'm trying to get the port in FreeBSD ports updated, so even though I can workaround the problem, we need to address this issue if it is to be merged upstream.

@einsibjarni einsibjarni added the 2.x Related to ModSecurity version 2.x label Sep 11, 2024
@airween
Copy link
Member

airween commented Sep 11, 2024

@einsibjarni thanks for reporting.

Could you share your stacktrace output? Eg. bt full.

@marcstern could you take a look at this? I assume this can be related to PR #3191.

@einsibjarni
Copy link
Author

Could you share your stacktrace output? Eg. bt full.

The core dump is attached to the post as httpd.zip, but the output from bt full is:

(lldb) target create "/usr/local/sbin/httpd" --core "/tmp/httpd.core"
Core file '/tmp/httpd.core' (x86_64) was loaded.
(lldb) bt all
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
* thread #1, name = 'httpd', stop reason = signal SIGSEGV
  * frame #0: 0x0000000823a07699 libapr-1.so.0`apr_global_mutex_lock + 9
    frame #1: 0x0000000837291a78 mod_security2.so`sec_audit_logger_native + 504
    frame #2: 0x0000000837289e0a mod_security2.so`modsecurity_process_phase + 1594
    frame #3: 0x000000083728866a mod_security2.so`hook_log_transaction + 826
    frame #4: 0x00000000002808b6 httpd`ap_run_log_transaction + 54
    frame #5: 0x0000000000292d22 httpd`___lldb_unnamed_symbol4032 + 50
    frame #6: 0x0000000823a09aae libapr-1.so.0`apr_pool_destroy + 46
    frame #7: 0x0000000000292d73 httpd`___lldb_unnamed_symbol4033 + 35
    frame #8: 0x00000000002937a0 httpd`ap_core_output_filter + 1088
    frame #9: 0x0000000828873665 mod_mpm_event.so`___lldb_unnamed_symbol388 + 1429
    frame #10: 0x0000000828872f96 mod_mpm_event.so`___lldb_unnamed_symbol387 + 854
    frame #11: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e2700) at thr_create.c:289:16
  thread #2, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x0000000825b0256a libc.so.7`__sys_read at _read.S:4
    frame #1: 0x000000082639a773 libthr.so.3`__thr_read(fd=9, buf=0x00000008205d9967, nbytes=1) at thr_syscalls.c:417:8
    frame #2: 0x0000000000275800 httpd`ap_mpm_podx_check + 48
    frame #3: 0x000000082887212b mod_mpm_event.so`___lldb_unnamed_symbol378 + 1467
    frame #4: 0x0000000828871a46 mod_mpm_event.so`___lldb_unnamed_symbol376 + 358
    frame #5: 0x00000008288712ac mod_mpm_event.so`___lldb_unnamed_symbol367 + 4748
    frame #6: 0x0000000000270ded httpd`ap_run_mpm + 77
    frame #7: 0x000000000025e193 httpd`main + 2483
    frame #8: 0x0000000825a4ea6a libc.so.7`__libc_start1(argc=2, argv=0x00000008205d9d18, env=0x00000008205d9d30, cleanup=<unavailable>, mainX=(httpd`main)) at libc_start1.c:157:7
    frame #9: 0x000000000025d760 httpd`_start + 48
  thread #3, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e2000) at thr_create.c:289:16
  thread #4, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e2e00) at thr_create.c:289:16
  thread #5, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e3500) at thr_create.c:289:16
  thread #6, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e3c00) at thr_create.c:289:16
  thread #7, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e4300) at thr_create.c:289:16
  thread #8, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e4a00) at thr_create.c:289:16
  thread #9, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e5100) at thr_create.c:289:16
  thread #10, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e5800) at thr_create.c:289:16
  thread #11, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e5f00) at thr_create.c:289:16
  thread #12, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e6600) at thr_create.c:289:16
  thread #13, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e6d00) at thr_create.c:289:16
  thread #14, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e7400) at thr_create.c:289:16
  thread #15, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e7b00) at thr_create.c:289:16
  thread #16, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e8200) at thr_create.c:289:16
  thread #17, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e8900) at thr_create.c:289:16
  thread #18, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2844000) at thr_create.c:289:16
  thread #19, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2844700) at thr_create.c:289:16
  thread #20, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2844e00) at thr_create.c:289:16
  thread #21, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2845500) at thr_create.c:289:16
  thread #22, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2845c00) at thr_create.c:289:16
  thread #23, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2846300) at thr_create.c:289:16
  thread #24, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2846a00) at thr_create.c:289:16
  thread #25, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2847100) at thr_create.c:289:16
  thread #26, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2847800) at thr_create.c:289:16
  thread #27, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x0000000825b0242a libc.so.7`__sys_kevent at _kevent.S:4
    frame #1: 0x000000082639ac24 libthr.so.3`__thr_kevent(kq=16, changelist=0x0000000000000000, nchanges=0, eventlist=0x000019ddd2172028, nevents=114, timeout=0x0000000860ce5ed0) at thr_syscalls.c:397:8
    frame #2: 0x0000000823a10bf0 libapr-1.so.0`___lldb_unnamed_symbol1234 + 128
    frame #3: 0x00000008288749f2 mod_mpm_event.so`___lldb_unnamed_symbol393 + 2178
    frame #4: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2847f00) at thr_create.c:289:16
(lldb) q

@einsibjarni
Copy link
Author

I've tested building 2.9.7 and the error does not occur there, but if I build 2.9.7 with pcre2, ldd shows it links with both pcre and pcre2.

I've tried 2.9.8 with both pcre and pcre2 and both show this error.

@marcstern
Copy link

I think it's a side-effect of an already existing problem: when a mutex creation fails (for any reason), we try to use later, although it's null.
This PR should fix the problem: #3257

@einsibjarni
Copy link
Author

I applied the patch from #3257 and I can confirm that it fixes the crashes.
It does write "Global mutex was not created" in httpd error log, but the audit log does get written

@airween
Copy link
Member

airween commented Sep 13, 2024

It's good to see that the segfault has gone away - but I still wonder why the mutex cannot be created.

@einsibjarni do you think is there any special settings? I tried to set up the SecAuditLogType Concurrent but I was not able reproduce your behavior.

I'm worrying about that - as you described - this didn't happen in 2.9.7, right? So why happens this?

@einsibjarni
Copy link
Author

I tried to set up the SecAuditLogType Concurrent

It only happens with ‘SecAuditLogType Serial’

@airween
Copy link
Member

airween commented Sep 14, 2024

Ah, sorry, my bad...

Then it's more weird: this is the default setting, and we also check this behavior in our CI.

I also use this setting on many servers without any problem - it would be highly recommended if we could find out the cause of the error (I mean the team - with your help).

Could you give some details about your FreeBSD system (architecture, version, special settings, etc...)? I'm going to install a VM and try to reproduce it.

@einsibjarni
Copy link
Author

I've tested on FreeBSD 14.1-RELEASE-p4 with apache24 from binary pkg and I get the error.

So just install base FreeBSD 14.1-RELEASE and then pkg install apache24

@einsibjarni
Copy link
Author

I'm building mod_security from ports, but with patches to update it to 2.9.8.

Resulting configure cmd is: ./configure --with-apxs=/usr/local/sbin/apxs --with-pcre2=/usr/local --with-yajl=/usr/local --with-curl=/usr/local --without-ssdeep --without-lua --disable-mlogc --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/share/man --disable-silent-rules --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd14.1

@marcstern
Copy link

@einsibjarni : can you check if #3257 solves the problem?

@einsibjarni
Copy link
Author

@marcstern I did try it, and it does fix the segfault.

It does write "Global mutex was not created" in httpd error log, but the audit log does get written

@marcstern
Copy link

That's the expected behaviour, indeed.
Why is the global mutex not created, that's another issue (concurrency?), but at least we handle that case correctly now.

@airween
Copy link
Member

airween commented Sep 20, 2024

@einsibjarni,

I need some help to bring up ModSecurity2 on FreeBSD. I already installed the VM, Apache 2.4 is running, and compiled the module from source with your ./configure instruction.

After I added the module I got an error message:

httpd: Syntax error on line 184 of /usr/local/etc/apache24/httpd.conf: Syntax error on line 1 of /usr/local/etc/apache24/modules.d/001_security.conf: Can't locate API module structure `mod_security2' in file /usr/local/modsecurity/lib/mod_security2.so: Undefined symbol "mod_security2"

Would you mind to come to Slack and ping me in private? You can join here and look at the #project-modsecurity channel.

@EinalButtocks
Copy link

I can't seem to join the slack server unless I have a owasp email address.

I'm building ModSecurity2 through ports, patched to update it to 2.9.8.

https://docs.freebsd.org/en/books/handbook/ports/#ports-using

Attached is my proposed patch to the port to update it to 2.9.8.
0001-Update-www-mod_security-to-2.9.8.patch

@airween
Copy link
Member

airween commented Sep 24, 2024

I can't seem to join the slack server unless I have a owasp email address.

hmm... I registered on OWASP Slack with my gmail.com address. I don't think there is any restriction.

I'm building ModSecurity2 through ports, patched to update it to 2.9.8.

https://docs.freebsd.org/en/books/handbook/ports/#ports-using

Attached is my proposed patch to the port to update it to 2.9.8. 0001-Update-www-mod_security-to-2.9.8.patch

I followed your instructions, and installed Apache from ports, then I built the module. After that I got the result above.

@EinalButtocks
Copy link

image

@EinalButtocks
Copy link

What's the contents of /usr/local/etc/apache24/modules.d/001_security.conf?

I don't have /usr/local/modsecurity/lib/mod_security2.so, it's in /usr/local/lib/mod_security2.so

@airween
Copy link
Member

airween commented Sep 24, 2024

Ah, I see.

Please share your e-mail address (you can send me in private - see my e-mail address on my GH page), and I can send you an invitation.

@airween
Copy link
Member

airween commented Sep 24, 2024

What's the contents of /usr/local/etc/apache24/modules.d/001_security.conf?

LoadModule mod_security2 /usr/local/modsecurity/lib/mod_security2.so

I don't have /usr/local/modsecurity/lib/mod_security2.so, it's in /usr/local/lib/mod_security2.so

# ls -l /usr/local/modsecurity/lib/mod_security2.so
-rwxr-xr-x  1 root wheel 2572608 Sep 20 14:37 /usr/local/modsecurity/lib/mod_security2.so

@airween
Copy link
Member

airween commented Sep 30, 2024

I could review the issue on FreeBSD, and found an interesting behavior.

mod_security2 uses files for mutexes, see this part:

    rc = apr_file_mktemp(&lock_name, path, 0, mp);
    if (rc != APR_SUCCESS) {
        ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not create temporary file for global lock");
        return -1;
    }
    // below func always return APR_SUCCESS
    apr_file_name_get(&filename, lock_name);

This function (apr_file_mktemp()) returns with APR_SUCCESS, but file hasn't created (or may be it created but disappeared immediately). This is why the next function returns an error. The filename is something like /tmp/modsec-lock-tmp.yCfr5n - @einsibjarni, do you know is there any special restrictions under FreeBSD regarding to temp files under /tmp?

I tried the locking mechanism without file - simple pass a NULL to the function here:

    rc = apr_global_mutex_create(lock, NULL, APR_LOCK_DEFAULT, mp);

and this solved the problem on FreeBSD. I haven't tested it on other systems, but my question is: do we want to insist the files?

@marcstern
Copy link

marcstern commented Sep 30, 2024

  1. In acquire_global_lock(), we must initialize fname to NULL. This should avoid the crash in case apr_file_name_get() doesn't succeed (apr_global_mutex_create has a check if fname == NULL). I added it to the PR msr_global_mutex_lock: handle errors from apr_global_mutex_lock #3257
  2. In modsecurity_init(), we stop the processing in case one acquire_global_lock() fails, so the remaining ones are not tried. I think we shouldn't stop as the other locks may succeed.
  3. Using APR_LOCK_DEFAULT may not always work without file. Under Windows, APR_LOCK_DEFAULT (the only supported one) doesn't require a path. If we target only other platforms supporting APR_LOCK_POSIXSEM, we could hard-code it and remove all code related to path:
#ifdef WIN32
# define MSC_LOCK_MECH APR_LOCK_DEFAULT
#else
# define MSC_LOCK_MECH APR_LOCK_POSIXSEM
#endif
    rc = apr_global_mutex_create(lock, NULL, MSC_LOCK_MECH, mp);

@airween
Copy link
Member

airween commented Sep 30, 2024

This should avoid the crash in case apr_file_name_get() doesn't succeed (apr_global_mutex_create has a check if fname == NULL).

The problem is not this. apr_file_mktemp() returns the filename successfully, without NULL initialization. Then the file exists.

When the apr_global_mutex_create() function is called, then that function removes the lock file and returns with APR_EBADMASK or APR_EINPROGRESS (with avalue 17).

See this summary:

Breakpoint 1, acquire_global_lock (lock=0x8014a1e98, mp=0x801427028) at modsecurity.c:150
150	    rc = apr_global_mutex_create(lock, filename, APR_LOCK_DEFAULT, mp);
(gdb) p filename
$1 = 0x801458568 "/tmp/modsec-lock-tmp.fO33bJ"

check the file:

# ls -la /tmp/modsec-lock-tmp.fO33bJ
-rw-------   1 root    wheel  0 Sep 30 13:42 modsec-lock-tmp.fO33bJ

continue the investigation

Breakpoint 2, acquire_global_lock (lock=0x8014a1e98, mp=0x801427028) at modsecurity.c:152
152	    if (rc != APR_SUCCESS) {
(gdb) p rc
$2 = 17

check the file again:

# ls -la /tmp/modsec-lock-tmp.fO33bJ
ls: /tmp/modsec-lock-tmp.fO33bJ: No such file or directory

I think this is because of the OS or FS (ZFS vs Ext4) has different behavior. Initialize the filename with NULL does not solve this problem.

In other words: see the truss's output (it's like strace on Linux):

openat(AT_FDCWD,"/tmp/modsec-lock-tmp.fO33bJ",O_RDWR|O_CREAT|O_EXCL,0600) = 10 (0xa)
fcntl(10,F_GETFD,)				 = 0 (0x0)
fcntl(10,F_SETFD,FD_CLOEXEC)			 = 0 (0x0)
getpid()					 = 43156 (0xa894)
openat(AT_FDCWD,"/tmp/modsec-lock-tmp.fO33bJ",O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC,0600) ERR#17 'File exists'
unlink("/tmp/modsec-lock-tmp.fO33bJ")		 = 0 (0x0)

You can see that the lock file was created with openat(), but later the system calls it again, then it removes. There you can see the error code and message:

ERR#17 File exists

I'm almost sure we use this locking mechanism completely wrong.

@marcstern
Copy link

In Apache, they use a wrapper "ap_global_mutex_create" everywhere ...
We probably should use either that or option 3 (no filename at all).
I tested without filename under Windows and I never egt a warning (I performed some stress tests with concurrent accesses).

@airween
Copy link
Member

airween commented Sep 30, 2024

We probably should use either that or option 3 (no filename at all).

Yes, this works for me (on FreeBSD)

int acquire_global_lock(apr_global_mutex_t **lock, apr_pool_t *mp) {
    apr_status_t rc;
    const char *filename = NULL;

    rc = apr_global_mutex_create(lock, filename, APR_LOCK_DEFAULT, mp);
    if (rc != APR_SUCCESS) {
        ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not create global mutex");
        return -1;
    }

#if !defined(MSC_TEST)
#ifdef __SET_MUTEX_PERMS
#if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2
    rc = ap_unixd_set_global_mutex_perms(*lock);
#else
    rc = unixd_set_global_mutex_perms(*lock);
#endif
    if (rc != APR_SUCCESS) {
        ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not set permissions on global mutex"); 
        return -1;
    }
#endif /* SET_MUTEX_PERMS */
#endif /* MSC_TEST */
    return APR_SUCCESS;
}

I tested without filename under Windows and I never egt a warning (I performed some stress tests with concurrent accesses).

I'm not sure stress tests are relevant here, because - if I'm right - the lock is created only once, when the engine starts. But it's good to know that on Windows it works as well too. I'll check that on Linux too, and will notice you here. If it works, then we should modify that wrapper function (acquire_global_lock()).

@marcstern
Copy link

After looking at the code in APR, APR_LOCK_DEFAULT seems the right choice, no need to hard-code one.
Limitation is that it may not be portable to some exotic platforms, but does it really work on something else than Unix/Windows?
+1 for me

@airween
Copy link
Member

airween commented Oct 2, 2024

I asked this on Apache's developer mailing list - see the short thread here.

@marcstern
Copy link

marcstern commented Oct 3, 2024

PR with the fix: #3269

marcstern pushed a commit to marcstern/ModSecurity that referenced this issue Oct 3, 2024
We don't have to generate a temp name ourselves, it'll be done in apr_global_mutex_create().
We don't have to provide a filename, apr_global_mutex_create() generates one automatically.
Moreover, under Unix & Windows, the preferred mechanism won't use a file at all.
apr_file_mktemp() cannot be used as it creates the file (at least on FreeBSD).
Discussion in Apache mailing list: https://lists.apache.org/thread/ykb26kg4lgcqnldvxwd9p6hv16fy4z9l
marcstern pushed a commit to marcstern/ModSecurity that referenced this issue Oct 3, 2024
We don't have to generate a temp name ourselves, it'll be done in apr_global_mutex_create().
We don't have to provide a filename, apr_global_mutex_create() generates one automatically.
Moreover, under Unix & Windows, the preferred mechanism won't use a file at all.
apr_file_mktemp() cannot be used as it creates the file (at least on FreeBSD).
Discussion in Apache mailing list: https://lists.apache.org/thread/ykb26kg4lgcqnldvxwd9p6hv16fy4z9l
marcstern pushed a commit to marcstern/ModSecurity that referenced this issue Oct 3, 2024
We don't have to generate a temp name ourselves, it'll be done in apr_global_mutex_create().
We don't have to provide a filename, apr_global_mutex_create() generates one automatically.
Moreover, under Unix & Windows, the preferred mechanism won't use a file at all.
apr_file_mktemp() cannot be used as it creates the file (at least on FreeBSD).
Discussion in Apache mailing list: https://lists.apache.org/thread/ykb26kg4lgcqnldvxwd9p6hv16fy4z9l
@airween
Copy link
Member

airween commented Oct 14, 2024

Closing as solved with #3269.

@airween airween closed this as completed Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to ModSecurity version 2.x
Projects
None yet
Development

No branches or pull requests

4 participants