pdns-server: double free crash with --version

[devicenull]

• This is not a support question, I have read about opensource and will send support questions to the IRC channel, Github Discussions or the mailing list.
• I have read and understood the 'out in the open' support policy

• Program: Authoritative
• Issue type: Bug report

Short description

When the libbindbackend.so module is loaded, pdns_server --version crashes

Apr 03 19:40:28 PowerDNS Authoritative Server 4.9.0 (C) PowerDNS.COM BV
Apr 03 19:40:28 Using 64-bits mode. Built using gcc 12.2.0 on Apr  3 2024 19:01:37 by builder@devicenu11.sysops.wordpress.com.
Apr 03 19:40:28 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Apr 03 19:40:28 Features: libcrypto-ecdsa libcrypto-ed25519 libcrypto-ed448 libcrypto-eddsa lua lua-records protobuf curl scrypt
Apr 03 19:40:28 Built-in modules: bind gmysql
Apr 03 19:40:28 Loading '/usr/local/lib/pdns/libbindbackend.so'
Apr 03 19:40:28 Loaded modules: bind gmysql
Apr 03 19:40:28 Configured with: ""
free(): double free detected in tcache 2
Apr 03 19:40:28 Got a signal 6, attempting to print trace:
Apr 03 19:40:28 pdns_server(+0x1765f0) [0x5609755485f0]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7f84f7c5b050]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x7f84f7ca9e2c]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7f84f7c5afb2]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f84f7c45472]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x7f430) [0x7f84f7c9e430]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x947aa) [0x7f84f7cb37aa]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x96a36) [0x7f84f7cb5a36]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(__libc_free+0x6f) [0x7f84f7cb7e8f]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x3e55d) [0x7f84f7c5d55d]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x3e69a) [0x7f84f7c5d69a]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(+0x27251) [0x7f84f7c46251]
Apr 03 19:40:28 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f84f7c46305]
Apr 03 19:40:28 pdns_server(_start+0x21) [0x56097552c861]
Aborted

Environment

• Operating system: Debian Bookworm
• Software version: pdns 4.9.0
• Software source: self-compiled

Steps to reproduce

1. On debian bookworm:
2. apt-get -fqqy install pkg-config libboost-all-dev libssl-dev libluajit-5.1-dev liblua5.1-0-dev lua5.1 libcurl4-gnutls-dev patchelf default-libmysqlclient-dev libsqlite3-dev
3. extract powerdns source
4. ./configure; make; make install
5. pdns_server --version <-- crashes
6. rm -f /usr/local/lib/pdns/*bind*
7. pdns_server --version <-- doesn't crash

Expected behaviour

no crashes :)

Actual behaviour

crashes :)

Other information

I swear I'm going crazy

Note: I was testing with clang which is why the initial version had CC= options there, but it reproduces w/ gcc too

[zeha]

Given clang isn't the default compiler, this seems incomplete at best:
4. ./configure; make; make install

[devicenull]

==2409397== Memcheck, a memory error detector
==2409397== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==2409397== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==2409397== Command: pdns_server --version
==2409397==
Apr 03 19:39:21 PowerDNS Authoritative Server 4.9.0 (C) PowerDNS.COM BV
Apr 03 19:39:21 Using 64-bits mode. Built using gcc 12.2.0 on Apr  3 2024 19:01:37 by builder@devicenu11.sysops.wordpress.com.
Apr 03 19:39:21 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Apr 03 19:39:21 Features: libcrypto-ecdsa libcrypto-ed25519 libcrypto-ed448 libcrypto-eddsa lua lua-records protobuf curl scrypt
Apr 03 19:39:21 Built-in modules: bind gmysql
Apr 03 19:39:21 Loading '/usr/local/lib/pdns/libbindbackend.so'
Apr 03 19:39:21 Loaded modules: bind gmysql
Apr 03 19:39:21 Configured with: ""
==2409397== Invalid read of size 8
==2409397==    at 0x51EBF5: operator boost::multi_index::detail::ordered_index_node_compressed_base<boost::multi_index::detail::null_augment_policy, std::allocator<char> >::pointer (ord_index_node.hpp:176)
==2409397==    by 0x51EBF5: root (ord_index_impl.hpp:1050)
==2409397==    by 0x51EBF5: delete_all_nodes_ (ord_index_impl.hpp:812)
==2409397==    by 0x51EBF5: delete_all_nodes_ (multi_index_container.hpp:945)
==2409397==    by 0x51EBF5: ~multi_index_container (multi_index_container.hpp:334)
==2409397==    by 0x51EBF5: SharedLockGuarded<boost::multi_index::multi_index_container<BB2DomainInfo, boost::multi_index::indexed_by<boost::multi_index::ordered_unique<boost::multi_index::member<BB2DomainInfo, unsigned int, &BB2DomainInfo::d_id>, mpl_::na, mpl_::na>, boost::multi_index::ordered_unique<boost::multi_index::tag<NameTag, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<BB2DomainInfo, DNSName, &BB2DomainInfo::d_name>, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<BB2DomainInfo> > >::~SharedLockGuarded() (lock.hh:413)
==2409397==    by 0x51BF55C: __run_exit_handlers (exit.c:116)
==2409397==    by 0x51BF699: exit (exit.c:146)
==2409397==    by 0x51A8250: (below main) (libc_start_call_main.h:74)
==2409397==  Address 0x60fe188 is 296 bytes inside a block of size 320 free'd
==2409397==    at 0x484399B: operator delete(void*, unsigned long) (vg_replace_malloc.c:935)
==2409397==    by 0x51BF55C: __run_exit_handlers (exit.c:116)
==2409397==    by 0x51BF699: exit (exit.c:146)
==2409397==    by 0x51A8250: (below main) (libc_start_call_main.h:74)
==2409397==  Block was alloc'd at
==2409397==    at 0x4840F2F: operator new(unsigned long) (vg_replace_malloc.c:422)
==2409397==    by 0x625E9E4: allocate (new_allocator.h:137)
==2409397==    by 0x625E9E4: allocate (alloc_traits.h:464)
==2409397==    by 0x625E9E4: allocate_node (multi_index_container.hpp:644)
==2409397==    by 0x625E9E4: header_holder (header_holder.hpp:35)
==2409397==    by 0x625E9E4: multi_index_container (multi_index_container.hpp:176)
==2409397==    by 0x625E9E4: SharedLockGuarded (lock.hh:424)
==2409397==    by 0x625E9E4: __static_initialization_and_destruction_0 (bindbackend2.cc:77)
==2409397==    by 0x625E9E4: _GLOBAL__sub_I_bindbackend2.cc (bindbackend2.cc:1549)
==2409397==    by 0x40048BD: call_init (dl-init.c:90)
==2409397==    by 0x40048BD: call_init (dl-init.c:27)
==2409397==    by 0x40049A3: _dl_init (dl-init.c:137)
==2409397==    by 0x52D0023: _dl_catch_exception (dl-error-skeleton.c:182)
==2409397==    by 0x400B09D: dl_open_worker (dl-open.c:808)
==2409397==    by 0x52CFFC9: _dl_catch_exception (dl-error-skeleton.c:208)
==2409397==    by 0x400B437: _dl_open (dl-open.c:884)
==2409397==    by 0x5206437: dlopen_doit (dlopen.c:56)
==2409397==    by 0x52CFFC9: _dl_catch_exception (dl-error-skeleton.c:208)
==2409397==    by 0x52D007E: _dl_catch_error (dl-error-skeleton.c:227)
==2409397==    by 0x5205F26: _dlerror_run (dlerror.c:138)
==2409397==
==2409397== Invalid free() / delete / delete[] / realloc()
==2409397==    at 0x484399B: operator delete(void*, unsigned long) (vg_replace_malloc.c:935)
==2409397==    by 0x51BF55C: __run_exit_handlers (exit.c:116)
==2409397==    by 0x51BF699: exit (exit.c:146)
==2409397==    by 0x51A8250: (below main) (libc_start_call_main.h:74)
==2409397==  Address 0x60fe060 is 0 bytes inside a block of size 320 free'd
==2409397==    at 0x484399B: operator delete(void*, unsigned long) (vg_replace_malloc.c:935)
==2409397==    by 0x51BF55C: __run_exit_handlers (exit.c:116)
==2409397==    by 0x51BF699: exit (exit.c:146)
==2409397==    by 0x51A8250: (below main) (libc_start_call_main.h:74)
==2409397==  Block was alloc'd at
==2409397==    at 0x4840F2F: operator new(unsigned long) (vg_replace_malloc.c:422)
==2409397==    by 0x625E9E4: allocate (new_allocator.h:137)
==2409397==    by 0x625E9E4: allocate (alloc_traits.h:464)
==2409397==    by 0x625E9E4: allocate_node (multi_index_container.hpp:644)
==2409397==    by 0x625E9E4: header_holder (header_holder.hpp:35)
==2409397==    by 0x625E9E4: multi_index_container (multi_index_container.hpp:176)
==2409397==    by 0x625E9E4: SharedLockGuarded (lock.hh:424)
==2409397==    by 0x625E9E4: __static_initialization_and_destruction_0 (bindbackend2.cc:77)
==2409397==    by 0x625E9E4: _GLOBAL__sub_I_bindbackend2.cc (bindbackend2.cc:1549)
==2409397==    by 0x40048BD: call_init (dl-init.c:90)
==2409397==    by 0x40048BD: call_init (dl-init.c:27)
==2409397==    by 0x40049A3: _dl_init (dl-init.c:137)
==2409397==    by 0x52D0023: _dl_catch_exception (dl-error-skeleton.c:182)
==2409397==    by 0x400B09D: dl_open_worker (dl-open.c:808)
==2409397==    by 0x52CFFC9: _dl_catch_exception (dl-error-skeleton.c:208)
==2409397==    by 0x400B437: _dl_open (dl-open.c:884)
==2409397==    by 0x5206437: dlopen_doit (dlopen.c:56)
==2409397==    by 0x52CFFC9: _dl_catch_exception (dl-error-skeleton.c:208)
==2409397==    by 0x52D007E: _dl_catch_error (dl-error-skeleton.c:227)
==2409397==    by 0x5205F26: _dlerror_run (dlerror.c:138)
==2409397==
==2409397==
==2409397== HEAP SUMMARY:
==2409397==     in use at exit: 127,398 bytes in 63 blocks
==2409397==   total heap usage: 8,326 allocs, 8,264 frees, 2,257,971 bytes allocated
==2409397==
==2409397== LEAK SUMMARY:
==2409397==    definitely lost: 472 bytes in 4 blocks
==2409397==    indirectly lost: 0 bytes in 0 blocks
==2409397==      possibly lost: 0 bytes in 0 blocks
==2409397==    still reachable: 126,926 bytes in 59 blocks
==2409397==         suppressed: 0 bytes in 0 blocks
==2409397== Rerun with --leak-check=full to see details of leaked memory
==2409397==
==2409397== For lists of detected and suppressed errors, rerun with: -s
==2409397== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)```

[rgacogne]

I'm sorry if it looks like a dumb question but I have been bitten by this several times: are you sure that /usr/local/lib/pdns/libbindbackend.so is the one you just built (ie, it matches the pdns_server you are running)?

[zeha]

@rgacogne some discussion on IRC yesterday revealed that builds now produce a builtin bind AND and an .so module.

This point still needs investigating though.

[Habbie]

Yes, it looks like we might be generating the bind .so even when bindbackend is linked statically, perhaps as late as make install. Indeed this calls for investigation.

[tomwijnroks]

I can confirm this pdns_server --version crash using environment:

• Operating system: Debian 11 (bullseye)
• Software version: pdns 4.9.1
• Software source: self-compiled
• Configuration: ./configure --sysconfdir=/etc/powerdns

[Habbie]

Sep 16 13:45:54 Loading 'modules/bindbackend/.libs/libbindbackend.so'
=================================================================
==243763==ERROR: AddressSanitizer: odr-violation (0x7fea78ad98a0):
  [1] size=32 's_binddirectory' bindbackend2.cc:83:8
  [2] size=32 's_binddirectory' bindbackend2.cc:83:8
These globals were registered at these points:
  [1]:
    #0 0x7fea7aa3a1b8 in __asan_register_globals ../../../../src/libsanitizer/asan/asan_globals.cpp:341
    #1 0x7fea7b2479cd in call_init elf/dl-init.c:74
    #2 0x7fea7b2479cd in call_init elf/dl-init.c:26

  [2]:
    #0 0x7fea7aa3a1b8 in __asan_register_globals ../../../../src/libsanitizer/asan/asan_globals.cpp:341
    #1 0x7fea79e46375 in call_init ../csu/libc-start.c:145
    #2 0x7fea79e46375 in __libc_start_main_impl ../csu/libc-start.c:347

==243763==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 's_binddirectory' at bindbackend2.cc:83:8
==243763==ABORTING

the reason --version (and, somehow to a lesser extent, --list-modules) trigger this is that they load -all- .so they can find everything that is available

[Habbie]

perhaps as late as make install

just make