BOLT segfaults on optimizing CPython

[vmarkovtsev]

I catch the following stack trace while building a DEB package with CPython 3.11.0 and BOLT optimizations backported from python/cpython#95908
My BOLT version corresponds to the "qualification" (v15) branch of the official LLVM Ubuntu packages.

 #0 0x0000560336033bd1 (/usr/lib/llvm-15/bin/llvm-bolt+0x1ad7bd1)
 #1 0x0000560336031a9c (/usr/lib/llvm-15/bin/llvm-bolt+0x1ad5a9c)
 #2 0x00005603360340e6 (/usr/lib/llvm-15/bin/llvm-bolt+0x1ad80e6)
 #3 0x00007f089cddb520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x0000560336d73a30 (/usr/lib/llvm-15/bin/llvm-bolt+0x2817a30)
 #5 0x00005603360de800 (/usr/lib/llvm-15/bin/llvm-bolt+0x1b82800)
 #6 0x00005603360dd00e (/usr/lib/llvm-15/bin/llvm-bolt+0x1b8100e)
 #7 0x00005603360d8a16 (/usr/lib/llvm-15/bin/llvm-bolt+0x1b7ca16)
 #8 0x00005603360d7e49 (/usr/lib/llvm-15/bin/llvm-bolt+0x1b7be49)
 #9 0x000056033607f61b (/usr/lib/llvm-15/bin/llvm-bolt+0x1b2361b)
#10 0x000056033607768b (/usr/lib/llvm-15/bin/llvm-bolt+0x1b1b68b)
#11 0x0000560334d9ec62 (/usr/lib/llvm-15/bin/llvm-bolt+0x842c62)
#12 0x00007f089cdc2d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
#13 0x00007f089cdc2e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#14 0x0000560334d9cd95 (/usr/lib/llvm-15/bin/llvm-bolt+0x840d95)

Log

(cd /cpython/python3.11-3.11.0/debian/tmp/usr/bin; ln -s idle3.11 idle3)
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: <unknown>
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0xc00000, offset 0x800000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
/usr/bin/install -c -m 644 pyconfig.h /cpython/python3.11-3.11.0/debian/tmp/usr/include/python3.11/pyconfig.h
rm -f /cpython/python3.11-3.11.0/debian/tmp/usr/bin/pydoc3
(cd /cpython/python3.11-3.11.0/debian/tmp/usr/bin; ln -s pydoc3.11 pydoc3)
rm -f /cpython/python3.11-3.11.0/debian/tmp/usr/bin/2to3
(cd /cpython/python3.11-3.11.0/debian/tmp/usr/bin; ln -s 2to3-3.11 2to3)
if test "x" != "x" ; then \
	rm -f /cpython/python3.11-3.11.0/debian/tmp/usr/bin/python3-32; \
	(cd /cpython/python3.11-3.11.0/debian/tmp/usr/bin; ln -s python3.11-32 python3-32) \
fi
if test "x" != "x" ; then \
	rm -f /cpython/python3.11-3.11.0/debian/tmp/usr/bin/python3-intel64; \
	(cd /cpython/python3.11-3.11.0/debian/tmp/usr/bin; ln -s python3.11-intel64 python3-intel64) \
fi
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling -align-macro-fusion=all since no profile was specified
BOLT-INFO: enabling lite mode
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 2826
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 6676
BOLT-INSTRUMENTER: Number of function descriptors: 6676
BOLT-INSTRUMENTER: Number of branch counters: 122100
BOLT-INSTRUMENTER: Number of ST leaf node counters: 42643
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 164743
BOLT-INSTRUMENTER: Total size of counters: 1317944 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 188868 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 9271204 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /cpython/python3.11-3.11.0/build-static/python.bolt
BOLT-INFO: 0 out of 6839 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: the input contains 1072 (dynamic count : 0) opportunities for macro-fusion optimization that are going to be fixed
BOLT-INFO: 71462 instructions were shortened
BOLT-INFO: removed 165 empty blocks
BOLT-INFO: UCE removed 1161 blocks and 70068 bytes of code.
BOLT-INFO: SCTC: patched 0 tail calls (0 forward) tail calls (0 backward) from a total of 0 while removing 0 double jumps and removing 0 basic blocks totalling 0 bytes of code. CTCs total execution count is 0 and the number of times CTCs are taken is 0.
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x18036d0
BOLT-INFO: clear procedure is 0x17fe440
BOLT-INFO: setting _end to 0xa0dba0
BOLT-INFO: setting _end to 0xa0dba0
BOLT-INFO: patched build-id (flipped last bit)
./python.bolt_inst -m test --pgo --timeout=1200 || true
0:00:00 load avg: 2.89 Run tests sequentially (timeout: 20 min)
0:00:00 load avg: 2.89 [ 1/44] test_array
0:00:01 load avg: 2.89 [ 2/44] test_base64
0:00:02 load avg: 2.89 [ 3/44] test_binascii
0:00:02 load avg: 2.89 [ 4/44] test_binop
0:00:02 load avg: 2.89 [ 5/44] test_bisect
0:00:02 load avg: 2.89 [ 6/44] test_bytes
0:00:05 load avg: 2.74 [ 7/44] test_bz2
0:00:06 load avg: 2.74 [ 8/44] test_cmath
0:00:06 load avg: 2.74 [ 9/44] test_codecs
0:00:08 load avg: 2.68 [10/44] test_collections
0:00:09 load avg: 2.68 [11/44] test_complex
0:00:10 load avg: 2.68 [12/44] test_dataclasses
0:00:10 load avg: 2.68 [13/44] test_datetime
0:00:14 load avg: 2.46 [14/44] test_decimal
0:00:18 load avg: 2.46 [15/44] test_difflib
0:00:19 load avg: 2.43 [16/44] test_embed
0:00:21 load avg: 2.43 [17/44] test_float
0:00:21 load avg: 2.43 [18/44] test_fstring
0:00:22 load avg: 2.43 [19/44] test_functools
0:00:23 load avg: 2.43 [20/44] test_generators
0:00:23 load avg: 2.39 [21/44] test_hashlib
0:00:24 load avg: 2.39 [22/44] test_heapq
0:00:25 load avg: 2.39 [23/44] test_int
0:00:26 load avg: 2.39 [24/44] test_itertools
0:00:31 load avg: 2.36 [25/44] test_json
0:00:35 load avg: 2.33 [26/44] test_long
0:00:38 load avg: 2.46 [27/44] test_lzma
0:00:39 load avg: 2.46 [28/44] test_math
0:00:42 load avg: 2.46 [29/44] test_memoryview
0:00:44 load avg: 2.43 [30/44] test_operator
0:00:44 load avg: 2.43 [31/44] test_ordered_dict
0:00:47 load avg: 2.43 [32/44] test_patma
0:00:47 load avg: 2.43 [33/44] test_pickle
0:00:54 load avg: 2.28 [34/44] test_pprint
0:00:54 load avg: 2.28 [35/44] test_re
0:00:56 load avg: 2.28 [36/44] test_set
0:01:01 load avg: 2.50 [37/44] test_sqlite3
0:01:03 load avg: 2.38 [38/44] test_statistics
0:01:09 load avg: 2.27 [39/44] test_struct
0:01:10 load avg: 2.27 [40/44] test_tabnanny
0:01:11 load avg: 2.27 [41/44] test_time
0:01:13 load avg: 2.09 [42/44] test_unicode
0:01:16 load avg: 2.09 [43/44] test_xml_etree
0:01:17 load avg: 2.09 [44/44] test_xml_etree_c

Total duration: 1 min 19 sec
Tests result: SUCCESS
/usr/lib/llvm-15/bin/merge-fdata python.*.fdata > python.fdata
Using legacy profile format.
Merging data from python.bolt.48689.fdata...
Merging data from python.bolt.48692.fdata...
Merging data from python.bolt.48693.fdata...
Merging data from python.bolt.48694.fdata...
Merging data from python.bolt.48695.fdata...
Merging data from python.bolt.48696.fdata...
Merging data from python.bolt.48697.fdata...
Merging data from python.bolt.48698.fdata...
Merging data from python.bolt.48703.fdata...
Merging data from python.bolt.48704.fdata...
Merging data from python.bolt.48776.fdata...
Merging data from python.bolt.48817.fdata...
Merging data from python.bolt.48820.fdata...
Merging data from python.bolt.48879.fdata...
Merging data from python.bolt.48880.fdata...
Merging data from python.bolt.48881.fdata...
Merging data from python.bolt.48882.fdata...
Merging data from python.bolt.48883.fdata...
Merging data from python.bolt.48884.fdata...
Merging data from python.bolt.48885.fdata...
Merging data from python.bolt.48886.fdata...
Merging data from python.bolt.48887.fdata...
Merging data from python.bolt.48888.fdata...
Merging data from python.bolt.48889.fdata...
Merging data from python.bolt.48890.fdata...
Merging data from python.bolt.48891.fdata...
Merging data from python.bolt.48892.fdata...
Merging data from python.bolt.48893.fdata...
Merging data from python.bolt.48913.fdata...
Merging data from python.bolt.48934.fdata...
Merging data from python.bolt.48936.fdata...
Merging data from python.bolt.48937.fdata...
Merging data from python.bolt.48938.fdata...
Merging data from python.bolt.48939.fdata...
Merging data from python.bolt.48940.fdata...
Merging data from python.bolt.48941.fdata...
Merging data from python.bolt.48944.fdata...
Profile from 37 files merged.
/usr/lib/llvm-15/bin/llvm-bolt ./python -o python.bolt -data=python.fdata -update-debug-sections -reorder-blocks=ext-tsp -reorder-functions=hfsort+ -split-functions -icf=1 -inline-all -split-eh -reorder-functions-use-hot-size -peepholes=all -jump-tables=aggressive -inline-ap -indirect-call-promotion=all -dyno-stats -use-gnu-stack -frame-opt=hot
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: <unknown>
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling lite mode
BOLT-INFO: pre-processing profile using branch profile reader
BOLT-INFO: 4833 out of 6839 functions in the binary (70.7%) have non-empty execution profile
BOLT-INFO: 79 functions with profile could not be optimized
BOLT-INFO: profile for 1 objects was ignored
BOLT-INFO: the input contains 898 (dynamic count : 238818753) opportunities for macro-fusion optimization. Will fix instances on a hot path.
BOLT-INFO: 67272 instructions were shortened
BOLT-INFO: removed 1412 empty blocks
BOLT-INFO: ICF folded 109 out of 7336 functions in 4 passes. 0 functions had jump tables.
BOLT-INFO: Removing all identical functions will save 3.14 KB of code space. Folded functions were called 1978694 times based on profile.
BOLT-INFO: ICP Total indirect calls = 1805758399, 437 callsites cover 99% of all indirect calls
BOLT-INFO: ICP total indirect callsites with profile = 571
BOLT-INFO: ICP total jump table callsites = 242
BOLT-INFO: ICP total number of calls = 1085976712
BOLT-INFO: ICP percentage of calls that are indirect = 59.3%
BOLT-INFO: ICP percentage of indirect calls that can be optimized = 64.0%
BOLT-INFO: ICP percentage of indirect callsites that are optimized = 44.8%
BOLT-INFO: ICP number of method load elimination candidates = 0
BOLT-INFO: ICP percentage of method calls candidates that have loads eliminated = 0.0%
BOLT-INFO: ICP percentage of indirect branches that are optimized = 39.4%
BOLT-INFO: ICP percentage of jump table callsites that are optimized = 0.0%
BOLT-INFO: ICP number of jump table callsites that can use hot indices = 0
BOLT-INFO: ICP percentage of jump table callsites that use hot indices = 0.0%
BOLT-INFO: inlined 78277360 calls at 10671 call sites in 3 iteration(s). Change in binary size: 305785 bytes.
BOLT-INFO: basic block reordering modified layout of 2319 (32.09%) functions
BOLT-INFO: UCE removed 14 blocks and 0 bytes of code.
BOLT-INFO: splitting separates 1541675 hot bytes from 777665 cold bytes (66.47% of split functions is hot).
BOLT-INFO: 154 Functions were reordered by LoopInversionPass
BOLT-INFO: hfsort+ reduced the number of chains from 4805 to 1882
BOLT-INFO: program-wide dynostats after all optimizations before SCTC and FOP:

         16353736115 : executed forward branches
          1917612494 : taken forward branches
          3053468432 : executed backward branches
          1933269417 : taken backward branches
           499381481 : executed unconditional branches
          1638834757 : all function calls
           890137266 : indirect calls
           145966648 : PLT calls
        131065211117 : executed instructions
         35799628025 : executed load instructions
         17268616862 : executed store instructions
          1150826476 : taken jump table branches
                   0 : taken unknown indirect branches
         19906586028 : total branches
          4350263392 : taken branches
         15556322636 : non-taken conditional branches
          3850881911 : taken conditional branches
         19407204547 : all conditional branches

         15726955205 : executed forward branches (-3.8%)
          1295481232 : taken forward branches (-32.4%)
          4143150754 : executed backward branches (+35.7%)
          1677940307 : taken backward branches (-13.2%)
           518117768 : executed unconditional branches (+3.8%)
          1571189876 : all function calls (-4.1%)
           468466333 : indirect calls (-47.4%)
           145966587 : PLT calls (-0.0%)
        131917367436 : executed instructions (+0.7%)
         35801866734 : executed load instructions (+0.0%)
         17268268534 : executed store instructions (-0.0%)
          1150826476 : taken jump table branches (=)
                   0 : taken unknown indirect branches (=)
         20388223727 : total branches (+2.4%)
          3491539307 : taken branches (-19.7%)
         16896684420 : non-taken conditional branches (+8.6%)
          2973421539 : taken conditional branches (-22.8%)
         19870105959 : all conditional branches (+2.4%)

BOLT-INFO: SCTC: patched 141 tail calls (138 forward) tail calls (3 backward) from a total of 141 while removing 5 double jumps and removing 120 basic blocks totalling 597 bytes of code. CTCs total execution count is 29446913 and the number of times CTCs are taken is 23043945.
BOLT-INFO: Peephole: 1976 double jumps patched.
BOLT-INFO: Peephole: 289 tail call traps inserted.
BOLT-INFO: Peephole: 0 useless conditional branches removed.
BOLT-INFO: FOP optimized 4 redundant load(s) and 0 unused store(s)
BOLT-INFO: Frequency of redundant loads is 818 and frequency of unused stores is 0
BOLT-INFO: Frequency of loads changed to use a register is 818 and frequency of loads changed to use an immediate is 0
BOLT-INFO: FOP deleted 3 load(s) (dyn count: 8) and 0 store(s)
BOLT-INFO: FRAME ANALYSIS: 2497 function(s) were not optimized.
BOLT-INFO: FRAME ANALYSIS: 2320 function(s) (90.8% dyn cov) could not have its frame indices restored.
BOLT-INFO: Shrink wrapping moved 19 spills inserting load/stores and 0 spills inserting push/pops
BOLT-INFO: Shrink wrapping reduced 12733997 store executions (0.0% total instructions executed, 0.1% store instructions)
BOLT-INFO: Shrink wrapping failed at reducing 0 store executions (0.0% total instructions executed, 0.0% store instructions)
BOLT-INFO: Allocation combiner: 17 empty spaces coalesced (dyn count: 9376429).
 #0 0x0000560336033bd1 (/usr/lib/llvm-15/bin/llvm-bolt+0x1ad7bd1)
 #1 0x0000560336031a9c (/usr/lib/llvm-15/bin/llvm-bolt+0x1ad5a9c)
 #2 0x00005603360340e6 (/usr/lib/llvm-15/bin/llvm-bolt+0x1ad80e6)
 #3 0x00007f089cddb520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x0000560336d73a30 (/usr/lib/llvm-15/bin/llvm-bolt+0x2817a30)
 #5 0x00005603360de800 (/usr/lib/llvm-15/bin/llvm-bolt+0x1b82800)
 #6 0x00005603360dd00e (/usr/lib/llvm-15/bin/llvm-bolt+0x1b8100e)
 #7 0x00005603360d8a16 (/usr/lib/llvm-15/bin/llvm-bolt+0x1b7ca16)
 #8 0x00005603360d7e49 (/usr/lib/llvm-15/bin/llvm-bolt+0x1b7be49)
 #9 0x000056033607f61b (/usr/lib/llvm-15/bin/llvm-bolt+0x1b2361b)
#10 0x000056033607768b (/usr/lib/llvm-15/bin/llvm-bolt+0x1b1b68b)
#11 0x0000560334d9ec62 (/usr/lib/llvm-15/bin/llvm-bolt+0x842c62)
#12 0x00007f089cdc2d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
#13 0x00007f089cdc2e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#14 0x0000560334d9cd95 (/usr/lib/llvm-15/bin/llvm-bolt+0x840d95)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /usr/lib/llvm-15/bin/llvm-bolt ./python -o python.bolt -data=python.fdata -update-debug-sections -reorder-blocks=ext-tsp -reorder-functions=hfsort+ -split-functions -icf=1 -inline-all -split-eh -reorder-functions-use-hot-size -peepholes=all -jump-tables=aggressive -inline-ap -indirect-call-promotion=all -dyno-stats -use-gnu-stack -frame-opt=hot
make[1]: *** [Makefile:800: bolt-opt] Segmentation fault
make[1]: Leaving directory '/cpython/python3.11-3.11.0/build-static'

How to reproduce:

The crash happens during docker build -t whatever ., see the Dockerfile below. It takes ~30min to reach the place.

Dockerfile

FROM ubuntu:22.04

ENV LC_ALL=en_US.UTF-8 \
    SETUPTOOLS_USE_DISTUTILS=stdlib \
    DEBIAN_FRONTEND=noninteractive \
    TZ=Europe/Madrid \
    PYTHON_TARGET_VERSION="3.11.0-1+jammy1" \
    PYTHON_VERSION=3.11

ENV OPT="-pipe -fno-semantic-interposition -march=haswell -mabm -maes -mno-pku -mno-sgx --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=33792"
ADD patches/cpython_configure_ac.patch patches/cpython_makefile.patch patches/cpython_rules.patch /


RUN echo 'deb-src http://archive.ubuntu.com/ubuntu/ jammy main restricted' >>/etc/apt/sources.list && \
    echo 'deb-src http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted' >>/etc/apt/sources.list && \
    apt-get update && \
    apt-get upgrade -y && \
    apt-get install --no-install-suggests --no-install-recommends -y \
      ca-certificates wget \
      apt-utils git locales dpkg-dev devscripts software-properties-common \
      libexpat1-dev tzdata mime-support libsqlite3-0 libreadline8 \
      python3-distutils html2text libjs-sphinxdoc && \
    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
    locale-gen && \
    echo "deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-15 main" >>/etc/apt/sources.list.d/llvm.list && \
    wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && \
    apt-get update && \
    apt-get install -y bolt-15 llvm-15 && \
    export LLVM_SYMBOLIZER_PATH=/usr/lib/llvm-15/bin/llvm-symbolizer && \
    add-apt-repository -s ppa:deadsnakes/ppa && \
    mkdir /cpython && \
    cd /cpython && \
    apt-get source python$PYTHON_VERSION && \
    apt-get -s build-dep python$PYTHON_VERSION | grep "Inst " | cut -d" " -f2 | sort | tr '\n' ' ' >build_bloat && \
    DEBIAN_FRONTEND="noninteractive" TZ="Europe/Madrid" apt-get build-dep -y python$PYTHON_VERSION && \
    rm /etc/apt/sources.list.d/deadsnakes* /etc/apt/sources.list.d/llvm.list && \
    cd python$PYTHON_VERSION* && \
    sed -i 's/__main__/__skip__/g' Tools/scripts/run_tests.py && \
    dch --bin-nmu -Dunstable "Optimized build" && \
    echo 11 >debian/compat && \
    sed -i 's/debhelper (>= 9)/debhelper (>= 11)/g' debian/control.in && \
    patch configure.ac </cpython_configure_ac.patch && \
    patch Makefile.pre.in </cpython_makefile.patch && \
    patch debian/rules </cpython_rules.patch && \
    rm /cpython_configure_ac.patch /cpython_makefile.patch /cpython_rules.patch && \
    DEB_CFLAGS_SET="$OPT" DEB_LDFLAGS_SET="$OPT" dpkg-buildpackage -uc -b -j$(getconf _NPROCESSORS_ONLN) && \
    cd .. && \
	apt-get source python3 && \
    cd python3-defaults-*/debian && \
    for f in debian_defaults control control.in rules; do sed -i -e 's/3.10.6/3.11.0/g' -e 's/3.10/3.11/g' $f; done && \
    sed -i 's/python3.11:any (>= 3.11.0/python3.10:any (>= 3.10.0/g' control && \
    sed -i 's/python3-distutils (>= @STDLIBVER@/python3-distutils (>= 3.10.6-1~/g' control.in && \
    cd /cpython/python3-defaults-* && \
    dch -v "3.11.0-1+jammy1+b1" -Dunstable "3.11" && \
    dpkg-buildpackage -uc -b && \
    cd .. && \
    rm -f \
      2to3* \
      idle* \
      libpython3-all* \
      python3-all* \
      python3-examples* \
      python3-doc* \
      python3-full* \
      python3-nopie* \
      python3-venv* \
      libpython$PYTHON_VERSION-testsuite* \
      python$PYTHON_VERSION-examples* \
      python$PYTHON_VERSION-doc* \
      idle-python$PYTHON_VERSION* \
      python$PYTHON_VERSION-venv* \
      python$PYTHON_VERSION-full* && \
    echo "========" && ls && \
    apt-get purge -y dpkg-dev devscripts software-properties-common html2text bolt-15 llvm-15 $(cat build_bloat) && \
    apt-get autoremove -y && \
    dpkg -i *python3.11*.deb && \
    dpkg -i python3-minimal*.deb libpython3-stdlib*.deb && \
    rm -f python3-minimal*.deb libpython3-stdlib*.deb && \
    dpkg -i python3_*.deb && \
    dpkg -i *python3-*.deb && \
    apt-get purge -y python3.10* libpython3.10* && \
    apt-get autoremove -y && \
    echo "========" && python3 --version && ls /etc/python3* && \
    cd / && \
    rm -rf /cpython && \
    apt-mark hold python$PYTHON_VERSION python$PYTHON_VERSION-minimal libpython$PYTHON_VERSION libpython$PYTHON_VERSION-minimal && \
    wget -O - https://bootstrap.pypa.io/get-pip.py | python3 && \
    python3 -m pip install --no-cache-dir 'cython>=0.29.30' && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Dockerfile depends on the patch files at athenianco/athenian-api#3040

[llvmbot]

@llvm/issue-subscribers-bolt

[ptr1337]

Had the same issue with llvm 15.0.6.
Seems to be fixed with llvm-git

[aaupov]

Reproduced

[aaupov]

Reproduced with release/15.x (tags/llvmorg-15.0.6).

The issue appears to be with -update-debug-sections. Removing the flag avoids the issue but the debug information would become unavailable.

Trunk version of BOLT doesn't have this issue.

Between llvmorg-15.0.6 and trunk there were 7 fixes that affected debug info update:

f7a21317665a [BOLT][DWARF] Don't create extra .debug_str_offsets contributions
fcd7717ddf07 [BOLT][DWARF] Add support for DW_FORM_addr for DW_AT_call_return_pc
1ee74064e083 [BOLT][DWARF] Fix updating CU that has no entry in .debug_addr
928c2ba17966 [DWARF][BOLT] Fix handling of converting range accesss from ofset to index.
ccbf28b09d60 [BOLT][DWARF] Handle zero size DW_TAG_inlined_subroutine
b786e01f9340 [DWARF][BOLT] Handle getBinaryFunctionContainingAddress returning nullptr for DW_TAG_call_site
dd29b3c54201 [BOLT][DWARF] Fix handling of multiple DW_OP_addrx in an expression

with b786e01 being the one that fixes the crash. While it's possible to backport just it to llvm-15, other fixes also affect debugging experience (especially fcd7717), so the best suggestion is to update to the latest llvm-bolt package available from https://apt.llvm.org.

[vmarkovtsev]

Thanks a lot for debugging the crash 🙇‍♂️