chrash in hotspot in handling perfparser output

[GitMensch]

I'm rerunning my steps from #394 (comment) only modifying the record command:

perf record --call-graph dwarf -F97 -e "{cycles,instructions}:S" ./x test ui --stage 1 --force-rerun

perf.data.gz

It crashes both hotspot-v1.4.1-263-ga8d1440-x86_64.AppImage and hotspot-v1.3.0-277-g2bcd36d-x86_64.AppImage with a segfault

/tmp/.mount_hotspoWNLRPL/AppRun.wrapped: line 6: 2288130 Segmentation fault (core dumped) LD_LIBRARY_PATH="$d/usr/lib":$LD_LIBRARY_PATH "$bin/hotspot" "$@"

Originally posted by @the8472 in #394 (comment)

I can reproduce the crash on the gitpod environment but get a

ASSERT failure in QVector::operator[]: "index out of range", file /usr/include/x86_64-linux-gnu/qt5/QtCore/qvector.h, line 457
Aborted

instead, so I'm having a look at analyzing that next

Checking with GDB that's the backtrace:

Thread 22 "GlobalQueue[03]" hit Breakpoint 1, 0x00005569a6635d80 in qt_assert_x(char const*, char const*, char const*, int)@plt ()
bt
#0  0x00005569a6635d80 in qt_assert_x(char const*, char const*, char const*, int)@plt ()
#1  0x00005569a6675b90 in QVector<Data::PerCostFrequencyData>::operator[] (this=0x7f3d7c014c38, i=-1) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qvector.h:457
#2  0x00005569a666f060 in PerfParserPrivate::addSampleToFrequencyData (this=0x7f3d91ffa6a0, sample=...) at /workspace/hotspot/src/parsers/perf/perfparser.cpp:1065
#3  0x00005569a666f1d3 in PerfParserPrivate::addSample (this=0x7f3d91ffa6a0, sample=...) at /workspace/hotspot/src/parsers/perf/perfparser.cpp:1074
#4  0x00005569a666c352 in PerfParserPrivate::parseEvent (this=0x7f3d91ffa6a0) at /workspace/hotspot/src/parsers/perf/perfparser.cpp:737
#5  0x00005569a666ba6f in PerfParserPrivate::tryParse (this=0x7f3d91ffa6a0) at /workspace/hotspot/src/parsers/perf/perfparser.cpp:684
#6  0x00005569a666b4aa in PerfParserPrivate::setInput(QIODevice*)::{lambda()#1}::operator()() const (__closure=0x7f3d7c006020) at /workspace/hotspot/src/parsers/perf/perfparser.cpp:630
#7  0x00005569a668e313 in QtPrivate::FunctorCall<QtPrivate::IndexesList<>, QtPrivate::List<>, void, PerfParserPrivate::setInput(QIODevice*)::{lambda()#1}>::call(PerfParserPrivate::setInput(QIODevice*)::{lambda()#1}&, void**) (f=..., arg=0x7f3d91ffa080) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:146
#8  0x00005569a668bf06 in QtPrivate::Functor<PerfParserPrivate::setInput(QIODevice*)::{lambda()#1}, 0>::call<QtPrivate::List<>, void>(PerfParserPrivate::setInput(QIODevice*)::{lambda()#1}&, void*, void**) (f=..., arg=0x7f3d91ffa080) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:256
#9  0x00005569a66861d0 in QtPrivate::QFunctorSlotObject<PerfParserPrivate::setInput(QIODevice*)::{lambda()#1}, 0, QtPrivate::List<>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) (which=1, this_=0x7f3d7c006010, r=0x7f3d91ffa6a0, a=0x7f3d91ffa080, ret=0x0) at /usr/include/x86_64-linux-gnu/qt5/QtCore/qobjectdefs_impl.h:443
(gdb) frame 3
(gdb) p cost
$1 = (const (anonymous namespace)::SampleCost &) @0x7f3d7c00a848: {attributeId = -1, cost = 0}

the reason seems to be that the QPERFSTREAM that hotspot reads from hotspot-perparser has unexpected content.

That can be acquired by /workspace/hotspot/build/lib/x86_64-linux-gnu/libexec/hotspot-perfparser --input perf.data --max-frames 1024 > perfparser.out, if needed.

[GitMensch]

It seems that is handled in other places:

        if (type < 0) {
            qCWarning(LOG_PERFPARSER) << "Unexpected attribute id:" << sampleCost.attributeId << "Only know about"
                                      << attributeIdsToCostIds.size() << "attributes so far";
            return;
        }

So I'll do something similar.

[GitMensch]

@the8472 please check the appimage from https://github.com/KDAB/hotspot/actions/runs/7291612961 - but I'd guess that fixes the crash.

[the8472]

It does

[the8472]

This let me figure out what the issue was with "{cycles,instructions}:S". perf doesn't attach to subprocesses for some reason.
Adding -a to do a system-wide record works. But then hotspot crashes again (either segfault or getting OOM-killed)

[GitMensch]

if you can reproduce the crash and the perf recording is not huge - feel free to upload perf.data.gz (less than 50MB) here and I may be able to have another look.

[the8472]

It's several GB in size. Also it's a whole-system profile, I'd rather not share that.
Perhaps it is related to the -a flag and processing debug info for all the processes on the system.

[GitMensch]

When using this flag, it is most reasonable to limit the trigger of the record to some seconds and record with a small frequency.