Debuginfo tests don't control for ambient tools

[workingjubilee]

The debuginfo tests should probably use a specific debugger, like maybe an lldb we build with the compiler, so that they work consistently. In practice, they have been effectively broken on user's machines by just trying to use whatever is installed.

In other words: just because I use Linux doesn't mean I have gdb installed.

[workingjubilee]

The debuginfo test suite should also be insulated against gdb changing how it prints things, Because It Does.

[Noratrieb]

Some points:

• compiletest supports cdb, gdb and lldb, running the test suite for all the ones that are available (either via config.toml or via PATH).
• our LLVM builds do currently not contain lldb.

[workingjubilee]

That's... informative, thank you!

It does seem like we should reassess our strategy if we're gonna run a test suite that isn't guaranteed to pass even between different versions of a debugger on "any ol' random debugger lying around".

[bjorn3]

On macOS we can't ship our own debugger. Only an Apple signed lldb is allowed to use the debugging apis of the kernel.

[workingjubilee]

Wait, gdb is useless on macOS?

[bjorn3]

Supposedly you can use gdb on macOS jf you sign it with a self-signed code signing certificate which you add to the system keychain, mark as trusted for code signing and set the right entitlements when signing gdb. I don't know if you need to disable SIP for this to work though. We can't disable SIP on CI. https://dev.to/jasonelwood/setup-gdb-on-macos-in-2020-489k

[workingjubilee]

Hmm, okay. I think on Linux it is still desirable to control for the sheer amount of variability that Linux distros have, but I can believe that the debuginfo tests have to draw on the environment sometimes.

[jieyouxu]

I think it's fine if we can't necessarily control for the debugger used, but we should at least document it and/or report the debugger used when running the debuginfo test suite.

[workingjubilee]

yeah, that was also reported as #126628

[saethlin]

FYI all of the lldb tests have been failing for anyone who uses a rolling distro for the past 8 months because of llvm/llvm-project#70453

[jieyouxu]

In case anyone stumbled across this, IIRC lldb debuginfo tests (currently) require Python 3.10 dll to be present in PATH on Windows, not sure about Unixes but presumably only working with Python 3.10 is well-behaved.

[jieyouxu]

this has come back to bite us in #133107