Upgrade to LLVM 16 & C++17, and bundle libc++ on Linux

[directhex]

This includes building against, linking, and shipping our own libc++ in lieu of system libstdc++, for LLVM builds. This is because LLVM 16 has a hard requirement on C++17 - our existing code assumes C++11

[filipnavara]

Shrink the min Dwarf warning limit.

Sorry. I plead guilty.

[directhex]

Shrink the min Dwarf warning limit.

Sorry. I plead guilty.

Plz stop fixing bugs, those were load-bearing bugs

[directhex]

This affects folks on the NativeAOT side too. Maybe @agocke, @jkotas are the right people to loop in?

[jkotas]

[jkotas]

I am wondering whether it would be better to link libc++.so and libc++abi.so.1 statically into libobjwriter.so, so there is no risk of it colliding with system libraries that may depend on C++ runtime.

[agocke]

Agreed, statically linking libc++ would be fine with me.

Also, the warning shrink looks good to me, that may also have been impacted by dotnet/llvm-project#444

[filipnavara]

Also, the warning shrink looks good to me, that may also have been impacted by dotnet/llvm-project#444

Didn't mean to steal credit from you. 😅 Anyway, we are (back) at zero warnings with llvm-dwarfdump.

[agocke]

Nope, no worries, you've done tons in this space and deserve the credit. I was just pointing out that, if that change hadn't yet been integrated, it might cause us to lower the count once again. I have one more change that will lower the count again on Linux (for some reason Linux still seems to have errors, but Mac doesn't).

[filipnavara]

for some reason Linux still seems to have errors, but Mac doesn't

They ship different dwarfdump. macOS ships the LLVM one, which reports less errors but supports proprietary Apple sections (which were later replaced by DWARF5 standard).

[directhex]

If someone wants to run experiments on static linking vs dynamic, they're welcome to - though the place to start that work is in the llvm-project repo where objwriter is generated. In my testing, via use of rpath, dynamic linking is working at least well enough for CI.

I don't think static linking is a realistic option for Mono - the bloat on 20-30 clang/llvm binaries would push us past the Nupkg size limit again

[jkotas]

This introduced build break in Android extra-platforms Mono legs:

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/328385/logs/223

2023-07-03T16:59:17.6848371Z   CMake Error at /usr/share/cmake-3.21/Modules/CMakeTestCXXCompiler.cmake:62 (message):
2023-07-03T16:59:17.6856690Z     The C++ compiler
2023-07-03T16:59:17.6856932Z   
2023-07-03T16:59:17.6857259Z       "/usr/bin/c++"
2023-07-03T16:59:17.6857646Z   
2023-07-03T16:59:17.6857894Z     is not able to compile a simple test program.
2023-07-03T16:59:17.6858043Z   
2023-07-03T16:59:17.6858208Z     It fails with the following output:
2023-07-03T16:59:17.6858340Z   
2023-07-03T16:59:17.6858549Z       Change Dir: /__w/1/s/artifacts/obj/mono/android.arm64.Release/cross/CMakeFiles/CMakeTmp
2023-07-03T16:59:17.6858948Z       
2023-07-03T16:59:17.6859436Z       Run Build Command(s):/usr/bin/make -f Makefile cmTC_e98b0/fast && /usr/bin/make  -f CMakeFiles/cmTC_e98b0.dir/build.make CMakeFiles/cmTC_e98b0.dir/build
2023-07-03T16:59:17.6859987Z       make[1]: Entering directory '/__w/1/s/artifacts/obj/mono/android.arm64.Release/cross/CMakeFiles/CMakeTmp'
2023-07-03T16:59:17.6860286Z       Building CXX object CMakeFiles/cmTC_e98b0.dir/testCXXCompiler.cxx.o
2023-07-03T16:59:17.6861105Z       /usr/bin/c++   -I/__w/1/s/artifacts/obj/mono/android.arm64.Release/llvm//x64/include/c++/v1 -L/__w/1/s/artifacts/obj/mono/android.arm64.Release/llvm//x64/lib -stdlib=libc++  -o CMakeFiles/cmTC_e98b0.dir/testCXXCompiler.cxx.o -c /__w/1/s/artifacts/obj/mono/android.arm64.Release/cross/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
2023-07-03T16:59:17.6862084Z       c++: error: unrecognized command-line option ‘-stdlib=libc++’
2023-07-03T16:59:17.6862397Z       make[1]: *** [CMakeFiles/cmTC_e98b0.dir/build.make:78: CMakeFiles/cmTC_e98b0.dir/testCXXCompiler.cxx.o] Error 1
2023-07-03T16:59:17.6862993Z       make[1]: Leaving directory '/__w/1/s/artifacts/obj/mono/android.arm64.Release/cross/CMakeFiles/CMakeTmp'
2023-07-03T16:59:17.6863231Z       make: *** [Makefile:127: cmTC_e98b0/fast] Error 2
2023-07-03T16:59:17.