[build/ci] Enable native code analysis nightly

[pjcollins]

Context: aaa37c3

Updates the build to conditionally generate extra native libraries with
the ASAN and UBSAN runtime sanitizers when
'$(EnableNativeAnalyzers)' == 'true'.

We will only enable this behavior in our nightly builds, and the
*-checked+asan and *-checked+ubsa native libraries have been removed
from our regular installers.

[pjcollins]

@grendello please take a look at the general approach here when you get the chance. I am assuming we want to be running this ClangTidyCheck target during the build at some point, and also want to run some tests with /p:UseASAN=true and /p:UseUBSAN=true? Is there anything extra we need to do to validate test results that ran against these "checked" binaries?

[pjcollins]

Nightly run triggered here to test initial changes - https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=5362029&view=results

[grendello]

@grendello please take a look at the general approach here when you get the chance. I am assuming we want to be running this ClangTidyCheck target during the build at some point, and also want to run some tests with /p:UseASAN=true and /p:UseUBSAN=true? Is there anything extra we need to do to validate test results that ran against these "checked" binaries?

Yes, it would be good to run tests with these sanitizers enabled. They should run as usual, but we must capture stdout and stderr, as that's where the compiler will show any issues (and logcat, of course - it might be good to run adb logcat -G 16M before running these tests)

[pjcollins]

@grendello clang-tidy looks like it's failing for all architectures, here's one log: https://gist.github.com/pjcollins/b1aeefaddea98cdbd78505b12d402cd6

[pjcollins]

@grendello The tests which had support for running against these checked binaries appear to be crashing from this test run:
logcat-Release-Xamarin.Forms_Performance_Integration.txt
logcat-Release-Mono.Android_Tests.txt

Do any of the crash dumps look meaningful to you?

[grendello]

@grendello clang-tidy looks like it's failing for all architectures, here's one log: https://gist.github.com/pjcollins/b1aeefaddea98cdbd78505b12d402cd6

It appears it's getting confused by the macros. I need to run this locally and add exceptions to the code

[grendello]

@grendello The tests which had support for running against these checked binaries appear to be crashing from this test run: logcat-Release-Xamarin.Forms_Performance_Integration.txt logcat-Release-Mono.Android_Tests.txt

Do any of the crash dumps look meaningful to you?

They appear to have spotted an actual bug, but I can't tell where that is exactly as there are no symbols. I didn't find any APKs in the artifacts, do you think we could have them stored? With the apk we might be able to translate addresses to actual source lines.

[grendello]

@pjcollins Can you try applying this diff? It should leave debug symbols intact when building the sanitized runtimes:

diff --git a/src/monodroid/CMakeLists.txt b/src/monodroid/CMakeLists.txt
index f1aa3fc91..bf7aabf4a 100644
--- a/src/monodroid/CMakeLists.txt
+++ b/src/monodroid/CMakeLists.txt
@@ -35,9 +35,16 @@ endif()

 option(ENABLE_CLANG_ASAN "Enable the clang AddressSanitizer support" OFF)
 option(ENABLE_CLANG_UBSAN "Enable the clang UndefinedBehaviorSanitizer support" OFF)
+
+if(ENABLE_CLANG_ASAN OR ENABLE_CLANG_UBSAN)
+  set(STRIP_DEBUG_DEFAULT OFF)
+else()
+  set(STRIP_DEBUG_DEFAULT ON)
+endif()
+
 option(ENABLE_NET6 "Enable compilation for .NET6" OFF)
 option(ENABLE_TIMING "Build with timing support" OFF)
-option(STRIP_DEBUG "Strip debugging information when linking" ON)
+option(STRIP_DEBUG "Strip debugging information when linking" ${STRIP_DEBUG_DEFAULT})
 option(DISABLE_DEBUG "Disable the built-in debugging code" OFF)
 option(USE_CCACHE "Use ccache, if found, to speed up recompilation" ${CCACHE_OPTION_DEFAULT})

[pjcollins]

New test run https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=5371609&view=results

[pjcollins]

@grendello the test run above has .apk files and should include your debug symbol diff when you have a chance to take a look.

[pjcollins]

I filed #6433 and #6434 for the issues we've seen so far. I think we can merge these changes to remove native analysis pieces from our regular build, and incrementally fix the other issues in future PRs.