Add new NuGet package, Microsoft.NETCore.Runtime.JIT.Tools, includes FileCheck and llvm-mca
#256
Conversation
| <?xml version="1.0" encoding="utf-8"?> | ||
| <Project> | ||
| <ItemGroup> | ||
| <File Include="$(_LLVMInstallDir)\bin\llvm-mca" TargetPath="runtimes\$(PackageTargetRuntime)\native\" /> |
There was a problem hiding this comment.
The main (mono) .props files include the .pdb/.dbg/.dwarf files (though when I downloaded one, I didn't see them, so I'm not sure what is actually going on). However, the objwriter ones do not, so I'm not sure what the convention is to follow here.
There was a problem hiding this comment.
It is probably fine - I think the reason not to include them is because of the package size. When looking at the .pdb file for llvm-mca, it was 60mb; FileCheck.pdb was 20mb.
|
@TIHan can you help me understand why a JIT test tool is going into objwriter branch? The JIT team already has a LLVM-based test tool (coredistools) that is getting built in dotnet/jitutils. Can this not be built in the same spot? I would prefer not having to deal with the engineering issues that are the likes of:
I don't mind if the JIT team is also picking up the ownership of ObjWriter with this. The component was originally written by the JIT team but ended up in the "whoever last touches it owns it" pit. |
|
@MichalStrehovsky This is fair to ask. Over the weekend, I've been thinking about whether or not we should somehow move these FileCheck/llvm-mca to JitUtils instead of them being in objwriter branch - for similar reasons as you said. That way we could just make the JitUtils be part of the package too. We ended up in this branch because we decided this would be a better place than in main, and it had all the infrastructure set up. I'm not committed to keeping this package here and we can move it elsewhere if we need too. We don't have a reason to have higher versions of these tools yet, but long term it would be good to probably have it live in JitUtils. |
What were the reasons why main is not the right fit? I think the same reasons apply for this branch.
Could you please investigate moving it elsewhere (JitUtils) soon? ObjWriter is a tolerated component of NativeAOT. Nobody really likes it. If we had the resources to write object emitters in C#, we would have written them in C#. I don't want the engineering systems around ObjWriter to increase in complexity beyond what is absolutely needed. Unless the JIT team would like to own ObjWriter and the engineering infrastructure around it, then I'm okay with anything. |
|
I also wasn't aware of coredistools package and its dependency on LLVM; I think had I known that I would have tried to do this work there. |
|
Originally I targeted main, but we thought it was just meant for mono. We were made aware of objwriter branch and felt that was closer to the JIT and had a higher LLVM version. |
I'm not sure how complicated the infrastructure is to set up in JitUtils or even in another branch for dotnet/llvm-project. We probably would want another branch here because I think it would be difficult to set up building LLVM tools in JitUtils. If it is not too bad to set up a new branch here and have the nuget packages get updated from that branch, then I guess we could do that fairly quickly. However, I do not know how to set that up. Otherwise at the moment, we can't spend that much more time on determining where this should live and doing the work to get that infrastructure working when this branch already has that set up. We don't plan on requiring an updated version of LLVM - we just want to use FileCheck as is - so there is no risk of breaking objwriter. |
…s `FileCheck` and `llvm-mca` (dotnet#256) https://github.com/dotnet/runtime is wanting to start writing assembly (x64/ARM64) verification tests. Instead of building our own tool to support writing those kinds of tests, we want to leverage LLVM's `FileCheck`. We also want to include `llvm-mca` at the request of @EgorBo This PR creates a new NuGet package for `dotnet/runtime` to consume which we named `Microsoft.NETCore.Runtime.JIT.Tools`. So far, this package only includes LLVM's `FileCheck` and `llvm-mca` tools.
* Apply llvm.patch Taken from https://github.com/dotnet/runtime/blob/7ab969c84ef05ba948c0075392716ce335b47744/src/coreclr/tools/aot/ObjWriter/llvm.patch. * Add objwriter library * Taken from https://github.com/dotnet/runtime/tree/7ab969c84ef05ba948c0075392716ce335b47744/src/coreclr/tools/aot/ObjWriter. * Updated README.md * Updated CMakeLists.txt to remove reference to CORECLR_INCLUDE_DIR. * Added cordebuginfo.h, cvconst.h, cfi.h from coreclr/inc at the above commit. * Build the ObjWriter package * Add ObjWriter API to set DWARF version (#161) Contributes to https://github.com/dotnet/runtimelab/issues/1738. * Add `.note.GNU-stack` section to produced executables (#162) Do this unconditionally because there's no scenario where we would need executable stack for managed code. * Remove Darwin workaround (#163) This caught my attention as I was looking at the ObjWriter. LLVM no longer emits a `LC_VERSION_MIN_MACOSX` load command unless we explicitly set a version. I don't see a difference in `llvm-objdump -macho -x foo.o` with/without these lines (I didn't bother myself to boot into macOS to run `otool`). * Fix llvm-dwarfdump warnings (#164) Fixes https://github.com/dotnet/runtimelab/issues/1535. No warnings left with llvm-dwarfdump from LLVM 12. * Revert "Fix llvm-dwarfdump warnings (#164)" (#218) This reverts commit afc9070. * Add new NuGet package, `Microsoft.NETCore.Runtime.JIT.Tools`, includes `FileCheck` and `llvm-mca` (#256) https://github.com/dotnet/runtime is wanting to start writing assembly (x64/ARM64) verification tests. Instead of building our own tool to support writing those kinds of tests, we want to leverage LLVM's `FileCheck`. We also want to include `llvm-mca` at the request of @EgorBo This PR creates a new NuGet package for `dotnet/runtime` to consume which we named `Microsoft.NETCore.Runtime.JIT.Tools`. So far, this package only includes LLVM's `FileCheck` and `llvm-mca` tools. * [ObjWriter] Enable DWARF debug information emitting for Mach-O (#269) * Account for GOT VariantKind on osx-arm64 (#185) * Add API for emitting compact unwind encoding, enforce DWARF encoding if not explicitly overridden * Add comment * Update ObjWriter to LLVM 14 API * Add support for generating uninitialized sections (#306) We support `.bss` but not custom sections that are bss-like. This adds such support. * Do not indiscriminately create text section (#312) If we ended up with nothing in the text section, this line would error LLVM out in: https://github.com/dotnet/llvm-project/blob/3db8d68195c17386557f1a258312bbae4051dc05/llvm/lib/MC/ELFObjectWriter.cpp#L1458-L1459 Because we generate a reference to the empty text section in the `aranges` section. I double checked and debugging on Linux still works fine without this. `SetCodeSectionAttribute` is an objwriter API and we have access to it from the managed side. We should be calling it from there if it's needed for something that I didn't realize (we do call it from the managed side for the `.managed` section, but that one actually has debug information generated, unlike `.text`). * Fix off-by-one error in DWARF reg-reg location (#317) The DWARF specification states that the form of an exprloc consists of an unsigned LEB128 length value, followed by the encoded location bytes of the specified length. For some reason we were adding one to the length value being emitted. This looks incorrect to me. The above calculation for REG-REG (a variable stored in two registers) correctly calculates the length of each register type tag, plus the size of the interpolating PIECE tags, plus the size of notation for each register. The extra byte looks wrong. I've tested this locally and it appears to resolve dotnet/runtime#77407. Unfortunately, it also causes llvm-dwarfdump --verify to constantly complain about missing base addresses. I can't confirm at the moment, but my suspicion is that this is revealing an existing bug. Even if this is somehow causing a new bug, I think the resulting symbols with this change are better than the alternative (no working symbols at all). * Setting context object file info * Add verbosity to linux x64 pipeline In order to understand what is happening with std path error. * Revert "Add verbosity to linux x64 pipeline" This reverts commit 5c4636e. * Upgrading linux build image * [Temporary] Adding verbosity to get more pipeline error info * Update image name for linux x64 * Fix Linux x64 build * Revert "[Temporary] Adding verbosity to get more pipeline error info" This reverts commit 9d76b36. * Updating Build_Linux_musl timeout * Update linux-musl Docker images * Fix linux-musl-x64 build * Setting clang/++ version 15 for linux musl * Copying clang/clan++ vars to unix-like OS * Fix cut & paste error * Fix objcopy and strip path in cross-compilation * Update azure-pipelines.yml $(ClangVersion) $(ClangPlusVersion) weren't defined for OSX and should be defined for every Linux * Bump timeout for Linux musl build * Clean up .gitignore * Consolidate Clang[Plus]Version into ClangVersionArg * Move CLANG_TARGET from environment into build parameter Always quote _BuildConfig on command line so empty value is not accidentally using next parameter as the value * Update URL in cordebuginfo.h to point to dotnet/runtime * Bump Windows build timeout to 210 * Fix a typo in compiler name * Revert $(_BuildConfig) -> "$(_BuildConfig)" change * Change ClangTarget to ClangTargetArg since apparently it gets propagated as environment variable into wrong steps * Fix inadvertent change * Bump timeout everywhere --------- Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com> Co-authored-by: Andy Gocke <andy@commentout.net> Co-authored-by: Will Smith <lol.tihan@gmail.com> Co-authored-by: Adeel Mujahid <3840695+am11@users.noreply.github.com> Co-authored-by: Brian Bohe <brianbohe@gmail.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
Description
https://github.com/dotnet/runtime is wanting to start writing assembly (x64/ARM64) verification tests. Instead of building our own tool to support writing those kinds of tests, we want to leverage LLVM's
FileCheck.We also want to include
llvm-mcaat the request of @EgorBoThis PR creates a new NuGet package for
dotnet/runtimeto consume which we namedMicrosoft.NETCore.Runtime.JIT.Tools. So far, this package only includes LLVM'sFileCheckandllvm-mcatools.// cc @markples @JulieLeeMSFT @EgorBo @akoeplinger