Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade LLVM to 16 #418

Closed
wants to merge 10,000 commits into from
Closed

Conversation

directhex
Copy link
Member

@directhex directhex commented May 22, 2023

This PR updates our internal LLVM fork from 14 to 16.

One of the requirements of this is LLVM 16 depends on C++17, which is not available in our CBL Mariner target crossroots. To work around this, we need to build/ship/link against LLVM's C++ library, libc++, on Linux. ALL consumers which link against files produced from llvm-project.git need to switch to our build of libc++ instead of distro-provided libstdc++ (from GCC).

We are using rpath to direct searches for libc++ to ../lib/ which is a good match for the Mono consumers, but may be confusing for the CoreCLR consumers - we will need to see how this comes out in the wash.

LLVM 16 is a strict requirement for shipping self-build EMSDK, as we rely upon compiler flags which are in 16 but not 14. And from experience, we know trying to maintain multiple versions of LLVM within a single release is a huge maintenance burden.

mysterymath and others added 30 commits February 7, 2023 19:00
Indexed profiles already have a sorted and uniqued binary ID list, and
due to this, duplicates are harmless in the list of binary IDs found,
since it's set_differenced from the list in the indexed profile.

Differential Revision: https://reviews.llvm.org/D136702

(cherry picked from commit 0a51eed)
… Windows.

This is an alternative to D141074 to fix the problem by adjusting
the precision control dynamically.

Reviewed By: icedrocket

Differential Revision: https://reviews.llvm.org/D142178

(cherry picked from commit 11fb09e)
When lowering predicate permute builtins we incorrectly assume only
the typically "active" bits for the specified element type play a
role with all other bits zero'd.  This is not the case because all
bits are significant, with the element type specifying how they
are grouped:

  b8  - permute using a block size of 1 bit
  b16 - permute using a block size of 2 bits
  b32 - permute using a block size of 4 bits
  b64 - permute using a block size of 8 bits

The affected builtins are svrev, svtrn1, svtrn2, svuzp1, svuzp2,
svzip1 and svzip2.

This patch adds new intrinsics to support these operations and
changes the builtin lowering code to emit them.  The b8 case remains
unchanged because for that operation the existing intrinsics work
as required and their support for other predicate types has been
maintained as useful if only as a way to test the correctness of
their matching ISD nodes that code generation relies on.

Differential Revision: https://reviews.llvm.org/D142065

(cherry picked from commit 26b79ca)
If you are missing the iOS SDK on your macOS (for example you don't have
full Xcode but just CommandLineTools) then CMake currently errors
out without a helpful message. This patch disables iOS support in
compiler-rt if the iOS SDK is not found. This can be overriden by
passing -DCOMPILER_RT_ENABLE_IOS=ON.

Reviewed By: delcypher, thetruestblue

Differential Revision: https://reviews.llvm.org/D133273

(cherry picked from commit 78fb021)
Link: llvm#60346

Reviewed By: efriedma, void

Differential Revision: https://reviews.llvm.org/D142923

(cherry picked from commit 1cecfa4)
If this successor list is not correct, then branch-folding may
incorrectly think that the indirect target is dead and remove it. This
results in a dangling reference to the removed block as an operand to
the INLINEASM_BR, which later will get AsmPrinted into code that doesn't
assemble.

This was made more obvious by, but is not a regression of
https://reviews.llvm.org/D130316.

Fixes: llvm#60346

Reviewed By: efriedma, void

Differential Revision: https://reviews.llvm.org/D142924

(cherry picked from commit 07c7784)
These dissembler symbols are not used by LTO (see Apple ld64's use
in check-llvm-tools-lto). On ELF platforms, these symbols are not defined and are
rejected by ld --no-undefined-version.

I think this is a more proper solution than D139932 and this fixes
-DBUILD_SHARED_LIBS=on for ELF as well.

(cherry picked from commit c294bdd)
Remarks.exports is only intended for NOT (BUILD_SHARED_LIBS OR LLVM_LINK_LLVM_DYLIB)
builds.

For (unintended use case) BUILD_SHARED_LIBS OR LLVM_LINK_LLVM_DYLIB (the latter
is used by some Linux distros), the library defines just one symbol on ELF.
There is no need to use a version script.

I think this is a more proper solution than D139932 and fixes `symbol not
defined` errors after lld default change D135402.

(cherry picked from commit bf07de3)
…ther sources

This code is walking back through a worklist of sources. All of the sources need to be sign extending for the result to be true. We had a case which returned rather than continued, which causes a miscompile when another source was not sign extended. The flawed logic was introduced in Dec 22, by change 844430b.

This was recently exposed in a stage2 build of llvm-tablegen when we switched from using llvm::Optional to std::optional. The stars aligned in just the wrong way, and we started actively miscompiling idiomatic optional usage. std::optional<uint32_t> appears to use the top 32 bits of the word on RV64 for its tag.

Differential Revision: https://reviews.llvm.org/D143594

(cherry picked from commit db6bee5)
This is only useful when building the project in a "standalone" way: that is by
invoking cmake pointing at mlir/ to build against an already built LLVM.

Fixes llvm#60574
As explained in a comment in https://reviews.llvm.org/D134124, we tried
landing this unconditionally but this actually bit some users who were
sharing std::unordered_map across an ABI boundary. This shows that the
ABI break is not benign and it should be guarded behind ABI v2.

Differential Revision: https://reviews.llvm.org/D143688

(cherry picked from commit 0271843)
…for ClangScanDeps (1/4)

This patch prepares the necessary interfaces in the preprocessor part
for D137527 since we need to recognize if we're in a module unit, the
module kinds and the module declaration and the module we're importing
in the preprocessor.

Differential Revision: https://reviews.llvm.org/D137526
… Named Modules in P1689 format (2/4)

Close llvm#51792
Close llvm#56770

This patch adds ClangScanDeps support for C++20 Named Modules in P1689
format. We can find the P1689 format at:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html.
After we land the patch, we're able to compile C++20 Named
Modules with CMake! And although P1689 is written by kitware people,
other build systems should be able to use the format to compile C++20
Named Modules too.

TODO: Support header units in P1689 Format.
TODO2: Support C++20 Modules in the full dependency format of
ClangScanDeps. We also want to support C++20 Modules and clang modules
together according to
https://discourse.llvm.org/t/how-should-we-support-dependency-scanner-for-c-20-modules/66027.
But P1689 format cares about C++20 Modules only for now. So let's focus
on C++ Modules and P1689 format. And look at the full dependency format
later.

I'll add the ReleaseNotes and Documentations after the patch get landed.

Reviewed By: jansvoboda11

Differential Revision: https://reviews.llvm.org/D137527
…ecified compilation database in P1689 (3/4)

In a private chat with @ben.boeckel , we get in consensus it would be
great for cmake if the invocation of clang-scan-deps can get rid of
compilation database. Due to the compilation database can't do very well
for the files which are not existed yet. @ben.boeckel may have more
context to add here.

This patch should be innocent for others usages.

Reviewed By: jansvoboda11

Differential Revision: https://reviews.llvm.org/D137534
…DeclStateTest.cpp

In file included from /data/jiefu/llvm-project/clang/unittests/Lex/ModuleDeclStateTest.cpp:22:
/data/jiefu/llvm-project/third-party/unittest/googletest/include/gtest/gtest.h:1526:11: error: comparison of integers of different signs: 'const unsigned long' and 'const int' [-Werror,-Wsign-compare]
  if (lhs == rhs) {
      ~~~ ^  ~~~
/data/jiefu/llvm-project/third-party/unittest/googletest/include/gtest/gtest.h:1553:12: note: in instantiation of function template specialization 'testing::internal::CmpHelperEQ<unsigned long, int>' requested here
    return CmpHelperEQ(lhs_expression, rhs_expression, lhs, rhs);
           ^
/data/jiefu/llvm-project/clang/unittests/Lex/ModuleDeclStateTest.cpp:124:3: note: in instantiation of function template specialization 'testing::internal::EqHelper::Compare<unsigned long, int, nullptr>' requested here
  EXPECT_EQ(Callback->importNamedModuleNum(), 0);
  ^
/data/jiefu/llvm-project/third-party/unittest/googletest/include/gtest/gtest.h:2027:54: note: expanded from macro 'EXPECT_EQ'
  EXPECT_PRED_FORMAT2(::testing::internal::EqHelper::Compare, val1, val2)
                                                     ^
1 error generated.
…cy file within P1689 format (4/4)

Required in https://reviews.llvm.org/D137534.

The build systems needs the information to know that "header X changed,
scanning may have changed, so please rerun scanning". Although it is
possible to get the information by running clang-scan-deps for the
second time with make format, it is not user friendly clearly.

Reviewed By: jansvoboda11

Differential Revision: https://reviews.llvm.org/D139168
Don't use the ldrd instruction; that one requires armv5te. Instead
do two separate loads (or only one if OMPT_SUPPORT isn't defined).

This should fix llvm#60370.

Differential Revision: https://reviews.llvm.org/D143683

(cherry picked from commit 89197b5)
…y printer

So far, the pretty printer for `std::coroutine_handle` internally
dereferenced the contained frame pointer displayed the `promise`
as a sub-value. As noticed in https://reviews.llvm.org/D132624
by @labath, this can lead to an endless loop in lldb during printing
if the coroutine frame pointers form a cycle.

This commit breaks the cycle by exposing the `promise` as a pointer
type instead of a value type. The depth to which the `frame variable`
and the `expression` commands dereference those pointers can be
controlled using the `--ptr-depth` argument.

Differential Revision: https://reviews.llvm.org/D132815

(cherry picked from commit 8aa3137)
…e_handle` pretty printer

The pretty printer for `std::coroutine_handle` was running into
> Assertion failed: (target_ctx != source_ctx && "Can't import into itself")
from ClangASTImporter.h, line 270.

This commit fixes the issue by removing the `CopyType` call from the
pretty printer. While this call was necessary in the past, it seems to
be no longer required, at least all test cases are still passing. Maybe
something changed in the meantime around the handling of `TypesystemClang`
instances. I don't quite understand why `CopyType` was necessary earlier.

I am not sure how to add a regression test for this, though. It seems
the issue is already triggered by the exising `TestCoroutineHandle.py`,
but API tests seem to ignore all violations of `lldbassert` and still
report the test as "passed", even if assertions were triggered

Differential Revision: https://reviews.llvm.org/D143127

(cherry picked from commit 54d4a25)
Prevent errors and crash dumps for broken pipes on Windows.

Fixes: llvm#48672

Differential Revision: https://reviews.llvm.org/D142224

(cherry picked from commit 0b704d9)
Do a large edit of the Clang release notes.

This is meant to be merged into the Clang 16.x branch.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D142578
As describe in

llvm#60363

the following DebugInfo LogicalView Tests unit tests failed:

- ELFReader
- SelectElements

The tests fail only on the OSX-64 platform with the CMake options:

-DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON

Using the same options on a Linux platform all the tests pass:
- https://lab.llvm.org/buildbot/#/builders/196
- llvm-x86_64-debian-dylib

Basically it is a dynamic library initialization affecting a static
instance for the string pool (LVStringPool).

That string pool instance is accessed by all the logical elements
to store/retrieve any associated string during the creation of the
logical view.

For a logical view comparison, both logical readers (Reference and
Target) use retrieved indexes when comparing their strings.

Moved the static instance to LVSupport module (unnamed namespace).

Reviewed By: jmorse

Differential Revision: https://reviews.llvm.org/D143716

(cherry picked from commit 92716a4)
…s using 'target=...'

This patch aims to restrict the emit-asm-from-mlir.mlir test to the targets that
the test cases expect using the 'target=...' LIT expression.

rG9067de2a43145411eff8b7782a404c5834249717 has previously attempted to resolve
the LIT failure on the ppc64le-flang-rhel-clang buildbot coming from the
emit-asm-from-mlir.mlir test by restricting the test case for AArch64, x86 and
ARM (as long as these targets are built). The original solution of adding
"*-registered-target" works on ppc64le-flang-rhel-clang because this bot only
builds PowerPC in LLVM_TARGETS_TO_BUILD.

When we build all targets and also build the flang project, this test case will
fail because despite being on PowerPC, we are still building the AArch64, x86
and ARM targets which will cause this test to run. These targets expect the ret
instruction to be produced in this test case, while PowerPC produces a blr instead.

Differential Revision: https://reviews.llvm.org/D143366

(cherry picked from commit 71513f7)
Source tarball's are used from some distribution to build the project

Reviewed By: Amir

Differential Revision: https://reviews.llvm.org/D143809

(cherry picked from commit b673135)
@directhex
Copy link
Member Author

Green except for ARM32 Windows, which is being removed in #419

@directhex directhex changed the title [DO NOT MERGE] Upgrade LLVM to 16 Upgrade LLVM to 16 Jun 2, 2023
@directhex directhex marked this pull request as ready for review June 2, 2023 12:58
@SamMonoRT
Copy link
Member

@jandupej @vargaz - have we done and local validation ?

@vargaz
Copy link

vargaz commented Jun 2, 2023

Comments:

  • We use to maintain our branches by rebasing our changes on top of the llvm release branches, are we going to stop doing that ?
  • emscripten is probably tested with a fixed llvm version which might not be the one our fork is based on, so it might not actually work.

@directhex
Copy link
Member Author

directhex commented Jun 2, 2023

  • emscripten is probably tested with a fixed llvm version which might not be the one our fork is based on, so it might not actually work.

Emscripten does the literal opposite of that. It checks out LLVM git main and builds with that. Building the 3.1.34 tag today will give you a completely different emscripten than building 3.1.34 tag a week ago, which gives you a completely different result from when 3.1.34 was tagged. There is nothing resembling "fixed" in the emsdk build system.

@directhex
Copy link
Member Author

We use to maintain our branches by rebasing our changes on top of the llvm release branches, are we going to stop doing that ?

That's doable too.

@directhex directhex mentioned this pull request Jun 2, 2023
vargaz and others added 5 commits June 6, 2023 08:49
… new --mono-api-version option which returns a numerical version which is used by mono to detect which version of the mono llvm fork it is compiled with.

(cherry picked from commit 8d01bed)
…hose associated stack slot index will be saved in MachineFunction, and later can be saved in the mono specific EH tables.

(cherry picked from commit d8126bd)
@directhex
Copy link
Member Author

Closing for now, we have the dotnet/main-16.x merge done & will keep working in there for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.