LLVM 13 and 14 #7613
LLVM 13 and 14 #7613
Conversation
|
|
ghost
mentioned this pull request
|
I detected other pull requests that are modifying llvm/all recipe: This message is automatically generated by https://github.com/ericLemanissier/conan-center-conflicting-prs so don't hesitate to report issues/improvements there. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Failure in build 12 ( An unexpected error happened and has been reported. Help is on its way! 🏇 |
|
I took a look at job pool calculation, iam still unsure if this is solid enough to add it, but I want to at least share it because I brought it up.
The easiest way is therefore to set diff --git a/llvm/cmake/modules/HandleLLVMOptions.cmake b/llvm/cmake/modules/HandleLLVMOptions.cmake
index fcaa8f20bf94..00c6fbf6fa8e 100644
--- a/llvm/cmake/modules/HandleLLVMOptions.cmake
+++ b/llvm/cmake/modules/HandleLLVMOptions.cmake
@@ -31,8 +31,25 @@ string(TOUPPER "${LLVM_ENABLE_LTO}" uppercase_LLVM_ENABLE_LTO)
# Ninja Job Pool support
# The following only works with the Ninja generator in CMake >= 3.0.
+if (NOT LLVM_RAM_PER_COMPILE_JOB)
+ set(LLVM_RAM_PER_COMPILE_JOB "2000")
+endif()
set(LLVM_PARALLEL_COMPILE_JOBS "" CACHE STRING
"Define the maximum number of concurrent compilation jobs (Ninja only).")
+
+cmake_host_system_information(RESULT AVAILABLE_PHYSICAL_MEMORY QUERY AVAILABLE_PHYSICAL_MEMORY)
+cmake_host_system_information(RESULT NUMBER_OF_LOGICAL_CORES QUERY NUMBER_OF_LOGICAL_CORES)
+if(LLVM_RAM_PER_COMPILE_JOB)
+ math(EXPR memory_available_jobs "${AVAILABLE_PHYSICAL_MEMORY} / ${LLVM_RAM_PER_COMPILE_JOB}" OUTPUT_FORMAT DECIMAL)
+ if (memory_available_jobs LESS 1)
+ set(memory_available_jobs 1)
+ endif()
+ if (memory_available_jobs LESS NUMBER_OF_LOGICAL_CORES)
+ set(LLVM_PARALLEL_COMPILE_JOBS "${memory_available_jobs}")
+ else()
+ set(LLVM_PARALLEL_COMPILE_JOBS "${NUMBER_OF_LOGICAL_CORES}")
+ endif()
+endif()
if(LLVM_PARALLEL_COMPILE_JOBS)
if(NOT CMAKE_GENERATOR MATCHES "Ninja")
message(WARNING "Job pooling is only available with Ninja generators.")
@@ -42,8 +59,22 @@ if(LLVM_PARALLEL_COMPILE_JOBS)
endif()
endif()
+if (NOT LLVM_RAM_PER_LINK_JOB)
+ set(LLVM_RAM_PER_LINK_JOB "14000")
+endif()
set(LLVM_PARALLEL_LINK_JOBS "" CACHE STRING
"Define the maximum number of concurrent link jobs (Ninja only).")
+if(LLVM_RAM_PER_LINK_JOB)
+ math(EXPR memory_available_jobs "${AVAILABLE_PHYSICAL_MEMORY} / ${LLVM_RAM_PER_LINK_JOB}" OUTPUT_FORMAT DECIMAL)
+ if (memory_available_jobs LESS 1)
+ set(memory_available_jobs 1)
+ endif()
+ if (memory_available_jobs LESS NUMBER_OF_LOGICAL_CORES)
+ set(LLVM_PARALLEL_LINK_JOBS "${memory_available_jobs}")
+ else()
+ set(LLVM_PARALLEL_LINK_JOBS "${NUMBER_OF_LOGICAL_CORES}")
+ endif()
+endif()
if(CMAKE_GENERATOR MATCHES "Ninja")
if(NOT LLVM_PARALLEL_LINK_JOBS AND uppercase_LLVM_ENABLE_LTO STREQUAL "THIN")
message(STATUS "ThinLTO provides its own parallel linking - limiting parallel link jobs to 2.")
|
|
For anyone interested, update the recipe but wasnt sure to create a pr to conan-center or to rdeterres work.
Just one note, for this part i have no idea how to fix it or if its needed: self.cpp_info.components[component].build_modules["cmake_find_package"].extend([
self._alias_module_file_rel_path,
self._old_alias_module_file_rel_path,
])
self.cpp_info.components[component].build_modules["cmake_find_package_multi"].extend([
self._alias_module_file_rel_path,
self._old_alias_module_file_rel_path,
]) |
|
I Kindly request help from the conan team here. LLVM is a really important piece of stack and we are trying to merge this request since at least LLVM 9. I took over an abandoned MR and ported to a newer LLVM version, then abandoned if after a while, someone took over, ported to another newer LLVM version, then abandoned it after a while, then someone else took over... I'm currently using different bash / powershell scripts to download / compile llvm with the flags that I need on all systems bypassing conan, and that's really not something that I should be worried about doing. Should we keep pushing this MR, should we wait for conan2, what should we do here? |
|
Fun you say that... I gave up on trying to package llvm/clang for Conan exactly because it doesn't support a workflow that I'd like... I'd like for it to be possible for Conan to download or build clang, and then use that just built compiler to all the other dependencies and the project itself... But that can't be done... So unless you're using the llvm libraries, packaging compilers isn't useful. |
That's exactly what I'm doing, using libclang and libllvm. I'm not trying to use the binaries as compilers. |
|
I would be happy to see llvm in public conan. I had no time for a look into conan 2, so I have no idea how long it will take. We are using the version from my pr daily, updated it for 15.0.0. There you have the possibility to keep certain binaries clang / clang++ aso which we are using to decouple our application from a specific compiler. We are resolving the current llvm binary folder from conan and using clang to compile test files to llvm IR. So I can work with clang 15 but the project is using clang-14 for IR file generation. Secondly cmake can be used from conan with Ok tested it with my state of the recipe: @ruilvo
|
|
I too would like to see an llvm recipe in conan |
The compiler can now be built standalone much more easily using the Conan package manager for dependencies. ### Dependencies and packaging A `conanfile.py` and corresponding `conandata.yml` have been added at the repo root. These files are used by the Conan package manager to determine which external libraries are needed to build the compiler. In the build steps outlined below, `conan install` is used to instruct Conan to fetch and build necessary libraries. The `conanfile.py` also creates a Conan package for the compiler itself, called `qss-compiler`, which can be installed by other projects. All external libraries exist on [Conan center](https://conan.io/center/), except for: - llvm-core - clang-tools-extra - qasm For these libraries, we provide in-tree Conan recipes, which must be registered with your host's Conan installation. To do this, run `conan_deps.sh`. Run this any time you modify any of these in-tree recipes. There are two primary reasons we maintain these recipes in-tree: 1. They aren't available in Conan Center. 2. We update them more quickly than would be suitable for Conan Center (e.g. it takes [months to years](conan-io/conan-center-index#7613) to get folks on Conan Center to agree on `llvm-core` version bumps). ### Building Follow these steps to install dependencies with Conan and build the compiler. 1. Run `conan_deps.sh` to export local Conan recipes for `llvm-core`, `clang-tools-extra` and `qasm`. 3. Create a `build` directory and enter it. 4. Run `conan install --build=outdated <path to repo root>` 5. Run `conan build <path to repo root>` or use CMake to generate your favorite build system and invoke that. 6. Run tests with `ninja check-tests` (build system is `ninja` by default, use your own if relevant). If you don't want to use Conan, make sure that you configure your system with the necessary dependencies and tell CMake where to find corresponding `find_package` modules (e.g. by specifying `-DCMAKE_TOOLCHAIN_FILE=<path to toolchain>`). ### CI CI is now available for Linux x64 to validate builds, unit tests, and LLVM LIT testing for QASM and the mock target. This project builds both LLVM Core and Clang Tools Extra (via Conan recipes), which takes several hours. To accelerate development, a GitHub Cache Action is used to cache the resulting library binaries along with all other Conan dependencies. Due to GitHub's cache size limit, there's only enough space to cache builds on the `main` branch. However, PR builds will load the Conan cache from `main`, so they should finish in around 10 minutes. PRs that change the LLVM recipes will need to wait for a full build (~5 hours). Once a PR is merged to main, any changes to Conan files in the repository will trigger a full build / cache rebuild to ensure the cache is cleared of any stale dependencies (to prevent unbounded growth).
|
Edit: continued / current state here: #17509 (comment) |
| 'LLVM_ENABLE_LIBPFM': False, | ||
| 'LLVM_ENABLE_LIBEDIT': False, | ||
| 'LLVM_ENABLE_FFI': self.options.with_ffi, | ||
| 'LLVM_ENABLE_ZLIB': self.options.get_safe('with_zlib', False), |
There was a problem hiding this comment.
We should set these variables to FORCE_ON instead of True so they add the REQUIRED keyword. Otherwise if CMake fails to find the dependencies they get silently ignored.
There was a problem hiding this comment.
This is the code form version 16.0.03
if(LLVM_ENABLE_ZLIB)
if(LLVM_ENABLE_ZLIB STREQUAL FORCE_ON)
find_package(ZLIB REQUIRED)
elseif(NOT LLVM_USE_SANITIZER MATCHES "Memory.*")
find_package(ZLIB)
endif()
if(ZLIB_FOUND)
...
There was a problem hiding this comment.
Its also there in in llvm 13 and documentation showed that this is the case for other options as well. Can add it in my pr.
|
We are currently not prioritising packages that add toolchains (rather than libraries or pure executables), as we are prioritising other issues in our backlog. LLVM itself has a myriad of components, and I suspect different users would have different needs. We are opening this issue to track this: #23834 - actual uses cases with examples will help the team gain better insights at how a potential llvm recipe may be used, whether or not it is worth packaging different LLVM components in separate recipes, and so on. Any information of this would be appreciated in that issue so that we can move forward, thanks! |
Specify library name and version: llvm/13.0.0
This is a new PR for the changes from @Manu343726 and @tcanabrava to add an LLVM recipe.