Add recipe for the LLVM core libraries #3375

newlawrence · 2020-11-01T11:45:09Z

Specify library name and version: llvm-core/11.0.1

I've read the guidelines for contributing.
I've followed the PEP8 style guides for Python code in the recipes.
I've used the latest Conan client version.
I've tried at least one configuration locally with the
conan-center hook activated.

CLAassistant · 2020-11-01T11:45:13Z

All committers have signed the CLA.

Croydon · 2020-11-01T13:58:46Z

We also have #1450 (just as a reference)

newlawrence · 2020-11-01T20:16:21Z

The package maps each static library as a conan component (i.e. llvm-core::demangle, llvm-core::support, llvm-core::interpreter...), so they can be linked in the same way as by using the llvm-config tool; all of the dependencies information is automatically extracted from the cmake configuration files. This way, other packages for the rest of the llvm suite elements can be created relying on these ones.

As for shared builds, it uses the single library distribution (check out for LLVM_BUILD_LLVM_DYLIB in llvm documentation), so conan components are not used but the regular usage instead.

recipes/llvm-core/all/conanfile.py

newlawrence · 2020-11-02T11:47:57Z

recipes/llvm-core/all/conanfile.py

+            self.run('cmake --graphviz=graph/llvm.dot .')
+            with tools.chdir('graph'):
+                dot_text = tools.load('llvm.dot').replace(' ', '')
+
+            dep_regex = re.compile(r'//(.+)->(.+)$', re.MULTILINE)
+            deps = re.findall(dep_regex, dot_text)


What do you think about this @intelligide? Instead of parsing LLVMExports.cmake, I use cmake to output the dependency graph as a graphviz's dot file which is way easier to parse.

It still uses a regular expression, but it is a simple one. The alternative would be using external tools like graphviz plus networkx that will do the parsing for you, but then you'll introduce a dependency on those tools that may not be installed along conan. I think this solution is easy and maintainable enough.

It's better (not perfect yet but better than nothing).

recipes/llvm-core/all/conandata.yml

recipes/llvm-core/all/conanfile.py

Co-authored-by: Yoann Potinet <intelligide@hotmail.fr>

As it is not part of the install target. This is a kind of problematic library, see: https://reviews.llvm.org/D74588

newlawrence · 2021-04-13T06:59:09Z

Failure in build 42 (194cf21f70c5a74064d65d6b07ec8f49adb17293):

llvm-core/11.1.0@:
CI failed to create some packages (All logs)

Definitely, the issue with stalled builds does only occur when building shared libraries on Linux. Here, in this commit, I screwed up those builds and it can be seen how the rest of the configurations pass without problems.

Why does this happen? Well, the shared library is a bundle of 70+ (depending on the configured targets) libraries, so high memory consumption during the linking step is expected (see).

@jgsogo, @danimtb do you think there's a way to pass LLVM_PARALLEL_LINK_JOBS=1and CMAKE_EXE_LINKER_FLAGS="-Wl,--reduce-memory-overheads -Wl,--hash-size=1021" to cmake for the building step from outside? I wouldn't like to hardcode those in the conanfile as they're only valid when linking using ld (and also not to impose those on the users building the package on their own).

An alternative I can think of is just change the defaults for the targets and compile just the X86 backends (most of the people wouldn't use most of the other ones) hoping that this will be enough to fix the issue.

newlawrence · 2021-04-13T20:52:07Z

Just checked locally. I forgot to mention that for the shared build I had no other choice but to enable the building of the tools which add extra burden to the linker and increments the build time dramatically. In my local tests, with 2GB of ram assigned, even with the options mentioned, the build swapped a lot and lasted several hours upon completion.

I think I've exhausted my options so far, the shared builds need more memory to get built successfully.

stale · 2021-05-14T01:09:14Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

blockspacer · 2021-05-14T09:19:42Z

In my local tests, with 2GB of ram assigned, even with the options mentioned, the build swapped a lot and lasted several hours upon completion.

Sounds like LLVM recipe needs better hardware for CI.

Is it just free tier problem?

LLVM just too heavy and requires paid CI plan.

LLVM support may be beneficial for conan. A lot of people may want to use conan with sanitized libc++, libtooling, clang++, clang-tidy, scan-build, ccc-analyzer, c++-analyzer, clang-format. All that requires LLVM recipe...

@newlawrence
Note that current recipe lacks sanitizer options, see "LLVM_USE_SANITIZER" and "COMPILER_RT_BUILD_SANITIZERS" as example from https://github.com/blockspacer/llvm_tools/blob/master/conanfile.py

Note that some options may speedup builds and allow to use less RAM:

        cmake.definitions["LLVM_INCLUDE_TESTS"]="OFF"
        cmake.definitions["LLVM_BUILD_TESTS"]="OFF"
        cmake.definitions["LLVM_BUILD_EXAMPLES"]="OFF"
        cmake.definitions["LLVM_INCLUDE_EXAMPLES"]="OFF"
        cmake.definitions["LLVM_BUILD_BENCHMARKS"]="OFF"
        cmake.definitions["LLVM_INCLUDE_BENCHMARKS"]="OFF"
        cmake.definitions["LLVM_ENABLE_DOXYGEN"]="OFF"
        cmake.definitions["LLVM_ENABLE_OCAMLDOC"]="OFF"
        cmake.definitions["LLVM_ENABLE_SPHINX"]="OFF"
        # when making a debug or asserts build speed it up by building a release tablegen
        cmake.definitions["LLVM_OPTIMIZED_TABLEGEN"]="ON"

@Croydon
Is conan (jfrog?) interested in LLVM support or that pull request will be marked stale forever?

danimtb · 2021-05-17T15:19:51Z

@newlawrence thanks for trying it locally. Build time is not a concern for us in C3I and since we are not able to provide more resources at the moment, I would like to try to make the recipe build at least. We can constrain the configurations (drop linux shared builds) or even hardcode build flags in the recipe ATM. Do you think it is possible to make it build here? I think it is better to have this PR merged and then work on improvements over this situation. Sorry for the bad news and thanks a lot for moving this forward!

newlawrence · 2021-05-21T18:14:31Z

@danimtb we could disable all shared builds for the moment and see if we finally get green, given that in Windows they are not even available, I don't think it is such a loss.

@blockspacer I think I will give a try to enable builds with the sanitizers enabled. Thank you for your comment!

recipes/llvm-core/all/conandata.yml

Co-authored-by: Yoann Potinet <intelligide@hotmail.fr>

newlawrence · 2021-05-22T05:36:32Z

recipes/llvm-core/all/conanfile.py

-        if self.settings.os == 'Windows' and self.options.shared:
-            message = 'Shared build not supported on Windows'
+        if self.options.shared:  # Shared builds disabled just due to the CI
+            message = 'Shared builds not currently supported'
            raise ConanInvalidConfiguration(message)


I didn't remove the code to build the package as a shared library but to raise an invalid configuration exception in all cases instead. In case someone wants to tweak the package to recover the lost functionality or, for us in the future, it is as easy as to remove this two lines and uncomment the following ones.

newlawrence · 2021-05-22T05:38:55Z

All green in build 45 (e5b2dee54c8da8cf3c2ab4bb43ec789fbba6135f):

llvm-core/11.1.0@:
All packages built successfully! (All logs)

Ok, so I think we're good to go @danimtb 😁. I'm waiting for your comments @Croydon, @intelligide, @uilianries.

conan-center-bot · 2021-05-22T10:58:25Z

All green in build 46 (f6a023ba475bfb5819f70ad70268cfa4a4b02cc7):

llvm-core/11.1.0@:
All packages built successfully! (All logs)

intelligide · 2021-05-22T14:41:33Z

recipes/llvm-core/all/conanfile.py

+    def source(self):
+        tools.get(**self.conan_data['sources'][self.version])
+        os.rename('llvm-{}.src'.format(self.version), self._source_subfolder)
+        self._patch_sources()


Patches should be applied in build method.

Even in this case where we are setting no_copy_source to True?

The patches applied here work for all the different build configurations unconditionally, therefore they can be applied only once. If we were to apply the patches in the build method we'd have no other choice but to copy the sources, which I wouldn't recommend given their size.

I think it is always better to copy the sources. However, given the size of the package and the artifacts, and the limitation of the build machines, an exception is necessary.

blockspacer · 2021-05-25T13:21:23Z

@danimtb @Croydon, @intelligide, @uilianries, @newlawrence

Hi.

Current recipe is awesome, thank you very much.

Note that LLVM recipe may need some changes for wider adoptation:

LLVM docs say about multi-stage builds i.e. first stage builds statically linked clang to re-build all LLVM code. That step can be disabled with env. var, see proof-of-concept below.
To use sanitizers: First, need to build cxx and cxxabi with LLVM_USE_SANITIZER in separate folder. Second, build whole LLVM code WITHOUT LLVM_USE_SANITIZER, but with COMPILER_RT_BUILD_SANITIZERS. During package step copy cxx and cxxabi with LLVM_USE_SANITIZER enabled (make sure other build stages does not override cxx and cxxabi files!) Note that COMPILER_RT_BUILD_SANITIZERS required to create libclang_rt.asan*.so libs.
Some people want to use LLVM only to compile other code, so they may want del self.info.settings.compiler based on some condition (env. var?).
Some people want to use LLVM libs. For example, LibTooling can be used to parse source code. So they may want to keepself.info.settings.compiler based on some condition (env. var?). They also may want to configure self.cpp_info.libs, see llvm_libs in proof-of-concept conanfile.py below.

Note that it may be good idea to separate recipe into multiple smaller recipes:

Recipe that allows to link with LLVM libs (see llvm_libs and self.cpp_info.libs) may be separated (Is it possible?)
Recipe that allows to use clang as compiler, sets proper paths to LLVM tools (clang-format, etc) and propagates compiler flags. Please take look at separate recipe https://github.com/blockspacer/llvm_9_installer (Is it good idea?)

Please take look at updated proof-of-concept:
https://github.com/blockspacer/conan_llvm_9

* Added recipe for the LLVM core libraries * Reuse cmake configuration * Fix shared builds for 10x and 11x * Parse dependencies from graphviz output * Beware of dummy targets * Keep only latest version for the moment * Beware of Windows line endings * Relax cmake version requirement * Add list of unsupported compiler configurations * Commit suggestion Co-authored-by: Yoann Potinet <intelligide@hotmail.fr> * Cannot discriminate Visual Studio by its minor version Visual Studio version 16.4, which is known by miscompiling LLVM, is currently being used by conan-center-index's CCI. Let's use LLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN to make that compilation pass until Visual Studio is upgraded. * Disable object targets properly * Use external dependencies properly * Enable building with libffi in Windows * Do not force libc++ in clang builds * Wrong value for a cmake flag * Building native tools were making some configurations fail * Allow Visual Studio generators now that native tools are disabled * Disable relative paths for the sources in the debug info The latest versions of clang were not able to find the standard library headers if compiling with `-stdlib=libc++` and `-no-canonical-prefixes`. * Update supported compilers * String comparisons are dangerous * Remove iconv patch comment Turns out that the hack was justified. Both, iconv and charset libraries, own a localcharset symbol. In normal circumstances this is not an issue but libLLVM gets linked with `--whole-archive` or `-all_load` and the linker complains about duplicated symbols. It is safer to remove charset from the linking step, as its symbols are not needed, than to remove the linker option. * Commit suggestion Co-authored-by: Yoann Potinet <intelligide@hotmail.fr> * More meaningful exception messages * Remove unnecessary readme * Support older Python versions It is such a pity; pathlib is way elegant than os.path. * Empty commit to trigger the CI * Remove unnecessary status message Co-authored-by: Michael Keck <git@cr0ydon.com> * Less strict cmake minimum required version Co-authored-by: Michael Keck <git@cr0ydon.com> * LLVM version bumped to 11.0.1 * LLVM version bumped to 11.1.0 * Deactivate fPIC if shared * Fix fPIC flag for shared builds * Install LLVMTableGenGlobalISel manually As it is not part of the install target. This is a kind of problematic library, see: https://reviews.llvm.org/D74588 * Add option use_sanitizer * Disable all shared builds * Dedent conandata.yml Co-authored-by: Yoann Potinet <intelligide@hotmail.fr> Co-authored-by: Yoann Potinet <intelligide@hotmail.fr> Co-authored-by: Michael Keck <git@cr0ydon.com>

ericLemanissier · 2021-10-15T14:38:41Z

@newlawrence and all, there is a glitch in the test recipe: the variable components in https://github.com/conan-io/conan-center-index/blob/master/recipes/llvm-core/all/test_package/conanfile.py#L27 is never defined. On the other hand, the variable targets is used but never defined. What was intended ?

Added recipe for the LLVM core libraries

6c6d87b

This comment has been minimized.

Sign in to view

conan-center-bot added the No Beta user label Nov 1, 2020

newlawrence mentioned this pull request Nov 1, 2020

[request] libclang/9.0.0 #683

Open

Reuse cmake configuration

8e4e4d0

This comment has been minimized.

Sign in to view

Fix shared builds for 10x and 11x

daa072b

This comment has been minimized.

Sign in to view

intelligide reviewed Nov 1, 2020

View reviewed changes

recipes/llvm-core/all/conanfile.py Outdated Show resolved Hide resolved

Parse dependencies from graphviz output

b735998

This comment has been minimized.

Sign in to view

newlawrence commented Nov 2, 2020

View reviewed changes

Beware of dummy targets

edf9abc

This comment has been minimized.

Sign in to view

intelligide reviewed Nov 2, 2020

View reviewed changes

recipes/llvm-core/all/conandata.yml Outdated Show resolved Hide resolved

Keep only latest version for the moment

aebb740

This comment has been minimized.

Sign in to view

newlawrence force-pushed the llvm-core branch from df921b9 to aebb740 Compare November 2, 2020 19:40

This comment has been minimized.

Sign in to view

Beware of Windows line endings

6fdd6a6

conan-center-bot removed the No Beta user label Nov 3, 2020

This comment has been minimized.

Sign in to view

conan-center-bot added the Failed label Nov 3, 2020

newlawrence added 2 commits November 3, 2020 15:11

Relax cmake version requirement

00c52c8

Add list of unsupported compiler configurations

41562aa

intelligide suggested changes Nov 3, 2020

View reviewed changes

recipes/llvm-core/all/conanfile.py Outdated Show resolved Hide resolved

Commit suggestion

d876521

Co-authored-by: Yoann Potinet <intelligide@hotmail.fr>

newlawrence added 2 commits April 12, 2021 19:17

Fix fPIC flag for shared builds

e021639

Install LLVMTableGenGlobalISel manually

d99091d

As it is not part of the install target. This is a kind of problematic library, see: https://reviews.llvm.org/D74588

stale bot added the stale label May 14, 2021

stale bot removed the stale label May 14, 2021

newlawrence added 2 commits May 21, 2021 20:36

Add option use_sanitizer

c4ae50c

Disable all shared builds

e5b2dee

This comment has been minimized.

Sign in to view

conan-center-bot removed the Failed label May 22, 2021

intelligide suggested changes May 22, 2021

View reviewed changes

recipes/llvm-core/all/conandata.yml Outdated Show resolved Hide resolved

Dedent conandata.yml

f6a023b

Co-authored-by: Yoann Potinet <intelligide@hotmail.fr>

newlawrence commented May 22, 2021

View reviewed changes

intelligide reviewed May 22, 2021

View reviewed changes

intelligide approved these changes May 23, 2021

View reviewed changes

github-actions bot mentioned this pull request May 23, 2021

Pull Requests Ready to be Reviewed prince-chrismc/conan-center-index-pending-review#1

Open

danimtb approved these changes May 24, 2021

View reviewed changes

SSE4 approved these changes May 24, 2021

View reviewed changes

conan-center-bot merged commit bde927b into conan-io:master May 24, 2021

Croydon mentioned this pull request Jul 30, 2021

[question] Is there clang package similar to mingw-w64/8.1 #6548

Open

samuel-emrys mentioned this pull request Aug 11, 2021

[request] llvm-flang/12.0.1 #6816

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add recipe for the LLVM core libraries #3375

Add recipe for the LLVM core libraries #3375

newlawrence commented Nov 1, 2020 •

edited

Loading

CLAassistant commented Nov 1, 2020 •

edited

Loading

This comment has been minimized.

Croydon commented Nov 1, 2020

This comment has been minimized.

newlawrence commented Nov 1, 2020

This comment has been minimized.

This comment has been minimized.

newlawrence Nov 2, 2020

intelligide Nov 2, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

newlawrence commented Apr 13, 2021 •

edited

Loading

newlawrence commented Apr 13, 2021 •

edited

Loading

stale bot commented May 14, 2021

blockspacer commented May 14, 2021 •

edited

Loading

danimtb commented May 17, 2021

newlawrence commented May 21, 2021 •

edited

Loading

This comment has been minimized.

newlawrence May 22, 2021

newlawrence commented May 22, 2021

conan-center-bot commented May 22, 2021

intelligide May 22, 2021

newlawrence May 23, 2021

intelligide May 23, 2021

blockspacer commented May 25, 2021 •

edited

Loading

ericLemanissier commented Oct 15, 2021

Add recipe for the LLVM core libraries #3375

Add recipe for the LLVM core libraries #3375

Conversation

newlawrence commented Nov 1, 2020 • edited Loading

CLAassistant commented Nov 1, 2020 • edited Loading

This comment has been minimized.

Croydon commented Nov 1, 2020

This comment has been minimized.

newlawrence commented Nov 1, 2020

This comment has been minimized.

This comment has been minimized.

newlawrence Nov 2, 2020

Choose a reason for hiding this comment

intelligide Nov 2, 2020

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

newlawrence commented Apr 13, 2021 • edited Loading

newlawrence commented Apr 13, 2021 • edited Loading

stale bot commented May 14, 2021

blockspacer commented May 14, 2021 • edited Loading

danimtb commented May 17, 2021

newlawrence commented May 21, 2021 • edited Loading

This comment has been minimized.

newlawrence May 22, 2021

Choose a reason for hiding this comment

newlawrence commented May 22, 2021

conan-center-bot commented May 22, 2021

intelligide May 22, 2021

Choose a reason for hiding this comment

newlawrence May 23, 2021

Choose a reason for hiding this comment

intelligide May 23, 2021

Choose a reason for hiding this comment

blockspacer commented May 25, 2021 • edited Loading

ericLemanissier commented Oct 15, 2021

newlawrence commented Nov 1, 2020 •

edited

Loading

CLAassistant commented Nov 1, 2020 •

edited

Loading

newlawrence commented Apr 13, 2021 •

edited

Loading

newlawrence commented Apr 13, 2021 •

edited

Loading

blockspacer commented May 14, 2021 •

edited

Loading

newlawrence commented May 21, 2021 •

edited

Loading

blockspacer commented May 25, 2021 •

edited

Loading