-
Notifications
You must be signed in to change notification settings - Fork 6.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[vcpkg baseline] Re-fix lapack-reference to find blas on windows-static #19608
Conversation
cc @Neumann-A @cenit |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!
After committing all other changes, the version database must be updated
git add -u && git commit
git checkout d679a1e0befa9c9e36fcda169c94f6daba2224b7 -- versions
./vcpkg x-add-version --all
Diff
diff --git a/versions/b-/blas.json b/versions/b-/blas.json
index 7fb0192..6d991bf 100644
--- a/versions/b-/blas.json
+++ b/versions/b-/blas.json
@@ -1,5 +1,10 @@
{
"versions": [
+ {
+ "git-tree": "2c785ddee8825820b71ad4f8f3be588c8158906e",
+ "version-semver": "3.10.0",
+ "port-version": 0
+ },
{
"git-tree": "2877c1693c63195d4edacfb42156c9d8874ad046",
"version-string": "1",
diff --git a/versions/baseline.json b/versions/baseline.json
index 2855173..0ac5e18 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -421,7 +421,7 @@
"port-version": 0
},
"blas": {
- "baseline": "1",
+ "baseline": "3.10.0",
"port-version": 0
},
"blaze": {
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!
After committing all other changes, the version database must be updated
git add -u && git commit
git checkout d679a1e0befa9c9e36fcda169c94f6daba2224b7 -- versions
./vcpkg x-add-version --all
Diff
diff --git a/versions/b-/blas.json b/versions/b-/blas.json
index 7fb0192..801ba36 100644
--- a/versions/b-/blas.json
+++ b/versions/b-/blas.json
@@ -1,5 +1,10 @@
{
"versions": [
+ {
+ "git-tree": "8876f20d092c072b4f7a80db2e5a877ab000dda9",
+ "version-semver": "3.10.0",
+ "port-version": 0
+ },
{
"git-tree": "2877c1693c63195d4edacfb42156c9d8874ad046",
"version-string": "1",
diff --git a/versions/baseline.json b/versions/baseline.json
index 2855173..0ac5e18 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -421,7 +421,7 @@
"port-version": 0
},
"blas": {
- "baseline": "1",
+ "baseline": "3.10.0",
"port-version": 0
},
"blaze": {
?!?!? who removed the also what you are adding here is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!
After committing all other changes, the version database must be updated
git add -u && git commit
git checkout d679a1e0befa9c9e36fcda169c94f6daba2224b7 -- versions
./vcpkg x-add-version --all
Diff
diff --git a/versions/b-/blas.json b/versions/b-/blas.json
index 7fb0192..d1c61b5 100644
--- a/versions/b-/blas.json
+++ b/versions/b-/blas.json
@@ -1,5 +1,10 @@
{
"versions": [
+ {
+ "git-tree": "75b6ee2b0d4236f48b310cc69de037bfb9b34cc6",
+ "version-semver": "3.10.0",
+ "port-version": 0
+ },
{
"git-tree": "2877c1693c63195d4edacfb42156c9d8874ad046",
"version-string": "1",
diff --git a/versions/baseline.json b/versions/baseline.json
index 2855173..0ac5e18 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -421,7 +421,7 @@
"port-version": 0
},
"blas": {
- "baseline": "1",
+ "baseline": "3.10.0",
"port-version": 0
},
"blaze": {
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!
After committing all other changes, the version database must be updated
git add -u && git commit
git checkout d679a1e0befa9c9e36fcda169c94f6daba2224b7 -- versions
./vcpkg x-add-version --all
Diff
diff --git a/versions/b-/blas.json b/versions/b-/blas.json
index 7fb0192..a283f48 100644
--- a/versions/b-/blas.json
+++ b/versions/b-/blas.json
@@ -1,5 +1,10 @@
{
"versions": [
+ {
+ "git-tree": "a297c47bd805c00b2308f027d288f3d9544aae33",
+ "version-semver": "3.10.0",
+ "port-version": 0
+ },
{
"git-tree": "2877c1693c63195d4edacfb42156c9d8874ad046",
"version-string": "1",
diff --git a/versions/baseline.json b/versions/baseline.json
index 2855173..0ac5e18 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -421,7 +421,7 @@
"port-version": 0
},
"blas": {
- "baseline": "1",
+ "baseline": "3.10.0",
"port-version": 0
},
"blaze": {
@Neumann-A I want to clarify the relationship between blas and lapack's reference and non-reference, can you elaborate on it? |
@JackBoosY: Please keep blas a metaport. What you are adding here is blas-reference. There a many ways vcpkg could provide blas/lapack but the current idea is to have the blas port just check that blas can be found and depend on vcpkg defaults blas implementation (openblas). Using an overlay this could be changed to e.g. using mkl instead. |
@Neumann-A What about the dependencies between them? This PR start with a ci regression when building colmap:x64-windows-static:
Colmap's dependency chain: The fact is: lapack-reference generates blas and conflicts with openblas. |
@JackBoosY That is just someone linking openblas but not declaring the dependency correctly:
vs
so some target has a dependency on openblas but does not have it in its build-depends which is why ci is not restoring openblas. Find the target which links openblas and make the dependency on blas explicit. |
furthermore this is related to #15571. vcpkg needs to force the internally used blas and lapack version to avoid finding spurious other stuff. |
@Neumann-A But the libblas generated by lapack-reference contains the same symbols as the openblas generated by openblas. And according to the upstream issue OpenMathLib/OpenBLAS#203, the blas used in lapack-reference is not openblas. |
i think that issue talks about the opposite: openblas contains lapack-reference the two algebra libraries are really tangled together, the best option is disabling lapack in openblas and blas in lapack-reference, and then explicitly depend on both. as @Neumann-A is saying, colmap error should be just a matter of fact of a port depending (wrongly) on openblas(!static) |
After comparing the blas source code in lapack-reference with the source code in blas, I found that they are almost the same (except some comments). |
The problem is lapack-reference cannot link against openblas from vcpkg if |
Yeah because it is the reference implementation in both cases ;) lapack-reference could probably do a FETCH_CONTENT of the blas stuff |
That because: vcpkg/ports/lapack-reference/CONTROL Lines 15 to 17 in d679a1e
According to my poor experience with blas and lapack, I can't distinguish the relationship between them at all. |
@JackBoosY: Just think of the CMakeLists.txt of both ports as superbuilds. openblas comes with lapack vcpkg only wants to build one lib of those in the ports: due to the gfortran constrains lapack needs to build its internal blas in x64-windows-static. That is why vcpkg needs to force the correct blas library to be used via a wrapper. |
But why create a link to the pkgconfig file of openblas? lapack-reference also provides blas.pc. |
it does. since lapack is providing blas. in this case.
that is probably a bug. |
This will be fixed in #19628. |
This baseline regression blocked 5-6 PRs, @ras0219-msft please make some suggestions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Neumann-A Thanks for the summary (#19608 (comment))
openblas comes with lapack
lapack comes with blas, cblas, lapackevcpkg only wants to build one lib of those in the ports:
so lapack-reference should only build lapack
openblas should only build openblas
To put more words on the subject for other readers, here are our objectives:
- If a consumer depends on
blas
, thenfind_package(BLAS)
will resolve (+pkgconfig) - If a consumer depends on
lapack
, thenfind_package(LAPACK)
will resolve (+pkgconfig) - There will be only one provider of BLAS or LAPACK symbols in a given build tree
- (ideally) Official vcpkg triplets will provide a consistent default choice for lapack backend. I'm not yet certain whether the gfortran issue makes this intractable for static-crt triplets.
- Consumers have a choice of backends to provide blas and lapack by replacing
blas
orlapack
and their choice should be respected by the tree (including builds failing if appropriate)
Then, we need to determine what the space of backends looks like. Given @Neumann-A's summary, it looks to me like there are several possible configurations:
- lapack-reference[noblas] & openblas[nolapack]
- openblas[blas,lapack]
- mpl & lapack-reference[noblas] (?)
- lapack-reference[noblas] & blas-reference[noblas]
- Built in MacOS option for BLAS?
Open questions:
- How does
cblas
fit into this? Can we have it be implied byblas
or is it useful for it to be a unique metaport? - What restrictions do the major packages have w.r.t. incompatibilities (such as the gfortran issue)?
- What other major packages are relevant to this space?
- How do we test this? This is another "forks the entire graph" situation like openssl/libressl/boringssl. Perhaps we need to just pick the default subset that we will test the world against and relegate other configurations to community registries that can provide more domain-specific testing?
@@ -0,0 +1,1191 @@ | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should not be checking in a thousand lines of CMake upstream sources. Either:
- (bad option) This should be downloaded from GitHub (https://raw.githubusercontent.com/Kitware/CMake/v3.21.1/Modules/FindBLAS.cmake)
- (good option) The point of providing this overlay is to force the vcpkg BLAS version. Therefore, there is no need for hundreds of these lines -- we just need to define the outputs listed in https://cmake.org/cmake/help/latest/module/FindBLAS.html to point at the copy of BLAS we just built.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we just need to define the outputs listed in
Please also add the check that the libraries are actual usable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason why I manually put FindBLAS.cmake
into the port directory is that the version provided by cmake is not suitable for the port blas-reference
.
In windows, if BUILD_X64
is turned on, blas-reference
will generate libblas64.lib
, and when searching, only blas
will be found.
And FindBLAS.cmake
whcih provided by cmake doesn't provide any marco to fix that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the FindBLAS
from CMake is not usable and we need to replace it in this case, that's fine. However, we do not need a thousand lines of CMake to do that -- a much smaller file will do fine.
about the configurations: we have also clapack already in tree, it was our default lapack port before lapack-reference was accepted also, macOS Accelerate Framework contains BLAS but also LAPACK. Final note: all implementations must have compatible ABI and they have, at least as far as i know (not headers, it is common to just call libraries knowing their interface) |
set(BLA_VENDOR "Generic") | ||
_find_package(${ARGS}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
set(BLA_VENDOR "Generic") | |
_find_package(${ARGS}) | |
if (WIN32) | |
set(CMAKE_FIND_LIBRARY_PREFIXES_BAK ${CMAKE_FIND_LIBRARY_PREFIXES}) | |
set(CMAKE_FIND_LIBRARY_PREFIXES lib) | |
endif() | |
set(BLA_VENDOR "Generic") | |
_find_package(${ARGS}) | |
if (WIN32) | |
set(CMAKE_FIND_LIBRARY_PREFIXES ${CMAKE_FIND_LIBRARY_PREFIXES}) | |
endif() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In FindBLAS.cmake
line 1156:
# Generic BLAS library?
if(BLA_VENDOR STREQUAL "Generic" OR
BLA_VENDOR STREQUAL "NVHPC" OR
BLA_VENDOR STREQUAL "All")
if(NOT BLAS_LIBRARIES)
check_blas_libraries(
BLAS_LIBRARIES
BLAS
sgemm
""
"blas"
""
""
""
)
endif()
endif()
line 275:
function(CHECK_BLAS_LIBRARIES LIBRARIES _prefix _name _flags _list _deps _addlibdir _subdirs)
...
find_library(${_prefix}_${_lib_var}_LIBRARY
NAMES ${_library}
NAMES_PER_DIR
PATHS ${_extaddlibdir}
PATH_SUFFIXES ${_subdirs}
)
Unless we forced the 64bit openblas-reference library name to blas.lib
or libblas.lib
, we can't search it using FindBLAS.cmake
provided by cmake.
the problem lies mainly with the Fortran library. Because that is linked with the mingw stuff and requires ld as a linker due to internal symbol stuff. If We could link against a naively build windows Fortran library the problem goes away (#19413 does this by switching to ifort). One problem still is that the compiler Fortran libraries have not set the /APPCONTAINER bit so strictly speaking Fortran on UWP is currently not supported.
that would be the ideal situation but it is already broken due to intel-mkl also providing blas/lapack symbols. It is more important to force the lookup of
cblas is just a C wrapper to the legacy blas library. As such it can be installed using any blas implementation. It will wrap the fortran symbols using a
as said the incompatibility is mainly due to the linkage problems with libgfortran.lib in static builds. If we ca get a native windows variant there is only the issue with UWP an the missing appcontainer bit. There is no ABI implication.
Just look into CMakes
Give a reasonable default; Let the rest be handled by the community. There could be some basic testing of a minimal port tree which uses LAPACK/BLAS to check if it at least works. But that is always the case for features in ports. vcpkg never checks all features in CI. |
Hmm, maybe splitting the Edit: I'm not sure how to precisely set up the conditions to test this.
Isn't this handled by simply overlaying both
Agreed, this seems critical. If we think "cross" configurations like
Agreed, though I think that will still require the basic structure I've indicated above.
But can we assume every blas implementation will install it? Do we need to have an explicit port that just creates the wrapper? |
@@ -0,0 +1,5 @@ | |||
Use this package via the module FindBLAS that comes with CMake. To use in your CMakeLists.txt: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not true because we are checking-in our own modified version
The solution accepted, but the
|
In favor of #21479. |
The origin issue is:
openblas comes with lapack, lapack comes with blas, cblas, lapacke.
vcpkg only wants to build one lib of those in the ports, so lapack-reference should only build lapack, openblas should only build openblas. Due to the gfortran constrains lapack needs to build its internal blas in x64-windows-static.
The problem: FindBlas.cmake picks up a spurious openblas installation of vcpkg in one port since it was installed while another port was looking for it but that port did not declare the dependency on openblas -> error in colmap since openblas is not restored by CI.
This PR will
blas-reference
and disablelapack-reference
to build internalblas
.FindBLAS.cmake
provided by cmake and modify some codes to adapt to Windows or x64 build.vcpkg-cmake-wrapper
toblas-reference
to avoid finding openblas.Related: #19628
Thanks @Neumann-A!