Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add symbolprefix=scipy #125

Merged
merged 22 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
9d85bff
add 'symbolprefix=scipy'
mattip Oct 29, 2023
517d31e
update multibuild
mattip Oct 29, 2023
830a682
add SYMBOLPREFIX to 'make install'
mattip Oct 29, 2023
0225ae3
allow prefixed name in more places, do not use patchelf to rename to …
mattip Oct 29, 2023
5758b60
use prefix in more places
mattip Oct 29, 2023
d2c52da
use quotes on windows, fix lib name on posix
mattip Oct 29, 2023
bee5763
add more 'scipy_' to the python code, skip dynamic test on windows fo…
mattip Oct 31, 2023
400074e
debug windows, clean up posix (darwin), always build objconv
mattip Oct 31, 2023
a217777
windows: set DLL_BASENAME properly, macos: simplify and fix
mattip Oct 31, 2023
7fde304
typos
mattip Oct 31, 2023
4b0b196
unconditionally do 'make exports' on windows, add debug check
mattip Nov 1, 2023
ff6cfd4
simplify, for now disable SYMBOLPREFIX to make sure tests pass on win…
mattip Nov 2, 2023
c62aaba
allow for non-existing exp file
mattip Nov 2, 2023
685ece5
restore 'scipy_' prefix, edit fortran test files
mattip Nov 2, 2023
49f62b1
more debugging for 32-bit windows
mattip Nov 2, 2023
1d38514
more debugging
mattip Nov 2, 2023
9014d04
debug differently
mattip Nov 2, 2023
ade139e
add leading underscores for 32-bit symbol names
mattip Nov 3, 2023
027967f
update wheel version to match openblas-lib tarball name
mattip Nov 3, 2023
65fd7a3
debug why 32-bit build does not do objcopy properly
mattip Nov 4, 2023
4e7727f
add '-DBLAS_SYMBOL_PREFIX' to the wheel pkg-config cflags
mattip Nov 7, 2023
3053c45
add OPENBLAS_ILP64_NAMING_SCHEME so meson does not have to
mattip Nov 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 8 additions & 10 deletions .github/workflows/posix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
INTERFACE64: '1'
env:
REPO_DIR: OpenBLAS
OPENBLAS_COMMIT: "c2f4bdb"
OPENBLAS_COMMIT: "9d425a5f"
NIGHTLY: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}
MACOSX_DEPLOYMENT_TARGET: 10.9
MB_PYTHON_VERSION: ${{ matrix.python-version }}
Expand All @@ -68,17 +68,15 @@ jobs:
python-version: 3.7
- name: Set extra env
run: |
if [ "macos-11" == "${{ matrix.os }}" ]; then
echo "TRAVIS_OS_NAME=osx" >> $GITHUB_ENV;
echo "LDFLAGS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib" >> $GITHUB_ENV;
echo "LIBRARY_PATH=-L/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib" >> $GITHUB_ENV;
else
echo "TRAVIS_OS_NAME=${{ matrix.os }}" >> $GITHUB_ENV;
fi
# if [ "macos-11" == "${{ matrix.os }}" ]; then
# Use xcrun --sdk macosx --show-sdk-path instead of hardcoding the path
# echo "LDFLAGS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib" >> $GITHUB_ENV;
# echo "LIBRARY_PATH=-L/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib" >> $GITHUB_ENV;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll note that if this is needed again in the future or changes, there's a better way than hardcoding it. The path can be obtained from xcrun --sdk macosx --show-sdk-path. I just managed to get rid of this hardcoding in numpy CI.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment in the commit to add the BLAS_SYMBOL_PREFIX definition to the pkgconfig file

# else
# fi
echo "DOCKER_TEST_IMAGE=$(echo multibuild/xenial_${{ matrix.PLAT}})" >> $GITHUB_ENV;
- name: Print some Environment variable
run: |
echo "TRAVIS_OS_NAME: ${TRAVIS_OS_NAME}"
echo "PLAT: ${PLAT}"
echo "DOCKER_TEST_IMAGE: ${DOCKER_TEST_IMAGE}"
- name: Install VirtualEnv
Expand Down Expand Up @@ -115,7 +113,7 @@ jobs:
version=$(cd OpenBLAS && git describe --tags --abbrev=8 | sed -e "s/^v\(.*\)-g.*/\1/" | sed -e "s/-/./g")
sed -e "s/^version = .*/version = \"${version}\"/" -i.bak pyproject.toml
fi
if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
if [ "macos-11" == "${{ matrix.os }}" ]; then
source tools/build_wheel.sh
else
libc=${MB_ML_LIBC:-manylinux}
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:
branches: [ main ]

env:
OPENBLAS_COMMIT: "c2f4bdb"
OPENBLAS_COMMIT: "9d425a5f"
OPENBLAS_ROOT: "c:\\opt"
# Preserve working directory for calls into bash
# Without this, invoking bash will cd to the home directory
Expand All @@ -32,7 +32,7 @@ jobs:
- name: install-rtools
run: |
# rtools 42+ does not support 32 bits builds.
choco install -y rtools --noprogress --force --version=4.0.0.20220206
choco install -y rtools --no-progress --force --version=4.0.0.20220206

- name: Set env variables
run: |
Expand Down Expand Up @@ -94,7 +94,7 @@ jobs:
fi
mv local/scipy_openblas64/bin/*.dll local/scipy_openblas64/lib
rm local/scipy_openblas64/lib/*.a
rm local/scipy_openblas64/lib/*.exp
rm -f local/scipy_openblas64/lib/*.exp # may not exist?
rm local/scipy_openblas64/lib/*.def
rm -rf local/scipy_openblas64/lib/pkgconfig
if [[ -d local/scipy_openblas64/64 ]]; then
Expand All @@ -109,7 +109,7 @@ jobs:
# of a pyproject.toml project
sed -e "s/openblas64/openblas32/" -i pyproject.toml
sed -e "s/openblas_get_config64_/openblas_get_config/" -i local/scipy_openblas32/__init__.py
sed -e "s/cflags_suffix64 =.*/cflags_suffix64 = ''/" -i local/scipy_openblas32/__init__.py
sed -e "s/cflags =.*/cflags = '-DBLAS_SYMBOL_PREFIX=scipy_'/" -i local/scipy_openblas32/__init__.py
sed -e "s/openblas64/openblas32/" -i local/scipy_openblas32/__init__.py
sed -e "s/openblas64/openblas32/" -i local/scipy_openblas32/__main__.py
fi
Expand Down
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ env:
global:
# The archive that gets built has name from ``git describe`` on this
# commit.
- OPENBLAS_COMMIT="c2f4bdb"
- OPENBLAS_COMMIT="9d425a5f"
- REPO_DIR=OpenBLAS
# Following generated with:
# travis encrypt -r MacPython/openblas-libs OPENBLAS_LIBS_STAGING_UPLOAD_TOKEN=<secret token value>
Expand Down
2 changes: 1 addition & 1 deletion OpenBLAS
14 changes: 8 additions & 6 deletions local/scipy_openblas64/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ def get_library():
libs = [x for x in os.listdir(get_lib_dir()) if x.endswith(".lib")]
return os.path.splitext(libs[0])[0]
else:
return "openblas_python"
libs = [x for x in os.listdir(get_lib_dir()) if x.startswith("libscipy_openblas")]
# remove the leading lib from libscipy_openblas*
return os.path.splitext(libs[0])[0][3:]

def get_pkg_config():
"""Return a multi-line string that, when saved to a file, can be used with
Expand All @@ -59,9 +61,9 @@ def get_pkg_config():
extralib = "-defaultlib:advapi32 -lgfortran -lquadmath"
libs_flags = f"-L${{libdir}} -l{get_library()}"
else:
extralib = "-lm -lpthread -lgfortran -lquadmath -L${libdir} -lopenblas_python"
extralib = "-lm -lpthread -lgfortran -lquadmath -L${libdir} -l{get_library()}"
libs_flags = ""
cflags_suffix64 = "-DBLAS_SYMBOL_SUFFIX=64_ -DHAVE_BLAS_ILP64"
cflags = "-DBLAS_SYMBOL_PREFIX=scipy_ -DBLAS_SYMBOL_SUFFIX=64_ -DHAVE_BLAS_ILP64 -DOPENBLAS_ILP64_NAMING_SCHEME"
return dedent(f"""\
libdir={get_lib_dir()}
includedir={get_include_dir()}
Expand All @@ -74,7 +76,7 @@ def get_pkg_config():
URL: https://github.com/xianyi/OpenBLAS
Libs: {libs_flags}
Libs.private: ${{extralib}}
Cflags: -I${{includedir}} {cflags_suffix64}
Cflags: -I${{includedir}} {cflags}
""")


Expand Down Expand Up @@ -109,10 +111,10 @@ def get_openblas_config():
libnames = [x for x in os.listdir(lib_dir) if x.endswith(".dll")]
else:
# Get openblas*
libnames = [x for x in os.listdir(lib_dir) if x.startswith("libopenblas")]
libnames = [x for x in os.listdir(lib_dir) if x.startswith("libscipy")]

dll = ctypes.CDLL(os.path.join(lib_dir, libnames[0]), ctypes.RTLD_GLOBAL)
openblas_config = dll.openblas_get_config64_
openblas_config = dll.scipy_openblas_get_config64_
openblas_config.restype = ctypes.c_char_p
bytes = openblas_config()
return bytes.decode("utf8")
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "scipy_openblas64"
version = "0.3.23.293.2"
version = "0.3.24.95.0"
requires-python = ">=3.7"
description = "Provides OpenBLAS for python packaging"
readme = "README.md"
Expand Down
10 changes: 5 additions & 5 deletions test.f90
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
!testdpotr_test_gh_2691.f90
!testdpotr_test_gh_2691.f90 with scipy_ prefix
subroutine garbage(okflag)
implicit none
integer, intent(out) :: okflag
Expand All @@ -23,7 +23,7 @@ subroutine garbage(okflag)

a2 = a

call dpotrf('L', 3, a, 3, info)
call scipy_dpotrf('L', 3, a, 3, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DPOTRF failed'
Expand All @@ -39,20 +39,20 @@ subroutine garbage(okflag)
a(i,j) = 0
end do
end do
call dpotri('L', 3, a, 3, info)
call scipy_dpotri('L', 3, a, 3, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DPOTRI failed'
return
end if

call dgetrf(3, 3, a2, 3, ipiv, info)
call scipy_dgetrf(3, 3, a2, 3, ipiv, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DGETRF failed'
return
end if
call dgetri(3, a2, 3, ipiv, work, lwork, info)
call scipy_dgetri(3, a2, 3, ipiv, work, lwork, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DGETRI failed'
Expand Down
10 changes: 5 additions & 5 deletions test64_.f90
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
!testdpotr_test_gh_2691.f90 with _64 suffix
!testdpotr_test_gh_2691.f90 with _64 suffix and scipy_ prefix
subroutine garbage(okflag)
implicit none
integer, intent(out) :: okflag
Expand All @@ -23,7 +23,7 @@ subroutine garbage(okflag)

a2 = a

call dpotrf_64('L', 3, a, 3, info)
call scipy_dpotrf_64('L', 3, a, 3, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DPOTRF failed'
Expand All @@ -39,20 +39,20 @@ subroutine garbage(okflag)
a(i,j) = 0
end do
end do
call dpotri_64('L', 3, a, 3, info)
call scipy_dpotri_64('L', 3, a, 3, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DPOTRI failed'
return
end if

call dgetrf_64(3, 3, a2, 3, ipiv, info)
call scipy_dgetrf_64(3, 3, a2, 3, ipiv, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DGETRF failed'
return
end if
call dgetri_64(3, a2, 3, ipiv, work, lwork, info)
call scipy_dgetri_64(3, a2, 3, ipiv, work, lwork, info)
if (info.ne.0) then
okflag = 0
write(*,*) 'DGETRI failed'
Expand Down
5 changes: 4 additions & 1 deletion tools/build_gfortran.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Build gfortran binary against OpenBLAS

set -e
set -ex

rm -rf for_test
mkdir for_test
Expand All @@ -13,6 +13,9 @@ static_libname=$(find $OBP/lib -maxdepth 1 -type f -name '*.a' \! -name '*.dll.a
dynamic_libname=$(find $OBP/lib -maxdepth 1 -type f -name '*.dll.a' | tail -1)
dll_name=$(echo $dynamic_libname | sed 's#/lib/#/bin/#' | sed 's/.a$//')

grep dpotrf $OBP/include/*
nm $static_libname | grep dpotrf

cp $dll_name .

if [ "$INTERFACE64" == "1" ]; then
Expand Down
48 changes: 29 additions & 19 deletions tools/build_openblas.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
# realpath, cygpath, zip, gcc, make, ar, dlltool
# usually as part of an msys installation.

set -xe

# Convert to Unix-style path
openblas_root="$(cygpath ${1:-${OPENBLAS_ROOT:-c:\\opt}})"
build_bits="${2:-${BUILD_BITS:-64}}"
Expand Down Expand Up @@ -75,12 +77,13 @@ fflags="$fextra $cflags -frecursive -ffpe-summary=invalid,zero"
# Set suffixed-ILP64 flags
if [ "$if_bits" == "64" ]; then
SYMBOLSUFFIX="64_"
interface64_flags="INTERFACE64=1 SYMBOLSUFFIX=${SYMBOLSUFFIX}"
interface_flags="INTERFACE64=1 SYMBOLSUFFIX=${SYMBOLSUFFIX}"
# We override FCOMMON_OPT, so we need to set default integer manually
fflags="$fflags -fdefault-integer-8"
else
interface64_flags=""
interface_flags=""
fi
interface_flags="$interface_flags SYMBOLPREFIX=scipy_"

# Build name for output library from gcc version and OpenBLAS commit.
GCC_TAG="gcc_$(gcc -dumpversion | tr .- _)"
Expand All @@ -95,20 +98,28 @@ make BINARY=$build_bits DYNAMIC_ARCH=1 USE_THREAD=1 USE_OPENMP=0 \
COMMON_OPT="$cflags" \
FCOMMON_OPT="$fflags" \
MAX_STACK_ALLOC=2048 \
$interface64_flags
make PREFIX=$openblas_root/$build_bits $interface64_flags install
DLL_BASENAME=libopenblas${SYMBOLSUFFIX}_${LIBNAMESUFFIX}
if [ "$if_bits" == "64" ]; then
# OpenBLAS does not build a symbol-suffixed static library on Windows:
# do it ourselves
set -x # echo commands
static_libname=$(find . -maxdepth 1 -type f -name '*.a' \! -name '*.dll.a' | tail -1)
make -C exports $interface64_flags objcopy.def
objcopy --redefine-syms exports/objcopy.def "${static_libname}" "${static_libname}.renamed"
cp -f "${static_libname}.renamed" "$openblas_root/$build_bits/lib/${static_libname}"
cp -f "${static_libname}.renamed" "$openblas_root/$build_bits/lib/${DLL_BASENAME}.a"
set +x
$interface_flags
make PREFIX=$openblas_root/$build_bits $interface_flags install
DLL_BASENAME=libscipy_openblas${SYMBOLSUFFIX}_${LIBNAMESUFFIX}

# OpenBLAS does not build a symbol-suffixed static library on Windows:
# do it ourselves. On 32-bit builds, the objcopy.def names need a '_' prefix
static_libname=$(find . -maxdepth 1 -type f -name '*.a' \! -name '*.dll.a' | tail -1)
make -C exports $interface_flags objcopy.def

if [ "$build_bits" == "32" ]; then
sed -i "s/^/_/" exports/objcopy.def
sed -i "s/scipy_/_scipy_/" exports/objcopy.def
else
echo not updating objcopy,def, buildbits=$build_bits
fi
echo "\nshow some of objcopy.def"
head -10 exports/objcopy.def
echo
objcopy --redefine-syms exports/objcopy.def "${static_libname}" "${static_libname}.renamed"
cp -f "${static_libname}.renamed" "$openblas_root/$build_bits/lib/${static_libname}"
cp -f "${static_libname}.renamed" "$openblas_root/$build_bits/lib/${DLL_BASENAME}.a"

cd $openblas_root
# Copy library link file for custom name
cd $build_bits/lib
Expand All @@ -124,10 +135,6 @@ dlltool --input-def ${DLL_BASENAME}.def \
--output-lib ${DLL_BASENAME}.lib
# Replace the DLL name with the generated name.
sed -i "s/ -lopenblas.*$/ -l${DLL_BASENAME:3}/g" pkgconfig/openblas*.pc
echo After sed -i, pkgconfig/openblas*.pc is
echo ------------
cat pkgconfig/openblas*.pc
echo ------------
cd ../..
# Build template site.cfg for using this build
cat > ${build_bits}/site.cfg.template << EOF
Expand All @@ -136,6 +143,9 @@ libraries = $DLL_BASENAME
library_dirs = {openblas_root}\\${build_bits}\\lib
include_dirs = {openblas_root}\\${build_bits}\\include
EOF

ls $openblas_root/$build_bits/lib

zip_name="openblas${SYMBOLSUFFIX}-${OPENBLAS_VERSION}-${plat_tag}-${GCC_TAG}.zip"
zip -r $zip_name $build_bits
cp $zip_name ${builds_dir}
18 changes: 8 additions & 10 deletions tools/build_steps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,8 @@ function before_build {
# Deployment target set by gfortran_utils
echo "Deployment target $MACOSX_DEPLOYMENT_TARGET"

if [ "$INTERFACE64" = "1" ]; then
# Build the objconv tool
(cd ${ROOT_DIR}/objconv && bash ../tools/build_objconv.sh)
fi
# Build the objconv tool
(cd ${ROOT_DIR}/objconv && bash ../tools/build_objconv.sh)
fi
}

Expand Down Expand Up @@ -148,14 +146,14 @@ function do_build_lib {
esac
case $interface64 in
1)
local interface64_flags="INTERFACE64=1 SYMBOLSUFFIX=64_ OBJCONV=$PWD/objconv/objconv";
local interface_flags="INTERFACE64=1 SYMBOLSUFFIX=64_ SYMBOLPREFIX=scipy_ OBJCONV=$PWD/objconv/objconv";
local symbolsuffix="64_";
if [ -n "$IS_OSX" ]; then
$PWD/objconv/objconv --help
fi
;;
*)
local interface64_flags=""
local interface_flags="SYMBOLPREFIX=scipy_ OBJCONV=$PWD/objconv/objconv"
local symbolsuffix="";
;;
esac
Expand All @@ -165,11 +163,11 @@ function do_build_lib {
git config --global --add safe.directory '*'
pushd OpenBLAS
patch_source
CFLAGS="$CFLAGS -fvisibility=protected" \
CFLAGS="$CFLAGS -fvisibility=protected -Wno-uninitialized" \
make BUFFERSIZE=20 DYNAMIC_ARCH=1 \
USE_OPENMP=0 NUM_THREADS=64 \
BINARY=$bitness $interface64_flags $target_flags > /dev/null
make PREFIX=$BUILD_PREFIX $interface64_flags install
BINARY=$bitness $interface_flags $target_flags > /dev/null
make PREFIX=$BUILD_PREFIX $interface_flags install
popd
stop_spinner
if [ "$nightly" = "1" ]; then
Expand All @@ -193,7 +191,7 @@ function do_build_lib {
tar zcvf libs/$out_name \
$BUILD_PREFIX/include/*blas* \
$BUILD_PREFIX/include/*lapack* \
$BUILD_PREFIX/lib/libopenblas* \
$BUILD_PREFIX/lib/libscipy_openblas* \
$BUILD_PREFIX/lib/pkgconfig/openblas* \
$BUILD_PREFIX/lib/cmake/openblas
}
Expand Down
Loading