Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[10.0.x] Rebuild for libprotobuf4233_libgrpc156 #1106

Merged
merged 7 commits into from
Jul 10, 2023
Merged

[10.0.x] Rebuild for libprotobuf4233_libgrpc156 #1106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bc526a2
disable pytorch/tensorflow as test-deps
h-vetinari Jul 1, 2023
2332e35
Rebuild for libprotobuf4233_libgrpc156
regro-cf-autotick-bot Jul 3, 2023
fdf88fd
ensure pyarrow can depend on libarrow for different protobufs
h-vetinari Jul 10, 2023
f8b8972
MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.24.0, and co…
h-vetinari Jul 10, 2023
c6b2b87
clean up between libarrow builds also on windows
h-vetinari Jul 10, 2023
db22f14
run test suite against libarrow for both protobufs
h-vetinari Jul 10, 2023
6cb2c7a
backport patch to disable useless pkgconfig search
h-vetinari Jul 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .azure-pipelines/azure-pipelines-win.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .ci_support/linux_64_cuda_compiler_version10.2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/linux_64_cuda_compiler_versionNone.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/linux_aarch64_cuda_compiler_version11.2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/linux_aarch64_cuda_compiler_versionNone.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/linux_ppc64le_cuda_compiler_version11.2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/linux_ppc64le_cuda_compiler_versionNone.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
11 changes: 11 additions & 0 deletions .ci_support/migrations/libprotobuf4233_libgrpc156.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
__migrator:
build_number: 1
kind: version
migration_number: 1
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- '4.23.3'
migrator_ts: 1688154716.9176981
2 changes: 2 additions & 0 deletions .ci_support/osx_64_.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
macos_machine:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/osx_arm64_.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,10 @@ libabseil:
- '20230125'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
macos_machine:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/win_64_cuda_compiler_version10.2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,10 @@ libcurl:
- '8'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
2 changes: 2 additions & 0 deletions .ci_support/win_64_cuda_compiler_versionNone.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,10 @@ libcurl:
- '8'
libgrpc:
- '1.54'
- '1.56'
libprotobuf:
- '3.21'
- 4.23.3
lz4_c:
- 1.9.3
numpy:
Expand Down
10 changes: 5 additions & 5 deletions .scripts/run_osx_build.sh
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 16 additions & 11 deletions README.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 5 additions & 2 deletions recipe/build-arrow.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
@echo on

mkdir "%SRC_DIR%"\cpp\build
pushd "%SRC_DIR%"\cpp\build
mkdir cpp\build
pushd cpp\build

:: Enable CUDA support
if "%cuda_compiler_version%"=="None" (
Expand Down Expand Up @@ -58,3 +58,6 @@ cmake --build . --target install --config Release
if %ERRORLEVEL% neq 0 exit 1

popd

:: clean up between builds (and to save space)
rmdir /s /q cpp\build
2 changes: 1 addition & 1 deletion recipe/build-arrow.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,5 +116,5 @@ cmake --build . --target install --config Release

popd

# clean up artefacts afterwards to save space
# clean up between builds (and to save space)
rm -rf cpp/build
24 changes: 17 additions & 7 deletions recipe/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,15 @@ source:
- patches/0003-don-t-bake-non-relocatable-CMAKE_INSTALL_FULL_LIBDIR.patch
# backport apache/arrow#34747 for compatibility with grpc 1.54
- patches/0004-GH-34743-Python-Relax-condition-in-flaky-Flight-test.patch
# backport apache/arrow#34019 to disable useless pkgconfig search that takes ~15-60min
- patches/0005-GH-33882-C-Don-t-find-.pc-files-with-ARROW_BUILD_STA.patch
# testing-submodule not part of release tarball
- git_url: https://github.com/apache/arrow-testing.git
git_rev: 00c483283433b4c02cb811f260dbe35414c806a4
folder: testing

build:
number: 32
number: 33
# for cuda support, building with one version is enough to be compatible with
# all later versions, since arrow is only using libcuda, and not libcudart.
skip: true # [cuda_compiler_version not in ("None", cuda_compiler_version_min)]
Expand Down Expand Up @@ -210,6 +212,8 @@ outputs:
requirements:
host:
- {{ pin_subpackage('libarrow', exact=True) }}
# avoid wrappers for different builds colliding due to identical hashes
- libprotobuf
run:
- {{ pin_subpackage('libarrow', exact=True) }}
test:
Expand Down Expand Up @@ -245,7 +249,9 @@ outputs:
- cmake
- ninja
host:
- {{ pin_subpackage('libarrow', exact=True) }}
# we're building for two protobuf versions, cannot pin exactly
# - {{ pin_subpackage('libarrow', exact=True) }}
- libarrow ={{ version }}=*_{{ PKG_BUILDNUM }}_{{ build_ext }}
- clangdev {{ llvm_version }}
- llvmdev {{ llvm_version }}
- cython
Expand All @@ -255,7 +261,8 @@ outputs:
- setuptools
- setuptools_scm
run:
- {{ pin_subpackage('libarrow', exact=True) }}
# - {{ pin_subpackage('libarrow', exact=True) }}
- libarrow ={{ version }}=*_{{ PKG_BUILDNUM }}_{{ build_ext }}
- {{ pin_compatible('numpy') }}
- python
run_constrained:
Expand Down Expand Up @@ -349,6 +356,9 @@ outputs:
# CUDA builds for aarch/ppc consistently run out of disk space on azure for some reason
test:
requires:
# vary protobuf version in test suite (historically, test failures only have a very
# weak dependency on python version, so we don't lose coverage by doing half & half)
- libprotobuf <4 # [py % 2 == 0]
# temporary pin due to missing fixture
- pytest <7.4.0
- pytest-lazy-fixture
Expand All @@ -362,10 +372,10 @@ outputs:
# see https://github.com/conda-forge/arrow-cpp-feedstock/issues/1016
- pandas <2
- scipy
# not all OSes/arches available in conda-forge
- pytorch * # [unix and not ppc64le]
# not yet rebuilt for libabseil 20230125
# - tensorflow # [unix and x86_64 and py<311]
# these are generally (far) behind on migrating abseil/grpc/protobuf,
# and using them as test dependencies blocks the migrator unnecessarily
# - pytorch
# - tensorflow
# we're not building java bindings
# - jpype1
# doesn't get picked up correctly
Expand Down
5 changes: 1 addition & 4 deletions recipe/patches/0001-ARROW-18340-Python-PyArrow-C-header-files-no-longer-.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 984b9c8cf0b72e24e8917900862b37964fb27e58 Mon Sep 17 00:00:00 2001
From: Alenka Frim <AlenkaF@users.noreply.github.com>
Date: Tue, 22 Nov 2022 22:01:32 +0100
Subject: [PATCH 1/4] ARROW-18340: [Python] PyArrow C++ header files no longer
Subject: [PATCH 1/5] ARROW-18340: [Python] PyArrow C++ header files no longer
always included in installed pyarrow (#14656)

This PR adds code to `_run_cmake` to always copy PyArrow C++ header files to `pyarrow/iclude`.
Expand Down Expand Up @@ -81,6 +81,3 @@ index 2a7e5f7a3..528d9537f 100755

# Move the built C-extension to the place expected by the Python
# build
--
2.40.1.windows.1

Loading