Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python][FS][Azure] Enable Azure in pyarrow wheel builds #39352

Closed
Tom-Newton opened this issue Dec 22, 2023 · 3 comments · Fixed by #39971
Closed

[Python][FS][Azure] Enable Azure in pyarrow wheel builds #39352

Tom-Newton opened this issue Dec 22, 2023 · 3 comments · Fixed by #39971
Assignees
@Tom-Newton
Labels
Component: Python Type: enhancement
Milestone
16.0.0

Comments

@Tom-Newton
Copy link
Contributor

Describe the enhancement requested

Child issue of #39317

Currently Azure is disabled in pyarrow wheel builds.

There are a collection of scripts for pyarrow wheel builds on different OSs e.g. python_wheel_macos_build.sh, python_wheel_manylinux_build.sh, python_wheel_unix_test.sh, python_wheel_windows_build.bat, ...

Probably we should try to enable Azure by default in all the cases where S3 and GCS are also enabled by default.

Component(s)

Python
@Tom-Newton Tom-Newton mentioned this issue Dec 22, 2023
Open
@Tom-Newton
Copy link
Contributor Author

take

@Tom-Newton
Copy link
Contributor Author

Tom-Newton commented Jan 15, 2024 •
edited
Loading

I'm quite keen to target https://github.com/Azure/azure-sdk-for-cpp/releases/tag/azure-identity_1.6.0 as minimum version of the Azure SDK. This release has a fix for the default credential and adds workload identity to the default credential. Both of these are valuable to me and I assume others would benefit too.

Even the older version that we currently target requires an update to vcpkg though, unless we resort to doing a bundled build of the Azure SDK.

This was referenced Jan 15, 2024
Closed
Merged
kou added a commit that referenced this issue Feb 6, 2024
@Tom-Newton @kou @raulcd
GH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release (#39622)
1552510 
### Rationale for this change
Old version of vcpkg is blocking #39352

### What changes are included in this PR?
- Upgrade vcpkg 
- Fix ports patches
- Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg `abseil` build. 
- Move `VCPKG_FORCE_SYSTEM_BINARIES` to be set before vcpkg install to fix vcpkg install on linux ARM. 
- Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva. 
- Update image used for python wheel builds on windows to 2024-02-05

### Are these changes tested?
Does not change any behaviour so should be covered by existing tests

### Are there any user-facing changes?
There shouldn't be
* Closes: #39621

Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@Tom-Newton
Copy link
Contributor Author

This is unblocked now. I'm debating whether to go all in and enable Azure everywhere that GCS is enabled. Alternatively I could just start minimal until we actually have some pybinds.

@Tom-Newton Tom-Newton mentioned this issue Feb 6, 2024
Closed
@github-actions github-actions bot mentioned this issue Feb 7, 2024
Merged
@kou kou closed this as completed in #39971 Feb 9, 2024
kou pushed a commit that referenced this issue Feb 9, 2024
@Tom-Newton
GH-39352: [FS][Azure] Enable azure in builds (#39971)
a0dec7f 
### Rationale for this change

### What changes are included in this PR?
Enable Azure in linux and mac os wheel builds. Tried to copy GCS

Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR.  

### Are these changes tested?
There is no new functionality to test. 

### Are there any user-facing changes?
No

* Closes: #39352

Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@kou kou added this to the 16.0.0 milestone Feb 9, 2024
@Tom-Newton Tom-Newton mentioned this issue Feb 14, 2024
Closed
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@Tom-Newton @kou
@raulcd @dgreiss
apacheGH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release (a…
7fe27d1 
…pache#39622)

### Rationale for this change
Old version of vcpkg is blocking apache#39352

### What changes are included in this PR?
- Upgrade vcpkg 
- Fix ports patches
- Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg `abseil` build. 
- Move `VCPKG_FORCE_SYSTEM_BINARIES` to be set before vcpkg install to fix vcpkg install on linux ARM. 
- Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva. 
- Update image used for python wheel builds on windows to 2024-02-05

### Are these changes tested?
Does not change any behaviour so should be covered by existing tests

### Are there any user-facing changes?
There shouldn't be
* Closes: apache#39621

Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@Tom-Newton @dgreiss
apacheGH-39352: [FS][Azure] Enable azure in builds (apache#39971)
f86f122 
### Rationale for this change

### What changes are included in this PR?
Enable Azure in linux and mac os wheel builds. Tried to copy GCS

Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR.  

### Are these changes tested?
There is no new functionality to test. 

### Are there any user-facing changes?
No

* Closes: apache#39352

Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Feb 28, 2024
@Tom-Newton @kou
@raulcd @zanmato1984
apacheGH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release (a…
fc06112 
…pache#39622)

### Rationale for this change
Old version of vcpkg is blocking apache#39352

### What changes are included in this PR?
- Upgrade vcpkg 
- Fix ports patches
- Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg `abseil` build. 
- Move `VCPKG_FORCE_SYSTEM_BINARIES` to be set before vcpkg install to fix vcpkg install on linux ARM. 
- Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva. 
- Update image used for python wheel builds on windows to 2024-02-05

### Are these changes tested?
Does not change any behaviour so should be covered by existing tests

### Are there any user-facing changes?
There shouldn't be
* Closes: apache#39621

Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Feb 28, 2024
@Tom-Newton @zanmato1984
apacheGH-39352: [FS][Azure] Enable azure in builds (apache#39971)
a1a50c8 
### Rationale for this change

### What changes are included in this PR?
Enable Azure in linux and mac os wheel builds. Tried to copy GCS

Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR.  

### Are these changes tested?
There is no new functionality to test. 

### Are there any user-facing changes?
No

* Closes: apache#39352

Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
@Tom-Newton @kou
@raulcd @thisisnic
apacheGH-39621: [CI][Packaging] Update vcpkg to 2023.11.20 release (a…
8b66b89 
…pache#39622)

### Rationale for this change
Old version of vcpkg is blocking apache#39352

### What changes are included in this PR?
- Upgrade vcpkg 
- Fix ports patches
- Upgrade visual studio used in windows wheel builds. VS2019 is now required for the vcpkg `abseil` build. 
- Move `VCPKG_FORCE_SYSTEM_BINARIES` to be set before vcpkg install to fix vcpkg install on linux ARM. 
- Fix for LLVM 17 which requires that an executable exports "llvm_orc_registerEHFrameSectionWrapper()" and "llvm_orc_unregisterEHFrameSectionWrapper()". This effects the java builds that depend on llvm from vcpkg for gandiva. 
- Update image used for python wheel builds on windows to 2024-02-05

### Are these changes tested?
Does not change any behaviour so should be covered by existing tests

### Are there any user-facing changes?
There shouldn't be
* Closes: apache#39621

Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
@Tom-Newton @thisisnic
apacheGH-39352: [FS][Azure] Enable azure in builds (apache#39971)
73119fd 
### Rationale for this change

### What changes are included in this PR?
Enable Azure in linux and mac os wheel builds. Tried to copy GCS

Don't enable Azure for windows builds because windows builds where all failing. Failures were a combination of cmake version or `Could not find a package configuration file provided by "wil"`. I think it makes sense to come back to windows builds in another PR.  

### Are these changes tested?
There is no new functionality to test. 

### Are there any user-facing changes?
No

* Closes: apache#39352

Authored-by: Thomas Newton <thomas.w.newton@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Tom-Newton Tom-Newton

Labels
Component: Python Type: enhancement
Projects
None yet
Milestone
16.0.0
Development

Successfully merging a pull request may close this issue.

GH-39352: [FS][Azure] Enable azure in builds Tom-Newton/arrow
2 participants
@kou @Tom-Newton