Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add missing Python 3.11 dependencies. #29609

Closed
wants to merge 3 commits into from
Closed

Add missing Python 3.11 dependencies. #29609

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8178190
Add missing Python 3.11 dependencies.
tvalentyn Dec 5, 2023
6891007
Also install headers in the python dependency test suite.
tvalentyn Dec 5, 2023
e6627d2
Add snappy instructions for MacOS
tvalentyn Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions sdks/python/container/run_generate_requirements.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,18 @@
# You will need Python interpreters for all versions supported by Beam, see:
# https://s.apache.org/beam-python-dev-wiki

if ! [[ "$(expr substr $(uname -s) 1 5)" == "Linux" ]]; then
echo This script needs to be executed in a Linux environment.
exit 1
fi

if ! dpkg -s libsnappy-dev >/dev/null ; then
echo You must install libsnappy-dev to run this script. Run:
echo sudo apt install libsnappy-dev
if [[ "$(expr substr $(uname -s) 1 5)" == "Linux" ]]; then
if ! dpkg -s libsnappy-dev >/dev/null ; then
echo You must install libsnappy-dev to run this script. Run:
echo sudo apt install libsnappy-dev
exit 1
fi
elif [[ "$(expr substr $(uname -s) 1 6)" == "Darwin" ]]; then
if ! brew list snappy >/dev/null ; then
echo You must install snappy to run this script. Run:
echo brew install snappy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generating requirements from MacOS M1/M2 will remove snappy from the base_image_requirements.txt right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should not. MacOS users would have to install snappy package and rerun the script.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could try running the script on macos.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, let me try running it on M1 Mac.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, please let me know if you can get it to work. I briefly asked another coworker, seems like it's not enough - maybe restarting shell is required?

We also tried setting CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy

Copy link
Contributor

@AnandInguva AnandInguva Dec 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran > Task :sdks:python:container:py311:generatePythonRequirements FAILED
It failed for me.

  Building wheel for python-snappy (setup.py): started
  Building wheel for python-snappy (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
  Running setup.py clean for python-snappy
      creating build/lib.macosx-14.1-arm64-cpython-311
      creating build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/snappy.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/snappy_cffi.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/__init__.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/hadoop_snappy.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/snappy_formats.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/__main__.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      copying src/snappy/snappy_cffi_builder.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy
      running build_ext
      building 'snappy._snappy' extension
      creating build/temp.macosx-14.1-arm64-cpython-311
      creating build/temp.macosx-14.1-arm64-cpython-311/src
      creating build/temp.macosx-14.1-arm64-cpython-311/src/snappy
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/anandinguva/Desktop/projects/beam/build/python311_requirements_gen/include -I/Users/anandinguva/.pyenv/versions/3.11.5/include/python3.11 -c src/snappy/crc32c.c -o build/temp.macosx-14.1-arm64-cpython-311/src/snappy/crc32c.o
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/anandinguva/Desktop/projects/beam/build/python311_requirements_gen/include -I/Users/anandinguva/.pyenv/versions/3.11.5/include/python3.11 -c src/snappy/snappymodule.cc -o build/temp.macosx-14.1-arm64-cpython-311/src/snappy/snappymodule.o
      src/snappy/snappymodule.cc:33:10: fatal error: 'snappy-c.h' file not found
      #include <snappy-c.h>
               ^~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-snappy
Successfully built bs4 future google-cloud-profiler
Failed to build python-snappy
ERROR: Could not build wheels for python-snappy, which is required to install pyproject.toml-based projects

exit 1
fi
fi

if [[ $# -lt 2 ]]; then
Expand Down
Loading