Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow running examples on Apple Silicon M1 and fix image build errors for arm64 #1898

Merged
merged 1 commit into from
Jun 15, 2022
Merged

Allow running examples on Apple Silicon M1 and fix image build errors for arm64 #1898

merged 1 commit into from
Jun 15, 2022

Conversation

tenzen-y
Copy link
Member

@tenzen-y tenzen-y commented Jun 14, 2022 •
edited
Loading

What this PR does / why we need it:

I added codes to change the katib-mysql image to arm64v8/mysql:8.0.29-oracle to examples/v1beta1/kind-cluster/deploy.sh since we get the following errors when we run KinD Cluster examples on Apple Silicon M1 Mac.

Also I modified Dockerfile to fix this TensorFlow issue in tf-mnist-with-summaries and enas-cnn-cifar10.

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  26s   default-scheduler  Successfully assigned kubeflow/katib-mysql-6975d6c6c4-4h2cn to kind-control-plane
  Normal   Pulling    25s   kubelet            Pulling image "mysql:8.0.29"
  Warning  Failed     1s    kubelet            Failed to pull image "mysql:8.0.29": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/mysql:8.0.29": no match for platform in manifest: not found
  Warning  Failed     1s    kubelet            Error: ErrImagePull
  Normal   BackOff    1s    kubelet            Back-off pulling image "mysql:8.0.29"
  Warning  Failed     1s    kubelet            Error: ImagePullBackOff

In addition, I fixed some following errors when we build some images for arm64.

error logs for tfevent-metricscollector 1 
 => ERROR [5/6] RUN pip install --no-cache-dir -r requirements.txt                                                                                                                                                                37.5s
------                                                                                                                                                                                                                                  
 > [5/6] RUN pip install --no-cache-dir -r requirements.txt:                                                                                                                                                                            
#9 1.115 Ignoring tensorflow: markers 'platform_machine == "x86_64"' don't match your environment                                                                                                                                       
#9 1.301 Collecting psutil==5.8.0                                                                                                                                                                                                       
#9 34.89 Collecting oauthlib>=3.0.0
#9 34.90   Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
#9 34.92      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 KB 12.5 MB/s eta 0:00:00
#9 34.99 Building wheels for collected packages: psutil, googleapis-common-protos, termcolor
#9 34.99   Building wheel for psutil (setup.py): started
#9 35.13   Building wheel for psutil (setup.py): finished with status 'error'
#9 35.14   error: subprocess-exited-with-error
#9 35.14   
#9 35.14   × python setup.py bdist_wheel did not run successfully.
#9 35.14   │ exit code: 1
#9 35.14   ╰─> [43 lines of output]
#9 35.14       running bdist_wheel
#9 35.14       running build
#9 35.14       running build_py
#9 35.14       creating build
#9 35.14       creating build/lib.linux-aarch64-3.9
#9 35.14       creating build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psosx.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psaix.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_pswindows.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_pssunos.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psposix.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_common.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_compat.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_pslinux.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psbsd.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/__init__.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       creating build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_sunos.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_bsd.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_testutils.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_linux.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_system.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_osx.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_aix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_memleaks.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_misc.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_windows.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/runner.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_unicode.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_process.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/__main__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_connections.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_posix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_contracts.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/__init__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       running build_ext
#9 35.14       building 'psutil._psutil_linux' extension
#9 35.14       creating build/temp.linux-aarch64-3.9
#9 35.14       creating build/temp.linux-aarch64-3.9/psutil
#9 35.14       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=580 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-3.9/psutil/_psutil_common.o
#9 35.14       C compiler or Python headers are not installed on this system. Try to run:
#9 35.14       sudo apt-get install gcc python3-dev
#9 35.14       error: command 'gcc' failed: No such file or directory
#9 35.14       [end of output]
#9 35.14   
#9 35.14   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 35.14   ERROR: Failed building wheel for psutil
#9 35.14   Running setup.py clean for psutil
#9 35.26   Building wheel for googleapis-common-protos (setup.py): started
#9 35.42   Building wheel for googleapis-common-protos (setup.py): finished with status 'done'
#9 35.42   Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.6.0-py3-none-any.whl size=77593 sha256=549ac859f2d3a2c1015f9b8a93c18924865ba392c2bcb4d960e53ceded35d93c
#9 35.42   Stored in directory: /tmp/pip-ephem-wheel-cache-mdsgn90y/wheels/e8/1f/11/8233e1627c658357f6be4ea4e353e54034485fc9496afc333f
#9 35.42   Building wheel for termcolor (setup.py): started
#9 35.54   Building wheel for termcolor (setup.py): finished with status 'done'
#9 35.54   Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4848 sha256=de51087fa392b8c1c5659173cad37c613c36461341bdc8fcaa9b65354db58038
#9 35.54   Stored in directory: /tmp/pip-ephem-wheel-cache-mdsgn90y/wheels/b6/0d/90/0d1bbd99855f99cb2f6c2e5ff96f8023fad8ec367695f7d72d
#9 35.54 Successfully built googleapis-common-protos termcolor
#9 35.54 Failed to build psutil
#9 35.66 Installing collected packages: termcolor, tensorboard-plugin-wit, rfc3339, pyasn1, libclang, keras, flatbuffers, zipp, wrapt, werkzeug, urllib3, typing-extensions, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, rsa, pyparsing, pyasn1-modules, psutil, protobuf, oauthlib, numpy, idna, gast, charset-normalizer, certifi, cachetools, absl-py, requests, packaging, opt-einsum, keras-preprocessing, importlib-metadata, h5py, grpcio, googleapis-common-protos, google-pasta, google-auth, astunparse, requests-oauthlib, markdown, google-auth-oauthlib, tensorboard, tensorflow-aarch64
#9 37.03   Running setup.py install for psutil: started
#9 37.15   Running setup.py install for psutil: finished with status 'error'
#9 37.16   error: subprocess-exited-with-error
#9 37.16   
#9 37.16   × Running setup.py install for psutil did not run successfully.
#9 37.16   │ exit code: 1
#9 37.16   ╰─> [43 lines of output]
#9 37.16       running install
#9 37.16       running build
#9 37.16       running build_py
#9 37.16       creating build
#9 37.16       creating build/lib.linux-aarch64-3.9
#9 37.16       creating build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psosx.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psaix.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_pswindows.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_pssunos.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psposix.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_common.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_compat.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_pslinux.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psbsd.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/__init__.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       creating build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_sunos.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_bsd.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_testutils.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_linux.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_system.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_osx.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_aix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_memleaks.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_misc.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_windows.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/runner.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_unicode.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_process.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/__main__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_connections.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_posix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_contracts.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/__init__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       running build_ext
#9 37.16       building 'psutil._psutil_linux' extension
#9 37.16       creating build/temp.linux-aarch64-3.9
#9 37.16       creating build/temp.linux-aarch64-3.9/psutil
#9 37.16       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=580 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-3.9/psutil/_psutil_common.o
#9 37.16       C compiler or Python headers are not installed on this system. Try to run:
#9 37.16       sudo apt-get install gcc python3-dev
#9 37.16       error: command 'gcc' failed: No such file or directory
#9 37.16       [end of output]
#9 37.16   
#9 37.16   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 37.16 error: legacy-install-failure
#9 37.16 
#9 37.16 × Encountered error while trying to install package.
#9 37.16 ╰─> psutil
#9 37.16 
#9 37.16 note: This is an issue with the package mentioned above, not pip.
#9 37.16 hint: See above for output from the failure.
#9 37.23 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
#9 37.23 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --no-cache-dir -r requirements.txt]: exit code: 1
error logs for tfevent-metricscollector 2 
 => ERROR [5/6] RUN pip install --no-cache-dir -r requirements.txt                                                                                                                                                                37.5s
------                                                                                                                                                                                                                                  
 > [5/6] RUN pip install --no-cache-dir -r requirements.txt:                                                                                                                                                                            
#9 1.115 Ignoring tensorflow: markers 'platform_machine == "x86_64"' don't match your environment                                                                                                                                       
#9 1.301 Collecting psutil==5.8.0                                                                                                                                                                                                       
#9 34.89 Collecting oauthlib>=3.0.0
#9 34.90   Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
#9 34.92      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.5/151.5 KB 12.5 MB/s eta 0:00:00
#9 34.99 Building wheels for collected packages: psutil, googleapis-common-protos, termcolor
#9 34.99   Building wheel for psutil (setup.py): started
#9 35.13   Building wheel for psutil (setup.py): finished with status 'error'
#9 35.14   error: subprocess-exited-with-error
#9 35.14   
#9 35.14   × python setup.py bdist_wheel did not run successfully.
#9 35.14   │ exit code: 1
#9 35.14   ╰─> [43 lines of output]
#9 35.14       running bdist_wheel
#9 35.14       running build
#9 35.14       running build_py
#9 35.14       creating build
#9 35.14       creating build/lib.linux-aarch64-3.9
#9 35.14       creating build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psosx.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psaix.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_pswindows.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_pssunos.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psposix.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_common.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_compat.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_pslinux.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/_psbsd.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       copying psutil/__init__.py -> build/lib.linux-aarch64-3.9/psutil
#9 35.14       creating build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_sunos.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_bsd.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_testutils.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_linux.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_system.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_osx.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_aix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_memleaks.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_misc.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_windows.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/runner.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_unicode.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_process.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/__main__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_connections.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_posix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/test_contracts.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       copying psutil/tests/__init__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 35.14       running build_ext
#9 35.14       building 'psutil._psutil_linux' extension
#9 35.14       creating build/temp.linux-aarch64-3.9
#9 35.14       creating build/temp.linux-aarch64-3.9/psutil
#9 35.14       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=580 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-3.9/psutil/_psutil_common.o
#9 35.14       C compiler or Python headers are not installed on this system. Try to run:
#9 35.14       sudo apt-get install gcc python3-dev
#9 35.14       error: command 'gcc' failed: No such file or directory
#9 35.14       [end of output]
#9 35.14   
#9 35.14   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 35.14   ERROR: Failed building wheel for psutil
#9 35.14   Running setup.py clean for psutil
#9 35.26   Building wheel for googleapis-common-protos (setup.py): started
#9 35.42   Building wheel for googleapis-common-protos (setup.py): finished with status 'done'
#9 35.42   Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.6.0-py3-none-any.whl size=77593 sha256=549ac859f2d3a2c1015f9b8a93c18924865ba392c2bcb4d960e53ceded35d93c
#9 35.42   Stored in directory: /tmp/pip-ephem-wheel-cache-mdsgn90y/wheels/e8/1f/11/8233e1627c658357f6be4ea4e353e54034485fc9496afc333f
#9 35.42   Building wheel for termcolor (setup.py): started
#9 35.54   Building wheel for termcolor (setup.py): finished with status 'done'
#9 35.54   Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4848 sha256=de51087fa392b8c1c5659173cad37c613c36461341bdc8fcaa9b65354db58038
#9 35.54   Stored in directory: /tmp/pip-ephem-wheel-cache-mdsgn90y/wheels/b6/0d/90/0d1bbd99855f99cb2f6c2e5ff96f8023fad8ec367695f7d72d
#9 35.54 Successfully built googleapis-common-protos termcolor
#9 35.54 Failed to build psutil
#9 35.66 Installing collected packages: termcolor, tensorboard-plugin-wit, rfc3339, pyasn1, libclang, keras, flatbuffers, zipp, wrapt, werkzeug, urllib3, typing-extensions, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, rsa, pyparsing, pyasn1-modules, psutil, protobuf, oauthlib, numpy, idna, gast, charset-normalizer, certifi, cachetools, absl-py, requests, packaging, opt-einsum, keras-preprocessing, importlib-metadata, h5py, grpcio, googleapis-common-protos, google-pasta, google-auth, astunparse, requests-oauthlib, markdown, google-auth-oauthlib, tensorboard, tensorflow-aarch64
#9 37.03   Running setup.py install for psutil: started
#9 37.15   Running setup.py install for psutil: finished with status 'error'
#9 37.16   error: subprocess-exited-with-error
#9 37.16   
#9 37.16   × Running setup.py install for psutil did not run successfully.
#9 37.16   │ exit code: 1
#9 37.16   ╰─> [43 lines of output]
#9 37.16       running install
#9 37.16       running build
#9 37.16       running build_py
#9 37.16       creating build
#9 37.16       creating build/lib.linux-aarch64-3.9
#9 37.16       creating build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psosx.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psaix.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_pswindows.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_pssunos.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psposix.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_common.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_compat.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_pslinux.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/_psbsd.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       copying psutil/__init__.py -> build/lib.linux-aarch64-3.9/psutil
#9 37.16       creating build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_sunos.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_bsd.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_testutils.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_linux.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_system.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_osx.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_aix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_memleaks.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_misc.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_windows.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/runner.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_unicode.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_process.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/__main__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_connections.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_posix.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/test_contracts.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       copying psutil/tests/__init__.py -> build/lib.linux-aarch64-3.9/psutil/tests
#9 37.16       running build_ext
#9 37.16       building 'psutil._psutil_linux' extension
#9 37.16       creating build/temp.linux-aarch64-3.9
#9 37.16       creating build/temp.linux-aarch64-3.9/psutil
#9 37.16       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=580 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-3.9/psutil/_psutil_common.o
#9 37.16       C compiler or Python headers are not installed on this system. Try to run:
#9 37.16       sudo apt-get install gcc python3-dev
#9 37.16       error: command 'gcc' failed: No such file or directory
#9 37.16       [end of output]
#9 37.16   
#9 37.16   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 37.16 error: legacy-install-failure
#9 37.16 
#9 37.16 × Encountered error while trying to install package.
#9 37.16 ╰─> psutil
#9 37.16 
#9 37.16 note: This is an issue with the package mentioned above, not pip.
#9 37.16 hint: See above for output from the failure.
#9 37.23 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
#9 37.23 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --no-cache-dir -r requirements.txt]: exit code: 1
error logs for suggestion-chocolate 1 
=> ERROR [stage-1 7/8] RUN pip install --no-cache-dir -r requirements.txt                                                                                                                                                        22.0s
------
 > [stage-1 7/8] RUN pip install --no-cache-dir -r requirements.txt:
#14 1.132 Collecting git+https://github.com/AIworx-Labs/chocolate@master (from -r requirements.txt (line 10))
#14 20.51 Collecting MarkupSafe>=0.9.2
#14 20.52   Downloading MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26 kB)
#14 20.52 Building wheels for collected packages: googleapis-common-protos, chocolate, ghalton
#14 20.52   Building wheel for googleapis-common-protos (setup.py): started
#14 20.71   Building wheel for googleapis-common-protos (setup.py): finished with status 'done'
#14 20.71   Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.6.0-py3-none-any.whl size=77593 sha256=6fdb27eedcd49934c77a66aaa50c71e2123e633f2feff2c3aeed969bdaf1c926
#14 20.71   Stored in directory: /tmp/pip-ephem-wheel-cache-pyd5c_jf/wheels/e8/1f/11/8233e1627c658357f6be4ea4e353e54034485fc9496afc333f
#14 20.71   Building wheel for chocolate (setup.py): started
#14 21.05   Building wheel for chocolate (setup.py): finished with status 'done'
#14 21.05   Created wheel for chocolate: filename=chocolate-0.6-cp39-cp39-linux_aarch64.whl size=43127 sha256=00873b40dd1c883af4a739b1b19e491d160949bce16008d373e581518619696f
#14 21.05   Stored in directory: /tmp/pip-ephem-wheel-cache-pyd5c_jf/wheels/68/33/a2/82eeb15c0669e449fb81d64e03ebb1f04a97338857164087d1
#14 21.05   Building wheel for ghalton (setup.py): started
#14 21.16   Building wheel for ghalton (setup.py): finished with status 'error'
#14 21.17   error: subprocess-exited-with-error
#14 21.17   
#14 21.17   × python setup.py bdist_wheel did not run successfully.
#14 21.17   │ exit code: 1
#14 21.17   ╰─> [17 lines of output]
#14 21.17       running bdist_wheel
#14 21.17       running build
#14 21.17       running build_py
#14 21.17       creating build
#14 21.17       creating build/lib.linux-aarch64-3.9
#14 21.17       creating build/lib.linux-aarch64-3.9/ghalton
#14 21.17       copying ghalton/constants.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.17       copying ghalton/ghalton_wrapper.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.17       copying ghalton/__init__.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.17       running build_ext
#14 21.17       building 'ghalton._ghalton_wrapper' extension
#14 21.17       creating build/temp.linux-aarch64-3.9
#14 21.17       creating build/temp.linux-aarch64-3.9/src
#14 21.17       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/Halton.cpp -o build/temp.linux-aarch64-3.9/src/Halton.o
#14 21.17       gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
#14 21.17       compilation terminated.
#14 21.17       error: command '/usr/bin/gcc' failed with exit code 1
#14 21.17       [end of output]
#14 21.17   
#14 21.17   note: This error originates from a subprocess, and is likely not a problem with pip.
#14 21.17   ERROR: Failed building wheel for ghalton
#14 21.17   Running setup.py clean for ghalton
#14 21.27 Successfully built googleapis-common-protos chocolate
#14 21.27 Failed to build ghalton
#14 21.38 Installing collected packages: pytz, nose, ghalton, cloudpickle, banal, threadpoolctl, six, protobuf, numpy, MarkupSafe, joblib, greenlet, filelock, cython, SQLAlchemy, scipy, python-dateutil, Mako, grpcio, googleapis-common-protos, scikit-learn, pandas, alembic, forestci, dataset, chocolate
#14 21.50   Running setup.py install for ghalton: started
#14 21.60   Running setup.py install for ghalton: finished with status 'error'
#14 21.60   error: subprocess-exited-with-error
#14 21.60   
#14 21.60   × Running setup.py install for ghalton did not run successfully.
#14 21.60   │ exit code: 1
#14 21.60   ╰─> [17 lines of output]
#14 21.60       running install
#14 21.60       running build
#14 21.60       running build_py
#14 21.60       creating build
#14 21.60       creating build/lib.linux-aarch64-3.9
#14 21.60       creating build/lib.linux-aarch64-3.9/ghalton
#14 21.60       copying ghalton/constants.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.60       copying ghalton/ghalton_wrapper.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.60       copying ghalton/__init__.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.60       running build_ext
#14 21.60       building 'ghalton._ghalton_wrapper' extension
#14 21.60       creating build/temp.linux-aarch64-3.9
#14 21.60       creating build/temp.linux-aarch64-3.9/src
#14 21.60       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/Halton.cpp -o build/temp.linux-aarch64-3.9/src/Halton.o
#14 21.60       gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
#14 21.60       compilation terminated.
#14 21.60       error: command '/usr/bin/gcc' failed with exit code 1
#14 21.60       [end of output]
#14 21.60   
#14 21.60   note: This error originates from a subprocess, and is likely not a problem with pip.
#14 21.60 error: legacy-install-failure
#14 21.60 
#14 21.60 × Encountered error while trying to install package.
#14 21.60 ╰─> ghalton
#14 21.60 
#14 21.60 note: This is an issue with the package mentioned above, not pip.
#14 21.60 hint: See above for output from the failure.
#14 21.75 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
#14 21.75 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --no-cache-dir -r requirements.txt]: exit code: 1
error logs for suggestion-chocolate 2 
=> ERROR [stage-1 7/8] RUN pip install --no-cache-dir -r requirements.txt                                                                                                                                                        22.6s
------
 > [stage-1 7/8] RUN pip install --no-cache-dir -r requirements.txt:
#14 1.149 Collecting git+https://github.com/AIworx-Labs/chocolate@master (from -r requirements.txt (line 10))
#14 21.21 Collecting MarkupSafe>=0.9.2
#14 21.22   Downloading MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26 kB)
#14 21.23 Building wheels for collected packages: googleapis-common-protos, chocolate, ghalton
#14 21.23   Building wheel for googleapis-common-protos (setup.py): started
#14 21.42   Building wheel for googleapis-common-protos (setup.py): finished with status 'done'
#14 21.42   Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.6.0-py3-none-any.whl size=77593 sha256=6a6b2f066bd8797cef36124bb0cb2586470ac28b1a5384aa49c2a7db75986dbc
#14 21.42   Stored in directory: /tmp/pip-ephem-wheel-cache-w9_d5wnc/wheels/e8/1f/11/8233e1627c658357f6be4ea4e353e54034485fc9496afc333f
#14 21.42   Building wheel for chocolate (setup.py): started
#14 21.76   Building wheel for chocolate (setup.py): finished with status 'done'
#14 21.76   Created wheel for chocolate: filename=chocolate-0.6-cp39-cp39-linux_aarch64.whl size=43127 sha256=88c568c252a686725638e364ca5667036564fdb9bf47b9e2cdd2d6f6f213c5a1
#14 21.76   Stored in directory: /tmp/pip-ephem-wheel-cache-w9_d5wnc/wheels/68/33/a2/82eeb15c0669e449fb81d64e03ebb1f04a97338857164087d1
#14 21.76   Building wheel for ghalton (setup.py): started
#14 21.87   Building wheel for ghalton (setup.py): finished with status 'error'
#14 21.88   error: subprocess-exited-with-error
#14 21.88   
#14 21.88   × python setup.py bdist_wheel did not run successfully.
#14 21.88   │ exit code: 1
#14 21.88   ╰─> [17 lines of output]
#14 21.88       running bdist_wheel
#14 21.88       running build
#14 21.88       running build_py
#14 21.88       creating build
#14 21.88       creating build/lib.linux-aarch64-3.9
#14 21.88       creating build/lib.linux-aarch64-3.9/ghalton
#14 21.88       copying ghalton/constants.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.88       copying ghalton/ghalton_wrapper.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.88       copying ghalton/__init__.py -> build/lib.linux-aarch64-3.9/ghalton
#14 21.88       running build_ext
#14 21.88       building 'ghalton._ghalton_wrapper' extension
#14 21.88       creating build/temp.linux-aarch64-3.9
#14 21.88       creating build/temp.linux-aarch64-3.9/src
#14 21.88       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/Halton.cpp -o build/temp.linux-aarch64-3.9/src/Halton.o
#14 21.88       gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
#14 21.88       compilation terminated.
#14 21.88       error: command '/usr/bin/gcc' failed with exit code 1
#14 21.88       [end of output]
#14 21.88   
#14 21.88   note: This error originates from a subprocess, and is likely not a problem with pip.
#14 21.88   ERROR: Failed building wheel for ghalton
#14 21.88   Running setup.py clean for ghalton
#14 21.98 Successfully built googleapis-common-protos chocolate
#14 21.98 Failed to build ghalton
#14 22.09 Installing collected packages: pytz, nose, ghalton, cloudpickle, banal, threadpoolctl, six, protobuf, numpy, MarkupSafe, joblib, greenlet, filelock, cython, SQLAlchemy, scipy, python-dateutil, Mako, grpcio, googleapis-common-protos, scikit-learn, pandas, alembic, forestci, dataset, chocolate
#14 22.21   Running setup.py install for ghalton: started
#14 22.31   Running setup.py install for ghalton: finished with status 'error'
#14 22.31   error: subprocess-exited-with-error
#14 22.31   
#14 22.31   × Running setup.py install for ghalton did not run successfully.
#14 22.31   │ exit code: 1
#14 22.31   ╰─> [17 lines of output]
#14 22.31       running install
#14 22.31       running build
#14 22.31       running build_py
#14 22.31       creating build
#14 22.31       creating build/lib.linux-aarch64-3.9
#14 22.31       creating build/lib.linux-aarch64-3.9/ghalton
#14 22.31       copying ghalton/constants.py -> build/lib.linux-aarch64-3.9/ghalton
#14 22.31       copying ghalton/ghalton_wrapper.py -> build/lib.linux-aarch64-3.9/ghalton
#14 22.31       copying ghalton/__init__.py -> build/lib.linux-aarch64-3.9/ghalton
#14 22.31       running build_ext
#14 22.31       building 'ghalton._ghalton_wrapper' extension
#14 22.31       creating build/temp.linux-aarch64-3.9
#14 22.31       creating build/temp.linux-aarch64-3.9/src
#14 22.31       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/Halton.cpp -o build/temp.linux-aarch64-3.9/src/Halton.o
#14 22.31       gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
#14 22.31       compilation terminated.
#14 22.31       error: command '/usr/bin/gcc' failed with exit code 1
#14 22.31       [end of output]
#14 22.31   
#14 22.31   note: This error originates from a subprocess, and is likely not a problem with pip.
#14 22.31 error: legacy-install-failure
#14 22.31 
#14 22.31 × Encountered error while trying to install package.
#14 22.31 ╰─> ghalton
#14 22.31 
#14 22.31 note: This is an issue with the package mentioned above, not pip.
#14 22.31 hint: See above for output from the failure.
#14 22.38 WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
#14 22.38 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --no-cache-dir -r requirements.txt]: exit code: 1

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

  • Docs included if any changes are user facing

@tenzen-y tenzen-y changed the title allow to run examples on Apple Silicon M1 Allow running examples on Apple Silicon M1 Jun 14, 2022
@tenzen-y tenzen-y force-pushed the allow-to-run-examples-in-m1-mac branch from f8e8830 to 9e31474 Compare June 14, 2022 21:05
@tenzen-y tenzen-y changed the title Allow running examples on Apple Silicon M1 Allow running examples on Apple Silicon M1 and fix image build errors for arm64 Jun 14, 2022
@tenzen-y
Copy link
Member Author

We should contain these fixes in the next Katib release to resolve image build errors for arm64.
/cc @kubeflow/wg-automl-leads

@google-oss-prow google-oss-prow bot requested a review from a team June 14, 2022 21:24
@coveralls
Copy link

coveralls commented Jun 14, 2022 •
edited
Loading

Coverage Status

Coverage remained the same at 73.697% when pulling 9e31474 on tenzen-y:allow-to-run-examples-in-m1-mac into 170647d on kubeflow:master.

johnugeorge
johnugeorge reviewed Jun 15, 2022
View reviewed changes
examples/v1beta1/kind-cluster/deploy.sh
@@ -36,8 +36,8 @@ if [ -z "$(command -v kubectl)" ]; then
exit 1
fi

# Step 1. Create Kind cluster with Kubernetes v1.22.9
kind create cluster --image kindest/node:v1.22.9
# Step 1. Create Kind cluster with Kubernetes v1.23.6
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason for changing default?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I upgraded the KinD Kubernetes version since we have upgraded Kubernetes dependencies to v0.23.
Also, according to this doc, K8s v1.22 reach EoL before we will release after the next Katib major version.

johnugeorge
johnugeorge reviewed Jun 15, 2022
View reviewed changes
scripts/v1beta1/build.sh
fi

echo -e "\nBuilding Tensorflow with summaries mnist training container example...\n"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are arm64 images built only for tf-mnist-with-summaries and enas-cnn-cifar10 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all trial images for amd64 except tf-mnist-with-summaries and enas-cnn-cifar10 can work on the M1 Mac since we can emulate x86_64 in docker desktop with Rosetta2.

Would you like to make all images conform to arm64 in this PR?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, we can fix later as well as it works now. We need to cleanup scripts for building images of different architecture.(rather than having arch checks per image etc)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense. @johnugeorge
It might be better to introduce a multi-arch build after the next Katib release.

ref: https://docs.docker.com/desktop/multi-arch/

I will create an issue to keep tracking this feature.

johnugeorge
johnugeorge reviewed Jun 15, 2022
View reviewed changes
examples/v1beta1/kind-cluster/deploy.sh
# If the local machine's CPU architecture is arm64, rewrite mysql image.
if [ "$(uname -m)" = "arm64" ]; then
kubectl patch deployments -n kubeflow katib-mysql --type json -p \
'[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value": "arm64v8/mysql:8.0.29-oracle"}]'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any better replacement solution with kustomize itself?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have only 3 requirement tools to work on this KinD cluster example, I added that code not to make increase requirement tools.

@johnugeorge
Copy link
Member

Ready to merge?

@tenzen-y
Copy link
Member Author

Ready to merge?

Sure!

@johnugeorge
Copy link
Member

Thanks

/lgtm
/approve

@google-oss-prow google-oss-prow bot added the lgtm label Jun 15, 2022
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 2c8758b into kubeflow:master Jun 15, 2022
@tenzen-y tenzen-y deleted the allow-to-run-examples-in-m1-mac branch June 15, 2022 05:50
@tenzen-y tenzen-y mentioned this pull request Jun 19, 2022
Closed
25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johnugeorge johnugeorge johnugeorge left review comments

@anencore94 anencore94 Awaiting requested review from anencore94

@sperlingxx sperlingxx Awaiting requested review from sperlingxx

Assignees

@johnugeorge johnugeorge

Labels
approved lgtm size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cannot run TensorFlow 2.7 in Docker on M1 (Apple Silicon)
3 participants
@tenzen-y @coveralls @johnugeorge