Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure CLI 2.51.0 cannot be released to Homebrew #27047

Closed
jiasli opened this issue Aug 1, 2023 · 20 comments
Closed

Azure CLI 2.51.0 cannot be released to Homebrew #27047

jiasli opened this issue Aug 1, 2023 · 20 comments
Assignees
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. Installation
Milestone

Comments

@jiasli
Copy link
Member

jiasli commented Aug 1, 2023

Latest pip fails to install azure-mgmt-consumption 2.0.0 from source code https://pypi.org/project/azure-mgmt-consumption/2.0.0/#files

This make homebrew fail to install Azure CLI (Homebrew/homebrew-core#138158):

https://github.com/Homebrew/homebrew-core/actions/runs/5722058178/job/15504679804?pr=138158

2023-08-01T03:12:33.5413100Z   creating '/private/tmp/pip-modern-metadata-vjgv3zcx/azure_mgmt_consumption-2.0.0.dist-info'
2023-08-01T03:12:33.5413250Z   error: error in setup.cfg: command 'bdist_wheel' has no such option 'azure_namespace_package'
2023-08-01T03:12:33.5413340Z   error: subprocess-exited-with-error

To repro locally:

> pip uninstall -y wheel
> pip install https://files.pythonhosted.org/packages/8c/f0/e2d94b246e2dce71eff8d362836a1979f02b4185f5403a13e4fb26c07ccb/azure-mgmt-consumption-2.0.0.zip
Collecting https://files.pythonhosted.org/packages/8c/f0/e2d94b246e2dce71eff8d362836a1979f02b4185f5403a13e4fb26c07ccb/azure-mgmt-consumption-2.0.0.zip
  Using cached azure-mgmt-consumption-2.0.0.zip (56 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      Wheel is not available, disabling bdist_wheel hook
      running dist_info
      creating C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info
      writing C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\PKG-INFO
      writing dependency_links to C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\dependency_links.txt
      writing requirements to C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\requires.txt
      writing top-level names to C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\top_level.txt
      writing manifest file 'C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\SOURCES.txt'
      reading manifest file 'C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      writing manifest file 'C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption.egg-info\SOURCES.txt'
      creating 'C:\Users\xxx\AppData\Local\Temp\pip-modern-metadata-v6_4olsi\azure_mgmt_consumption-2.0.0.dist-info'
      error: error in setup.cfg: command 'bdist_wheel' has no such option 'azure_namespace_package'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
@jiasli jiasli added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Aug 1, 2023
@ghost ghost added Auto-Assign Auto assign by bot Installation labels Aug 1, 2023
@ghost ghost assigned jiasli Aug 1, 2023
@ghost ghost added this to the Backlog milestone Aug 1, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Aug 1, 2023

Thank you for opening this issue, we will look into it.

@jiasli
Copy link
Member Author

jiasli commented Aug 1, 2023

@jsntcy, It seems azure-mgmt-consumption 2.0.0 is still Track 1. Can we migrate it to Track 2?

@bebound
Copy link
Contributor

bebound commented Aug 1, 2023

I guess this is related to changes in pip 23.1:

When the wheel package is not installed, pip now uses the default build backend instead of setup.py install and setup.py develop for project without pyproject.toml. (#8559) -- https://pip.pypa.io/en/stable/news/#v23-1

Another helpful reply about the error: #14416 (comment)
In short, pip calls python setup.py bdist_wheel instead of python setup.py install during installation in new version, but azure-mgmt-consumption 2.0 is not compatible with latest wheel when running python setup.py bdist_wheel

python -m pip install ./azure-mgmt-consumption-2.0.0 --no-deps
Processing ./azure-mgmt-consumption-2.0.0
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: azure-mgmt-consumption
  Building wheel for azure-mgmt-consumption (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Wheel is not available, disabling bdist_wheel hook
      running bdist_wheel
      error: error in setup.cfg: command 'bdist_wheel' has no such option 'azure_namespace_package'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for azure-mgmt-consumption
  Running setup.py clean for azure-mgmt-consumption
Failed to build azure-mgmt-consumption
ERROR: Could not build wheels for azure-mgmt-consumption, which is required to install pyproject.toml-based projects

@jiasli jiasli changed the title pip fails to install azure-mgmt-consumption 2.0.0 from source code Azure CLI 2.51.0 cannot be released to Homebrew Aug 1, 2023
@jiasli jiasli pinned this issue Aug 1, 2023
@jiasli
Copy link
Member Author

jiasli commented Aug 1, 2023

https://github.com/Azure/azure-cli/blob/dev/doc/track_2_migration_roadmap.md keeps track of the Track 2 SDK migration progress. To verify the current status, I wrote the below script to print all Track 1 SDKs Azure CLI is still using:

import os
from importlib import import_module

import azure.mgmt

azure_mgmt_dir = azure.mgmt.__path__[0]

print("""| sdk | version | pypi |
| - | - | - |""")

track_1_sdks = {}
for sdk in os.listdir(azure_mgmt_dir):
    sdk_path = os.path.join(azure_mgmt_dir, sdk)
    # Track 1 SDK contains version.py; Track 2 SDK contains _version.py
    if os.path.isdir(sdk_path) and 'version.py' in os.listdir(sdk_path):
        sdk_module_name = f'azure.mgmt.{sdk}'
        sdk_module = import_module(sdk_module_name)
        version = sdk_module.version.VERSION
        package_name = f'azure-mgmt-{sdk}'
        pypi_link = f'https://pypi.org/project/{package_name}/{version}/#files'
        print(f'| {sdk_module_name} | {version} | {pypi_link} |')

Result:

sdk version pypi contains azure_bdist_wheel.py
azure.mgmt.consumption 2.0.0 https://pypi.org/project/azure-mgmt-consumption/2.0.0/#files ✔️
azure.mgmt.devtestlabs 4.0.0 https://pypi.org/project/azure-mgmt-devtestlabs/4.0.0/#files
azure.mgmt.kusto 0.3.0 https://pypi.org/project/azure-mgmt-kusto/0.3.0/#files
azure.mgmt.managedservices 1.0.0 https://pypi.org/project/azure-mgmt-managedservices/1.0.0/#files
azure.mgmt.relay 0.1.0 https://pypi.org/project/azure-mgmt-relay/0.1.0/#files ✔️
azure.mgmt.datalake.analytics* 0.2.1 https://pypi.org/project/azure-mgmt-datalake-analytics/0.2.1/#files ✔️

* azure.mgmt.datalake.analytics is manually discoverd, #27163 (comment) explains why.

Additional information

Using version.py and _version.py to detect Track 1 or Track 2 SDK is inspired by

# Add azure-mgmt-resource version
try:
# Track 2 >=15.0.0
# pylint: disable=protected-access
from azure.mgmt.resource._version import VERSION as azure_mgmt_resource_version
except ImportError:
try:
# Track 1 <=13.0.0
from azure.mgmt.resource.version import VERSION as azure_mgmt_resource_version
except ImportError:
azure_mgmt_resource_version = "N/A"

The detection method Azure CLI core uses is too complex in this case:

def is_track2(client_class):
""" IS this client a autorestv3/track2 one?.
Could be refined later if necessary.
"""
from inspect import getfullargspec as get_arg_spec
args = get_arg_spec(client_class.__init__).args
return "credential" in args

@jiasli
Copy link
Member Author

jiasli commented Aug 3, 2023

#25452 contains detailed explanation for breaking changes discovered during one of the most complex Track 2 SDK migrations - azure-mgmt-authorization. If none or only a few of these breaking changes are hit by consumption and relay command modules, the effort for migrating to Track 2 SDK is rather small, as we only need to bump the SDK and re-record all tests.

We also need to evaluate other options such as using code-gen to regenerate these modules.

@chenrui333
Copy link
Contributor

I closed the bot PR, once the issue got sorted out, feel free to open a new one.

@bebound
Copy link
Contributor

bebound commented Aug 14, 2023

azure-mgmt-datalake-analytics 0.2.1 also fails with same error.


I tried to remove those three packages. The modified formula can be installed. But fails to run:
azure-cli.txt

linuxbrew@794b47da48b2:/down$ az -v
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.51.0/libexec/lib/python3.10/site-packages/azure/__init__.py", line 1, in <module>
    __import__('pkg_resources').declare_namespace(__name__)
ModuleNotFoundError: No module named 'pkg_resources'

Run python3.10 -m pip --python=/home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.51.0/libexec/bin/python install setuptools can fix the issue.

If I use --build-from-source, /home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.51.0/libexec/lib/python3.10/site-packages/ does not contains pip and setuptools but 2.50 contains:

ls /home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.50.0_1/libexec/lib/python3.10/site-packages/ | grep setup
setuptools
setuptools-67.6.1.dist-info
ls /home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.50.0_1/libexec/lib/python3.10/site-packages/ | grep pip
pip
pip-23.0.1.dist-info

@bebound
Copy link
Contributor

bebound commented Aug 15, 2023

@chenrui333 If I install this formula with brew install azure-cli.rb --build-from-source, setuptools and pip are not installed automatically, do you have any idea? Thank you.
azure-cli.rb

@bebound
Copy link
Contributor

bebound commented Aug 17, 2023

I see, it's caused by Homebrew/brew#15792.
We need to add without_pip: false in formula. I'll fix it.

@branchvincent
Copy link

Yea without_pip: false is a good short-term workaround. Long term, I'd recommend either

  1. Explicitly depending on setupools (needed anyway for python 3.12+ Remove bundled setuptools python/cpython#95299)
  2. If you just need to declare a namespace package, use an implicit namespace

@bebound
Copy link
Contributor

bebound commented Aug 17, 2023

@branchvincent Thanks for your helpful reminder.
Both setuptools and pip are required in azure-cli.

@chenrui333

This comment was marked as off-topic.

@chenrui333
Copy link
Contributor

@bebound looks like azure-mgmt-consumption 3.0.0 has already removed azure_bdist_wheel, why not bump to that dependency? (I have done the pip install for that sdist, which works fine in my local)

@bebound
Copy link
Contributor

bebound commented Aug 22, 2023

azure-mgmt-relay and azure-mgmt-consumption are not removed yet.
We'll migrate to aaz to removed them.

@scanyard
Copy link

scanyard commented Sep 4, 2023

This issue hasn't been updated in a bit - the build of azure-cli available in homebrew is still 2.50. Is this close to being fixed?

@bebound
Copy link
Contributor

bebound commented Sep 5, 2023

Close as 2.52.0 is available in Homebrew: Homebrew/homebrew-core#141437

@chenrui333
Copy link
Contributor

@bebound this can also be removed from the pinned issue list?

@bebound bebound unpinned this issue Oct 25, 2023
@bebound
Copy link
Contributor

bebound commented Jun 26, 2024

azure-mgmt-datalake-store==0.5.0 makes formula fail to build again and the error is also no such option 'azure_namespace_package'

Here is the output of brew install --build-from-source azure-cli:

==> python3.11 -m pip --python=/usr/local/Cellar/azure-cli/2.61.0/libexec/bin/python install /tmp/azure-cli--azure-mgmt-datalake-store-20240627-18659-nu6r0p/azure-mgmt-datalake-store-0.5.0
Last 15 lines from /Users/runner/Library/Logs/Homebrew/azure-cli/41.python3.11:
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/Cellar/azure-cli/2.61.0/libexec/bin/python /usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /private/tmp/tmp8t6xbm2g
  cwd: /private/tmp/azure-cli--azure-mgmt-datalake-store-20240626-12379-yfbs8f/azure-mgmt-datalake-store-0.5.0
  Preparing metadata (pyproject.toml): finished with status 'error'
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: python -m pip install --upgrade pip

Do not report this issue to Homebrew/brew or Homebrew/homebrew-core!

Log: https://dev.azure.com/azclitools/public/_build/results?buildId=167640&view=logs&jobId=ebe8970d-a8af-5d7e-4086-8f4ce0be006b&j=ebe8970d-a8af-5d7e-4086-8f4ce0be006b&t=3f22f999-770c-5d06-3bb8-1a936f5d8384

In 2.51.0, lots of Track 1 SDK are not compatible with latest wheel and fails to install. This package is also Track 1, but is compatible with latest wheel, so we don't migrate it.

I can repro the similar error in python:3.11-slim image after removing wheel:

pip uninstall -y wheel
pip install  https://files.pythonhosted.org/packages/00/13/037f0128bdfcd47253f69a3b4ca6a7ff7b673b35832bc48f7c74df24a9be/azure-mgmt-datalake-store-0.5.0.zip --no-deps
pip install  https://files.pythonhosted.org/packages/00/13/037f0128bdfcd47253f69a3b4ca6a7ff7b673b35832bc48f7c74df24a9be/azure-mgmt-datalake-store-0.5.0.zip --no-deps
Collecting https://files.pythonhosted.org/packages/00/13/037f0128bdfcd47253f69a3b4ca6a7ff7b673b35832bc48f7c74df24a9be/azure-mgmt-datalake-store-0.5.0.zip
  Downloading azure-mgmt-datalake-store-0.5.0.zip (101 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.3/101.3 kB 394.7 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      Wheel is not available, disabling bdist_wheel hook
      /tmp/pip-build-env-eodfhzkp/overlay/lib/python3.11/site-packages/setuptools/dist.py:478: SetuptoolsDeprecationWarning: Invalid dash-separated options
      !!

              ********************************************************************************
              Usage of dash-separated 'azure-namespace-package' will not be supported in future
              versions. Please use the underscore name 'azure_namespace_package' instead.

              By 2024-Sep-26, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        opt = self.warn_dash_deprecation(opt, section)
      running dist_info
      creating /tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info
      writing /tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/requires.txt
      writing top-level names to /tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      writing manifest file '/tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store.egg-info/SOURCES.txt'
      creating '/tmp/pip-modern-metadata-je37iaya/azure_mgmt_datalake_store-0.5.0.dist-info'
      error: error in setup.cfg: command 'bdist_wheel' has no such option 'azure_namespace_package'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I have no idea why it worked before and fails now.
Maybe there are some changes in brew or setuptools?

AFAIK, latest setuptools now has a vendored wheel 0.4.3.

@bebound
Copy link
Contributor

bebound commented Jun 26, 2024

@chenrui333 Are there any changes in brew that relate to my previous comment?

@bebound
Copy link
Contributor

bebound commented Jun 28, 2024

I find the cause: the change in setuptools 70.1: pypa/setuptools@b3d3e93#diff-0480ca57cef22161092786457725dbe93e08344a9d8fb3bb0c1b68d19d5ea395R325 (PR: pypa/setuptools#4369)

When building wheel from source code, setuptools craetes an isolated build environment. Previously, it installs wheel because wheel isrequired in python setup.py bdist_wheel command.
In 70.1, setuptools has a vendor wheel, and it removes wheel from requirement.
However, azure-mgmt-datalake-store==0.5.0 still requires wheel in https://github.com/Azure/azure-sdk-for-python/blob/azure-mgmt-datalake-analytics_0.5.0/azure-mgmt-datalake-store/azure_bdist_wheel.py#L10. So it fails to build.

This change in setuptools also make us pin 70.0 in azure-cli-extension to keep using wheel 0.3.0: Azure/azure-cli-extensions#7739


Here are some old technologies we're using:
Track 1 SDK

azure-mgmt-kusto==0.3.0
azure-mgmt-devtestlabs==4.0.0
azure-graphrbac==0.60.0

azure-mgmt-managedservices~=1.0 in setup.py

wheel 0.30.0
Used in azure-cli-extension: Azure/azure-cli-extensions#6842

Legacy PEP420
It uses setuptools instead of find_namespace_pacakge:

__path__ = __import__("pkgutil").extend_path(__path__, __name__)

We need to add setuptools to dependency in 3.12: #27196

No pyproject.toml
https://setuptools.pypa.io/en/latest/build_meta.html#

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. Installation
Projects
None yet
Development

No branches or pull requests

6 participants