ML Extension 2.30.0 responds with: 'ml' is misspelled or not recognized by the system when triggered.

[yogeshg]

Describe the bug

ML Extension 2.30.0 responds with: 'ml' is misspelled or not recognized by the system when triggered.

Related command

$ az ml --help

Errors

'ml' is misspelled or not recognized by the system.

Issue script & Debug output

$ az extension add --name ml
Default enabled including preview versions for extension installation now. Disabled in May 2024. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only. 
 - Searching ..
Extension 'ml' 2.29.0 is already installed.
$ az extension update --name ml
Default enabled including preview versions for extension installation now. Disabled in May 2024. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only. 
 - Installing ..

$ az version
cannot import name 'AccessTokenInfo' from 'azure.core.credentials' (/opt/homebrew/Cellar/azure-cli/2.60.0/libexec/lib/python3.11/site-packages/azure/core/credentials.py)
{
  "azure-cli": "2.60.0",
  "azure-cli-core": "2.60.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "ml": "2.30.0",
    "storage-preview": "1.0.0b1"
  }
}
$ az ml --help
cannot import name 'AccessTokenInfo' from 'azure.core.credentials' (/opt/homebrew/Cellar/azure-cli/2.60.0/libexec/lib/python3.11/site-packages/azure/core/credentials.py)
'ml' is misspelled or not recognized by the system.

Expected behavior

It should return the error message.

Environment Summary

{
  "azure-cli": "2.60.0",
  "azure-cli-core": "2.60.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "ml": "2.30.0",
    "storage-preview": "1.0.0b1"
  }
}

I am using this in mac os. M3 machine

Additional context

No response

[azure-client-tools-bot-prd]

Hi @yogeshg,

2.60.0 is not the latest Azure CLI(2.64.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[github-actions]

Here are some similar issues that might help you. Please check if they can solve your problem.

• Azure ml extension ERROR: 'ml' is misspelled or not recognized by the system. when running from Azure DEVOPS with Agent poolself-hosted  #26954
• ERROR: 'ml' is misspelled or not recognized by the system. #22843
• az ml Broken on Installation #26295
• az ml : 'ml' is misspelled or not recognized by the system #21390
• az ml ... command in Dev Ops does not install extension #18258

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

[levu74]

I'm facing the same issue after upgrade ml extension to latest version

 {
    "experimental": false,
    "extensionType": "whl",
    "name": "ml",
    "path": "<removed>",
    "preview": false,
    "version": "2.30.0"
  }

[miladh]

Started to see same error since yesterday, even with older versions (2.23.0).

Created this ticket in azure-sdk-for-python (although it should probably have been created in this repo). Cross sharing for awareness: Azure/azure-sdk-for-python#37491

[AnOdyssey]

Since today we have started seeing the same issues. Both on windows machines and in our devops pipelines.

[MetelStairs]

Similar issue on our end, it was all working fine yesterday but now running into this issue.

[rikwatson]

Also seeing since today (2024-09-20)

[AndreasJ91]

Also seeing it since today (2024-09-20)

[alehpa]

Same here. My Azure devops pipelines are broken. Even if I specify previous version of ml 2.29.0. Found that pipeline was working with OS: Ubuntu 22.04.4 LTS. Now pool returns Ubuntu 22.04.5 LTS. Code of pipeline the same. The only thing environment changed. I have tried

pool:
    vmImage: ubuntu-20.04

Nothing changed - the same error.

/usr/bin/az --version
azure-cli                         2.64.0

core                              2.64.0
telemetry                          1.1.0

Extensions:
azure-devops                       1.0.1
ml                                2.29.0

Dependencies:
msal                              1.30.0
azure-mgmt-resource               23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/opt/az/azcliextensions'

Python (Linux) 3.11.8 (main, Aug 28 2024, 05:45:28) [GCC 9.4.0]

Legal docs and information: aka.ms/AzureCliLegal

/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1726823397662.sh
ERROR: 'ml' is misspelled or not recognized by the system.

Examples from AI knowledge base:
az extension add --name anextension
Add extension by name

az extension list-available
List all publicly available extensions

https://docs.microsoft.com/en-US/cli/azure/extension#az_extension_add
Read more about the command in reference docs

YAML code:

...
 - script: |
      az --version
      az extension list
      az extension add -n ml --version 2.29.0
      az ml -h
      az extension list
      az --version
    displayName: 'Install azure cli ML module'
...

[pvaneck]

It appears that there is some intermingling of dependencies. Do extension Python package dependencies consider the base azure-cli dependencies when installed?

Looking at where CLI extension dependencies are installed for me, (~/.azure/cliextensions/ml), I do see that there are compatible versions of azure-core and azure-identity installed. However, the versions of these packages installed in the azure-cli Python virtual environment (e.g. /opt/az/lib/python3.11/site-packages/) seem to take precedence at runtime causing incompatibilities.

I think something may need updating in the dependency resolution logic when installing an extension. Ideally, azure-cli extension Python requirements should remain compatible with the base azure-cli requirements when the extension is installed (i.e. azure-identity 1.17.1 should automatically have been installed with the ML extension).

The workaround is to install an older version of azure-identity in whatever virtual environment azure-cli is using.

Example workarounds:

Linux:

sudo /opt/az/bin/python3 -m pip install azure-identity==1.17.1

Windows (in elevated/administrator terminal):

C:\"Program Files (x86)"\"Microsoft SDKs"\Azure\CLI2\python.exe -m pip install azure-identity==1.17.1

Not sure about Mac, but probably just need to run the python executable somewhere in /opt/homebrew/Cellar/azure-cli/2.60.0/libexec/ ?

I think you can glean the location of the environment your azure-cli is using based on the error message:

cannot import name 'AccessTokenInfo' from 'azure.core.credentials' (/opt/az/lib/python3.11/site-packages/azure/core/credentials.py) -> Need to use python executable somewhere in /opt/az

[yogeshg]

Thanks @pvaneck I am trying out the workaround, and seems promising. I tried pinning the cli and the extension versions y'day, but was seeing the issue with older versions too, just like this report. Thanks for debugging and explaining the issue, now it makes sense why even the older versions would be broken. I hope you can address the root cause by updating the packaging strategy, perhaps syncing the release cadence of the Azure CLI and its extensions. Python dependency hell is well-named for a reason!

[kairu-ms]

It's resolved by Azure/azure-cli-extensions#8006

[jiasli]

However, the versions of these packages installed in the azure-cli Python virtual environment (e.g. /opt/az/lib/python3.11/site-packages/) seem to take precedence at runtime causing incompatibilities.

It is by design that packages installed by base Azure CLI take precedence. Previously, there was a PR #12778 to prioritize packages installed by extensions, but it was rejected as explained in #12778 (comment).

The base Azure CLI has no dependency on azure-identity (#19853 (comment)), but it depends on

azure-cli/src/azure-cli/requirements.py3.Linux.txt

Line 11 in 9f506a3

azure-core==1.28.0

We have to bump to azure-core>=1.31.0 (#29944) so that Azure CLI can be compatible with azure-identity>=1.18.0 (Azure/azure-sdk-for-python#36882).

[emibcn]

We have to bump to azure-core>=1.31.0 (#29944) so that Azure CLI can be compatible with azure-identity>=1.18.0 (Azure/azure-sdk-for-python#36882).

In Ubuntu, for doing this manually (until updated APT packages are publicly available), you can execute the commands published in #30209 (comment) :

sudo /opt/az/bin/python3 -m pip install \
    azure-core==1.31.0 \
    azure-identity==1.18.0