Error on az connectedk8s connect: cannot import name '_psutil_linux' from partially initialized module 'psutil' #18222

derkoe · 2021-05-25T19:35:17Z

Describe the bug

When running az connectedk8s connect I get an error on Linux (Fedora 34).

Command Name
az connectedk8s connect

Errors:

The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 556, in execute
    self.commands_loader.load_arguments(command)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 517, in load_arguments
    self.command_table[command].load_arguments()  # this loads the arguments via reflection
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 313, in load_arguments
    super(AzCliCommand, self).load_arguments()
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/commands.py", line 104, in load_arguments
    cmd_args = self.arguments_loader()
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
    op = self.get_op_handler(self.op_path)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
    handler = import_module(mod_to_import)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/koc/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 22, in <module>
    from psutil import process_iter, NoSuchProcess, AccessDenied, ZombieProcess, net_connections
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Install extension az extension add --name connectedk8s
Register K8s Cluster az connectedk8s connect

Expected Behavior

The command works.

Environment Summary

Linux-5.12.6-300.fc34.x86_64-x86_64-with-glibc2.33
Python 3.9.5
Installer: RPM

azure-cli 2.24.0

Extensions:
k8sconfiguration 0.1.8
spring-cloud 1.2.0
aks-preview 0.4.64
connectedk8s 1.1.5

Additional Context

The text was updated successfully, but these errors were encountered:

ghost · 2021-05-26T00:13:56Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @akashkeshari.

Issue Details

Describe the bug

When running az connectedk8s connect I get an error on Linux (Fedora 34).

Command Name
az connectedk8s connect

Errors:

The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 556, in execute
    self.commands_loader.load_arguments(command)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 517, in load_arguments
    self.command_table[command].load_arguments()  # this loads the arguments via reflection
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 313, in load_arguments
    super(AzCliCommand, self).load_arguments()
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/commands.py", line 104, in load_arguments
    cmd_args = self.arguments_loader()
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
    op = self.get_op_handler(self.op_path)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
    handler = import_module(mod_to_import)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/koc/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 22, in <module>
    from psutil import process_iter, NoSuchProcess, AccessDenied, ZombieProcess, net_connections
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Install extension az extension add --name connectedk8s
Register K8s Cluster az connectedk8s connect

Expected Behavior

The command works.

Environment Summary

Linux-5.12.6-300.fc34.x86_64-x86_64-with-glibc2.33
Python 3.9.5
Installer: RPM

azure-cli 2.24.0

Extensions:
k8sconfiguration 0.1.8
spring-cloud 1.2.0
aks-preview 0.4.64
connectedk8s 1.1.5

Additional Context

Author:	derkoe
Assignees:	-
Labels:	`Connected Kubernetes`, `Service Attention`
Milestone:	-

yonzhan · 2021-05-26T00:14:03Z

route to service team

redhatstuart · 2021-05-28T03:52:38Z

Linux stkirk-fedora 5.12.6-300.fc34.x86_64 #1 SMP Sat May 22 20:42:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Fedora release 34 (Thirty Four)
azure-cli 2.24.0
core 2.24.0
telemetry 1.0.6

Extensions:
k8s-extension-private 0.2.1b2
connectedk8s 1.1.5

I have also experienced the exact same behavior as noted above. This issue does not seem to be present on debian/ubuntu type installs of the AZCLI. I have tried this with Fedora 33, Fedora 34 and RHEL 8 (all RPM based distributions) and the process fails each time:

[stkirk@stkirk-fedora aro4]$ az connectedk8s connect --name $arcName --resource-group $aroRG --location $arcLocation
The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/init.py)
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 556, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/init.py", line 517, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 313, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python3.6/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/usr/lib64/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 855, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/stkirk/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 22, in
from psutil import process_iter, NoSuchProcess, AccessDenied, ZombieProcess, net_connections
File "/usr/lib64/az/lib/python3.6/site-packages/psutil/init.py", line 99, in
from . import _pslinux as _psplatform
File "/usr/lib64/az/lib/python3.6/site-packages/psutil/_pslinux.py", line 26, in
from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/init.py)

derkoe · 2021-05-31T06:02:34Z

I have found a workaround:

In /usr/bin/az replace the python command with pythond3.6 (and then re-install the connectedk8s extension):

3c3
< AZ_INSTALLER=RPM PYTHONPATH="$bin_dir"/../lib64/az/lib/python3.6/site-packages python3 -sm azure.cli "$@"
---
> AZ_INSTALLER=RPM PYTHONPATH="$bin_dir"/../lib64/az/lib/python3.6/site-packages python3.6 -sm azure.cli "$@"

It seems that the RPM packages uses Python 3.6 but the python3 command points to 3.9. This results in the wrong packages to be installed in the extension.

redhatstuart · 2021-06-01T03:10:06Z

Confirmed this works. I had a feeling that this was the issue given the higher rev of python. I tried using "alternatives" to get Fedora to set the default python to 3.6 however that didn't seem to work so was at a loss. cc: @yonzhan

jiasli · 2021-06-02T10:02:31Z

@derkoe, you analysis helps a lot.

If you extract the rpm package azure-cli-2.24.0-1.el7.x86_64.rpm (downloadable from https://packages.microsoft.com/yumrepos/azure-cli/) and navigate to usr/lib64/az/lib/python3.6/site-packages/psutil, you will see only 2 so files:

_psutil_linux.cpython-36m-x86_64-linux-gnu.so
_psutil_posix.cpython-36m-x86_64-linux-gnu.so

This means psutil in Azure CLI only works for Python 3.6.

This is because Azure CLI is built using centos7:

azure-cli/scripts/release/rpm/Dockerfile.centos

Line 1 in f2855b8

ARG tag=centos7

Only python 3.6 is available in centos7.

I haven't tested Fedora 34. Could you verify if Python 3.6 exists on any Fedora 34 installation? If so we can fix it by using Python 3.6 for now:

azure-cli/scripts/release/rpm/azure-cli.spec

Line 57 in 753a11c

    
           printf "#!/usr/bin/env bash\nbin_dir=\`cd \"\$(dirname \"\$BASH_SOURCE[0]\")\"; pwd\`\nAZ_INSTALLER=RPM PYTHONPATH=\"\$bin_dir\"/../%{_lib}/az/lib/${python_version}/site-packages %{python_cmd} -sm azure.cli \"\$@\"" > %{buildroot}%{_bindir}/az

derkoe · 2021-06-02T11:19:41Z

Fedora 34 defaults to Python 3.9 but Python 3.6 is also available (as /usr/bin/python3.6).

derkoe · 2021-06-02T14:26:14Z

If you have a Docker (or another container runtime) you can do:

$ docker run -it --rm fedora:34

[root@d6bd91249557 /]# echo -e "[azure-cli]
name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc" |  tee /etc/yum.repos.d/azure-cli.repo
[root@d6bd91249557 /]# dnf install azure-cli
[root@d6bd91249557 /]# az connectedk8s connect
The command requires the extension connectedk8s. Do you want to install it now? The command will continue to run after the extension is installed. (Y/n): y
Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.
The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)
...

jiasli · 2021-06-03T01:48:36Z

@derkoe, thanks for the info. We will join the the Kick-off event recently. We will look at it after we return.

alphaWizard · 2021-07-28T06:53:36Z

@jiasli Any update on this issue?

jiasli · 2021-07-28T10:53:17Z

@derkoe, I am able to repro this issue on fedora:34 following Install the Azure CLI on Linux. However, fedora:34 doesn't seem to have python 3.6 pre-installed:

> docker run -it fedora:34

# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root     9 Jul 16 13:48 /usr/bin/python3 -> python3.9
-rwxr-xr-x 1 root root 15632 Jul 16 13:48 /usr/bin/python3.9

We got similar issues with the cffi library. Explanation and workarounds are provided in #18159 (comment).

Workarounds

In this case for psutil, you may either

Reinstall `psutil` with `pip`

dnf install python3-pip
pip3 install psutil --force-reinstall --upgrade --target /usr/lib64/az/lib/python3.6/site-packages/

Install Azure CLI with `pip`

dnf install python3-pip
pip3 install azure-cli

derkoe · 2021-07-28T20:47:57Z

Thx @jiasli this workaround also works.

Anyway, I think this should be fixed in the RPM package.

jiasli · 2021-07-29T02:41:57Z

@derkoe, exactly. However, I don't think this will be an easy fix, as we need to release different RPM packages for different versions of Python installed on the system, due to different dependencies on so files, such as:

Python 3.6: _psutil_posix.cpython-36m-x86_64-linux-gnu.so
Python 3.7: _psutil_posix.cpython-37m-x86_64-linux-gnu.so
Python 3.8: _psutil_posix.cpython-38m-x86_64-linux-gnu.so
Python 3.9: _psutil_posix.cpython-39m-x86_64-linux-gnu.so

Then we need to write an installation script to install corresponding RPM packages according to the output of python3 --version.

Another solution: Currently for Windows and Ubuntu, we bundle an embedded Python within the MSI and deb installer, perhaps we can adopt the same approach for RPM.

derkoe · 2021-07-29T08:05:17Z

In the old version you could have also fixed that in the shell script by replacing "python" with "python3.6".

I guess in the current version you could patch the python version here https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/az#L1 ("python" -> "python3.6")

jiasli · 2021-07-29T08:37:39Z

In the old version you could have also fixed that in the shell script by replacing "python" with "python3.6".

Per my observation, Fedora:34 doesn't seem to have python 3.6 pre-installed:

> docker run -it fedora:34

# ls -l /usr/bin/python*
lrwxrwxrwx 1 root root     9 Jul 16 13:48 /usr/bin/python3 -> python3.9
-rwxr-xr-x 1 root root 15632 Jul 16 13:48 /usr/bin/python3.9

derkoe · 2021-07-29T10:07:12Z

Per my observation, Fedora:34 doesn't seem to have python 3.6 pre-installed:

When you install Azure CLI RPM the 3.6 is installed as well (it is a dependency).

jiasli · 2021-07-29T12:01:29Z

Aha, I forgot about that. Thanks @derkoe. I got confused by --nodeps from #18159 (comment).

Let me create a PR to fix azure-cli.spec!

azure-cli/scripts/release/rpm/azure-cli.spec

Line 57 in 753a11c

    
           printf "#!/usr/bin/env bash\nbin_dir=\`cd \"\$(dirname \"\$BASH_SOURCE[0]\")\"; pwd\`\nAZ_INSTALLER=RPM PYTHONPATH=\"\$bin_dir\"/../%{_lib}/az/lib/${python_version}/site-packages %{python_cmd} -sm azure.cli \"\$@\"" > %{buildroot}%{_bindir}/az

However, azure-cli/src/azure-cli/az file is meant to be the entry point when installed with pip:

azure-cli/src/azure-cli/az

Lines 1 to 19 in abc5125

    
           #!/usr/bin/env python 
        
           import sys 
        
           import os 
        
           dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'src') 
        
           if os.environ.get('PYTHONPATH') is None: 
        
               os.environ['PYTHONPATH'] = dir 
        
           else: 
        
               os.environ['PYTHONPATH'] = os.pathsep.join([ 
        
                   dir, 
        
                   os.environ.get('PYTHONPATH'), 
        
               ]) 
        
           if os.environ.get('AZ_INSTALLER') is None: 
        
               os.environ['AZ_INSTALLER'] = 'PIP' 
        
           os.execl(sys.executable, sys.executable, '-m', 'azure.cli', *sys.argv[1:])

So it will be compatible with the python on your PATH.

ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels May 25, 2021

yonzhan added the Connected Kubernetes label May 26, 2021

ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label May 26, 2021

yonzhan added Service Attention This issue is responsible by Azure service team. and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels May 26, 2021

jiasli self-assigned this Jul 28, 2021

jiasli added Packaging/RPM and removed Connected Kubernetes Service Attention This issue is responsible by Azure service team. labels Jul 28, 2021

jiasli added this to the Backlog milestone Jul 28, 2021

yonzhan added the feature-request label Jul 28, 2021

jiasli mentioned this issue Aug 3, 2021

[Packaging] When installed with RPM, run python3.6 if available #19110

Merged

jiasli mentioned this issue Aug 24, 2021

{Core} Check return value of get_parent_proc_name #19325

Merged

jiasli closed this as completed in #19110 Aug 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error on az connectedk8s connect: cannot import name '_psutil_linux' from partially initialized module 'psutil' #18222

Error on az connectedk8s connect: cannot import name '_psutil_linux' from partially initialized module 'psutil' #18222

derkoe commented May 25, 2021 •

edited

Loading

ghost commented May 26, 2021

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

yonzhan commented May 26, 2021

redhatstuart commented May 28, 2021

derkoe commented May 31, 2021 •

edited by jiasli

Loading

redhatstuart commented Jun 1, 2021

jiasli commented Jun 2, 2021

derkoe commented Jun 2, 2021

derkoe commented Jun 2, 2021

jiasli commented Jun 3, 2021

alphaWizard commented Jul 28, 2021

jiasli commented Jul 28, 2021 •

edited

Loading

derkoe commented Jul 28, 2021

jiasli commented Jul 29, 2021

derkoe commented Jul 29, 2021

jiasli commented Jul 29, 2021

derkoe commented Jul 29, 2021

jiasli commented Jul 29, 2021

Error on az connectedk8s connect: cannot import name '_psutil_linux' from partially initialized module 'psutil' #18222

Error on az connectedk8s connect: cannot import name '_psutil_linux' from partially initialized module 'psutil' #18222

Comments

derkoe commented May 25, 2021 • edited Loading

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

ghost commented May 26, 2021

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

yonzhan commented May 26, 2021

redhatstuart commented May 28, 2021

derkoe commented May 31, 2021 • edited by jiasli Loading

redhatstuart commented Jun 1, 2021

jiasli commented Jun 2, 2021

derkoe commented Jun 2, 2021

derkoe commented Jun 2, 2021

jiasli commented Jun 3, 2021

alphaWizard commented Jul 28, 2021

jiasli commented Jul 28, 2021 • edited Loading

Workarounds

Reinstall psutil with pip

Install Azure CLI with pip

derkoe commented Jul 28, 2021

jiasli commented Jul 29, 2021

derkoe commented Jul 29, 2021

jiasli commented Jul 29, 2021

derkoe commented Jul 29, 2021

jiasli commented Jul 29, 2021

derkoe commented May 25, 2021 •

edited

Loading

derkoe commented May 31, 2021 •

edited by jiasli

Loading

jiasli commented Jul 28, 2021 •

edited

Loading

Reinstall `psutil` with `pip`

Install Azure CLI with `pip`