ImportError: cannot import name shlex_quote

[gergnz]

Looks like this (6a352c0) commit breaks aws cli on RHEL6 at least:

Traceback (most recent call last):
  File "/usr/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/usr/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/usr/lib/python2.6/site-packages/awscli/clidriver.py", line 49, in main
    driver = create_clidriver()
  File "/usr/lib/python2.6/site-packages/awscli/clidriver.py", line 58, in create_clidriver
    event_hooks=emitter)
  File "/usr/lib/python2.6/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/lib/python2.6/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/lib/python2.6/site-packages/awscli/handlers.py", line 61, in <module>
    from awscli.customizations.opsworks import initialize as opsworks_init
  File "/usr/lib/python2.6/site-packages/awscli/customizations/opsworks.py", line 24, in <module>
    from six.moves import shlex_quote
ImportError: cannot import name shlex_quote

[gergnz]

Looks like we need six > 1.8.0. (https://mail.python.org/pipermail/python-porting/2014-September/000449.html)

[root@ip-10-128-176-244 ~]# pip list | grep six
six (1.6.1)
[root@ip-10-128-176-244 ~]# pip install -U six
Downloading/unpacking six from https://pypi.python.org/packages/source/s/six/six-1.8.0.tar.gz#md5=1626eb24cc889110c38f7e786ec69885
  Downloading six-1.8.0.tar.gz
  Running setup.py egg_info for package six
    no previously-included directories found matching 'documentation/_build'
Installing collected packages: six
  Found existing installation: six 1.6.1
    Uninstalling six:
      Successfully uninstalled six
  Running setup.py install for six
    no previously-included directories found matching 'documentation/_build'
Successfully installed six
Cleaning up...
[root@ip-10-128-176-244 ~]# pip list | grep six
six (1.8.0)
[root@ip-10-128-176-244 ~]# aws
usage: aws [options] <command> <subcommand> [parameters]
aws: error: too few arguments

[bok-]

This is still happening on OS X 10.10.1 with six 1.8.0:

sh-3.2# pip list | grep six
six (1.8.0)
sh-3.2# aws
Traceback (most recent call last):
  File "/usr/local/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/usr/local/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 49, in main
    driver = create_clidriver()
  File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 58, in create_clidriver
    event_hooks=emitter)
  File "/Library/Python/2.7/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/Library/Python/2.7/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/Library/Python/2.7/site-packages/awscli/handlers.py", line 61, in <module>
    from awscli.customizations.opsworks import initialize as opsworks_init
  File "/Library/Python/2.7/site-packages/awscli/customizations/opsworks.py", line 24, in <module>
    from six.moves import shlex_quote
ImportError: cannot import name shlex_quote

Downgrading to aswcli 1.6.6 works for now.

[joeymink]

Seeing this on AWS CentOS7 as well (awscli v1.6.7). Also confirmed that downgrading to v1.6.6 solves the problem.

[frielp]

Also seeing this on Ubuntu 14.04, downgrade to v1.6.6 fixes on it as well.

[jamesls]

I'm trying to repro this. Can you share how you're installing the AWS CLI? I think we can bump the min version of six in the AWS CLI to 1.8.0, but the fact that botocore (a dependency) requires six >=1.8.0 means that pip should pull in this newer version of six automatically.

Install previous version in a fresh venv:

$ pip install awscli==1.6.6
$ aws --version
aws-cli/1.6.6 Python/2.7.7 Darwin/13.4.0
$ aws ec2 describe-instances --region eu-central-1
{
    "Reservations": []
}

Now upgrade to the latest version (1.6.7):

$ pip install --upgrade awscli
$ aws --version
aws-cli/1.6.7 Python/2.7.7 Darwin/13.4.0
$ aws ec2 describe-instances --region eu-central-1
{
    "Reservations": []
}

[3ch01c]

@jamesls does this help?

$ sudo -E pip install awscli==1.6.6
$ aws --version
aws-cli/1.6.6 Python/2.7.6 Darwin/14.0.0
$ sudo -E pip install --upgrade awscli
$ aws --version
Traceback (most recent call last):
  File "/usr/local/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/usr/local/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 49, in main
    driver = create_clidriver()
  File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 58, in create_clidriver
    event_hooks=emitter)
  File "/Library/Python/2.7/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/Library/Python/2.7/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/Library/Python/2.7/site-packages/awscli/handlers.py", line 61, in <module>
    from awscli.customizations.opsworks import initialize as opsworks_init
  File "/Library/Python/2.7/site-packages/awscli/customizations/opsworks.py", line 24, in <module>
    from six.moves import shlex_quote
ImportError: cannot import name shlex_quote

[kyleknap]

Here is the repo steps:

$ virtualenv test167
$ . test167/bin/activate
$ pip install six==1.1.0
$ pip install awscli

$ aws --version
Traceback (most recent call last):
  File "/Users/kyleknap/test167/bin/aws", line 23, in <module>
    sys.exit(main())
  File "/Users/kyleknap/test167/bin/aws", line 19, in main
    return awscli.clidriver.main()
  File "/Users/kyleknap/test167/lib/python2.7/site-packages/awscli/clidriver.py", line 49, in main
    driver = create_clidriver()
  File "/Users/kyleknap/test167/lib/python2.7/site-packages/awscli/clidriver.py", line 58, in create_clidriver
    event_hooks=emitter)
  File "/Users/kyleknap/test167/lib/python2.7/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/Users/kyleknap/test167/lib/python2.7/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/Users/kyleknap/test167/lib/python2.7/site-packages/awscli/handlers.py", line 61, in <module>
    from awscli.customizations.opsworks import initialize as opsworks_init
  File "/Users/kyleknap/test167/lib/python2.7/site-packages/awscli/customizations/opsworks.py", line 24, in <module>
    from six.moves import shlex_quote
ImportError: cannot import name shlex_quote

Any other ways to repo this would be great so we can check all of these when we make the fix.

[kyleknap]

Thanks for all of the help here!

We have a pull request merged that should fix it:
#1054

We are going to try to get a release out soon to patch this issue. Closing for now.
Once we get out the new version, please upgrade and reopen the issue if still does not work.

[jamesls]

Version 1.6.8 is now available and should fix this issue.

[gmllama]

I had to uninstall six and manually "pip install six==1.8.0" to stop the shlex_quote error

[ajkerr]

Version 1.6.8 still gives the shlex_quote error for me on OS X 10.10.1, even with six 1.8.0 installed. Reverting to 1.6.6 solves it for me.

[bok-]

Yep, 1.6.8 has not solved the issue on OS X 10.10.1 for me either. Back to 1.6.6 is fine.

[denismo]

Same here, Ubuntu 14.04.1, problem is not solved with six 1.8.0, Reverting to 1.6.6 helped.

[jamesls]

Taking a look. I don't have access to a 10.10.1 machine, but I'll try this on an ubuntu 14.04.1 instance and see what I can find. From what I can tell I'm guessing that six is not being upgraded to 1.8.0 despite the fact we call out this requirements in our setup.py.

How are you upgrading? Are you running pip install --upgrade awscli.

I'll see what I can find on an ubuntu machine.

[jamesls]

I'm still not able to repro this issue, at least on Ubuntu 14.04.

On OS X 10.10.1 and Ubuntu, it looks like there are reports of six==1.8.0 being installed, but not being able to shlex_quote. For users seeing that problem, could you try the following to help debug:

>>> import six
>>> print six.__version__
1.8.0
>>> print six.moves.shlex_quote
<function quote at 0x7fabd3d645f0>
>>> print six.__file__
/usr/local/lib/python2.7/dist-packages/six.pyc

Ideally if you have six==1.8.0 installed, you should see similar output, with the exception of the last line which prints the file path to where the six module is installed which may vary.

[ajkerr]

I think you're on to something. Here's what I'm getting on OS X 10.10.1:

Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> print six.__version__
1.4.1
>>> print six.moves.shlex_quote
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: '_MovedItems' object has no attribute 'shlex_quote'
>>> print six.__file__
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc

However:

$ pip show six
---
Name: six
Version: 1.8.0
Location: /Library/Python/2.7/site-packages
Requires: 

[jamesls]

Also, to help with additional debugging, here's what I tried on a fresh EC2 instance running ubuntu 14.04:

$ sudo apt-get update
$ sudo apt-get install python-pip
$ pip list

argparse (1.2.1)
chardet (2.0.1)
Cheetah (2.4.4)
cloud-init (0.7.5)
colorama (0.2.5)
configobj (4.7.2)
html5lib (0.999)
jsonpatch (1.3)
jsonpointer (1.0)
Landscape-Client (14.01)
oauth (1.0.1)
PAM (0.4.2)
pip (1.5.4)
prettytable (0.7.2)
pycurl (7.19.3)
pyOpenSSL (0.13)
pyserial (2.6)
python-apt (0.9.3.5)
python-debian (0.1.21-nmu2ubuntu2)
PyYAML (3.10)
requests (2.2.1)
setuptools (3.3)
six (1.5.2)
ssh-import-id (3.21)
Twisted-Core (13.2.0)
Twisted-Names (13.2.0)
Twisted-Web (13.2.0)
urllib3 (1.7.1)
wsgiref (0.1.2)
zope.interface (4.0.5)

## Now install the AWS CLI version 1.6.6, and verify the command works:

$ sudo -E pip install awscli==1.6.6
$ aws --version
aws-cli/1.6.6 Python/2.7.6 Linux/3.13.0-36-generic

## Now upgrade the AWS CLI.  Notice in the output below that it's picking up the new version of six, should I've denoted in the output:

$ sudo -E pip install --upgrade awscli
Downloading/unpacking awscli from https://pypi.python.org/packages/source/a/awscli/awscli-1.6.8.tar.gz#md5=08fa73eb1010e34a03dd820e14ff4fee
  Downloading awscli-1.6.8.tar.gz (280kB): 280kB downloaded
  Running setup.py (path:/tmp/pip_build_root/awscli/setup.py) egg_info for package awscli

Downloading/unpacking botocore>=0.78.0,<0.79.0 (from awscli)
  Downloading botocore-0.78.0.tar.gz (1.2MB): 1.2MB downloaded
  Running setup.py (path:/tmp/pip_build_root/botocore/setup.py) egg_info for package botocore

Requirement already up-to-date: bcdoc>=0.12.0,<0.13.0 in /usr/local/lib/python2.7/dist-packages (from awscli)
Downloading/unpacking six>=1.8.0,<2.0.0 (from awscli)
  Downloading six-1.8.0-py2.py3-none-any.whl    #### <-------- Note here so it's picking up the new version of six.
Requirement already up-to-date: colorama==0.2.5 in /usr/lib/python2.7/dist-packages (from awscli)
Requirement already up-to-date: docutils>=0.10 in /usr/local/lib/python2.7/dist-packages (from awscli)
Requirement already up-to-date: rsa==3.1.2 in /usr/local/lib/python2.7/dist-packages (from awscli)
Requirement already up-to-date: jmespath==0.5.0 in /usr/local/lib/python2.7/dist-packages (from botocore>=0.78.0,<0.79.0->awscli)
Requirement already up-to-date: python-dateutil>=2.1,<3.0.0 in /usr/local/lib/python2.7/dist-packages (from botocore>=0.78.0,<0.79.0->awscli)
Requirement already up-to-date: pyasn1>=0.1.3 in /usr/local/lib/python2.7/dist-packages (from rsa==3.1.2->awscli)
Installing collected packages: awscli, botocore, six
  Found existing installation: awscli 1.6.6
    Uninstalling awscli:
      Successfully uninstalled awscli
  Running setup.py install for awscli
    changing mode of build/scripts-2.7/aws from 644 to 755
    changing mode of build/scripts-2.7/aws.cmd from 644 to 755
    changing mode of build/scripts-2.7/aws_completer from 644 to 755
    changing mode of build/scripts-2.7/aws_zsh_completer.sh from 644 to 755

    changing mode of /usr/local/bin/aws_zsh_completer.sh to 755
    changing mode of /usr/local/bin/aws to 755
    changing mode of /usr/local/bin/aws.cmd to 755
    changing mode of /usr/local/bin/aws_completer to 755
  Found existing installation: botocore 0.77.0
    Uninstalling botocore:
      Successfully uninstalled botocore
  Running setup.py install for botocore

  Found existing installation: six 1.5.2
    Uninstalling six:
      Successfully uninstalled six     #### <---- Uninstalling the old 1.5.2 version, installing the new version.
Successfully installed awscli botocore six
Cleaning up...

## Now verify the new version works:

ubuntu@ip-10-213-171-2:~$ aws --version
aws-cli/1.6.8 Python/2.7.6 Linux/3.13.0-36-generic


## Also double check the version of six:

$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> six.__version__
'1.8.0'
>>> six.__file__
'/usr/local/lib/python2.7/dist-packages/six.pyc'
>>> six.moves.shlex_quote
<function quote at 0x7f31e21655f0>

[danielgtaylor]

Here's what I get on 10.10.1:

$ /usr/bin/python
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> six.__version__
'1.4.1'
>>> print(six.__file__)
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc
>>> 

From a virtualenv with it installed:

Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> six.__version__
'1.8.0'
>>> print six.moves.shlex_quote
<function quote at 0x10dc9c230>
>>> print(six.__file__)
/....../venv/lib/python2.7/site-packages/six.pyc

[jamesls]

@danielgtaylor what if you try to install the AWS CLI system wide instead of in a virtualenv (upgrading from 1.6.6)? i.e

sudo pip install awscli==1.6.6
sudo pip install --upgrade awscli

[ajkerr]

This seems to be the default sys.path on OS X, which seems to what is causing the problems. pip installs into /Library/Python/2.7/site-packages, which is last in the path.

Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pprint import pprint
>>> import sys
>>> pprint(sys.path)
['',
 '/Library/Python/2.7/site-packages/setuptools-2.0-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages']

[jamesls]

@ajkerr Ahh that explains it!

In that case, I'm not really sure there's much we can do with our current approach as the CLI shouldn't really be messing with sys.path and we don't really want to ask customers to alter the python path.

So here's what we'll do. Given six is just a single file, we'll go ahead and vendor six into our codebase instead of installing it via pip. Given six is used by a ton of python modules, this gives us the flexibility to upgrade the version of six as needed. This also should make it easier for distro package maintainers as they don't need to pull in the latest version of six either to update their version of the AWS CLI.

And given there's a couple of mac users on this thread, what does everything think about a pkg installer? Would any mac users find this useful? I think we'd probably use the bundled installer to power it, but I think this would likely prevent any possible issues like this we'd see in the future. We'd still support pip, but just wanted to gauge interest for a mac pkg installer. cc @ajkerr @bok-

[ajkerr]

@jamesls A pkg installer would be fine with me, as long as we can still use pip (which you're saying we can, which is good!)

[bok-]

A package installer is fine as long as there is a simple path for regular upgrades (via pip or otherwise).

[kyleknap]

Alright, we believe this should be working now via this PR:
#1057

We vendored six into botocore and the cli now uses botocore's vendored six. That way we won't run into anymore OS related issues related to the installation of six. This will be out in the next release. Closing for now. Let me know if any of you run into any similar issues when it does come out or if you pull down the head of the develop branch and still does not work.

[ajkerr]

Confirmed fixed with v1.6.9 on OS X 10.10.1. Thanks!