boto_test broken on mac

[lfpino]

Hi there,

I'm the Bazel release manager and I've been investigating the rules_python failures (https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/233#14f47bce-4863-42cf-9179-fe4b1be650a8) in the new release candidate for Bazel 0.14.0. After trying to reproduce this in my macOS I realized that the failures are also present with Bazel 0.12.0 as follows:

$ git clone https://github.com/bazelbuild/rules_python.git
$ cd rules_python
$ /tmp/download/bazel-0.12.0-darwin-x86_64 test //examples/boto:boto_test
INFO: Analysed target //examples/boto:boto_test (23 packages loaded).
INFO: Found 1 test target...
FAIL: //examples/boto:boto_test (see /private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/testlogs/examples/boto/boto_test/test.log)
Target //examples/boto:boto_test up-to-date:
  bazel-bin/examples/boto/boto_test
INFO: Elapsed time: 3.315s, Critical Path: 1.16s
INFO: Build completed, 1 test FAILED, 5 total actions
//examples/boto:boto_test                                                FAILED in 1.0s
  /private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/testlogs/examples/boto/boto_test/test.log

$ cat /private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/testlogs/examples/boto/boto_test/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/io_bazel_rules_python/examples/boto/boto_test.py", line 15, in <module>
    import boto3
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__boto3_1_4_7/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__boto3_1_4_7/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__botocore_1_7_48/botocore/session.py", line 26, in <module>
    import botocore.credentials
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__botocore_1_7_48/botocore/credentials.py", line 22, in <module>
    from dateutil.parser import parse
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__python_dateutil_2_7_3/dateutil/parser/__init__.py", line 2, in <module>
    from ._parser import parse, parser, parserinfo
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__python_dateutil_2_7_3/dateutil/parser/_parser.py", line 50, in <module>
    from .. import tz
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__python_dateutil_2_7_3/dateutil/tz/__init__.py", line 2, in <module>
    from .tz import *
  File "/private/var/tmp/_bazel_lpino/4cea827263cf09da0a5ddbf48df3f327/bazel-sandbox/5352791971010170471/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__python_dateutil_2_7_3/dateutil/tz/tz.py", line 19, in <module>
    from six.moves import _thread
ImportError: cannot import name _thread
Executed 1 out of 1 test: 1 fails locally.

Given this I think the failures are unrelated to a recent change in Bazel itself.

[lfpino]

Friendly ping @mattmoor @duggelz, any idea what's happening here?

[jin]

Ping?

[laurentlb]

Still broken with Bazel 0.15: https://storage.googleapis.com/bazel-buildkite-artifacts/e71e36da-530e-47cf-a038-698c2d1999b6/examples/boto/boto_test/attempt_1.log

  File "/private/var/tmp/_bazel_buildkite/ee3010beef19bc6526b5cec7b7da7249/sandbox/darwin-sandbox/3/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__python_dateutil_2_7_3/dateutil/tz/tz.py", line 19, in <module>
    from six.moves import _thread
ImportError: cannot import name _thread

[laszlocsomor]

Still failing on Mac in https://buildkite.com/bazel/bazel-toolchains/builds/363.

[laszlocsomor]

/cc @aragos

[c-parsons]

This is still failing on Mac at HEAD.

[hlopko]

And still failing on macs: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/386#aaf0f0e0-66a6-4167-b5ef-47be90bff33f

cc @brandjon

[meteorcloudy]

Any progress on this?

[brandjon]

Sounds like maybe on mac an old version of six is getting installed? Investigating...

dateutil/dateutil#6

[brandjon]

Confirmed that on mac six is 1.4.1 (the linked thread says it should be at least 1.5). On Ubuntu it's 1.10.0.

[brandjon]

Ah, this looks mighty suspicious: https://stackoverflow.com/questions/29485741/unable-to-upgrade-python-six-package-in-mac-osx-10-10-2

[brandjon]

So I dumped the PYTHONPATH and on the Mac it comes out as follows:

/Users/buildkite/builds/buildkite-imacpro-1/bazel/rules-python-python/examples/boto
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__python_dateutil_2_7_5
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__jmespath_0_9_3
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__docutils_0_14
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__botocore_1_7_48
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__futures_3_2_0
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__s3transfer_0_1_13
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/pypi__boto3_1_4_7
/private/var/tmp/_bazel_buildkite/99b266b1d54a6d514e8d04532aa60b4a/sandbox/darwin-sandbox/4/execroot/io_bazel_rules_python/bazel-out/darwin-fastbuild/bin/examples/boto/boto_test.runfiles/io_bazel_rules_python
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
/Library/Python/2.7/site-packages
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC

Yes, there are Frameworks paths there, but the repo runfiles paths come first so they take precedence. It's just that there's no repo path for six, even though there are repo paths for all other dependencies. The same is true on my local linux machine, so I take it that both platforms are using the system-installed version of six, and in the case of Mac that version's just too old.

I wonder why there's no repo for six. When I create a virtualenv and do pip wheel -w tempdir boto3==1.4.7, tempdir/ includes a .whl for it. So I'd expect it to be picked up in piptool.py's list_whls().

[brandjon]

Six doesn't appear as a dep in bazel query "deps(@pypi__boto3_1_4_7//:pkg)", while the other whl repos do...

[brandjon]

Ok, I've tracked it down. The python-dateutil package is the one that has the PyPI dependency on six. But if we inspect its generated Bazel target with bazel query "@pypi__python_dateutil_2_7_5//:pkg" --output=build, we see that it is missing a dependency on the library target for six.

Unpacking python_dateutil-2.7.5-py2.py3-none-any.whl, we see that its dist-info dir lacks a metadata.json file and instead has a METADATA file. The dependencies are determined here, and the code path for parsing METADATA has a big TODO.

METADATA parsing problems are covered by issues #70 and #95. In the meantime we can also use the workaround in the first comment of #70 and manually declare the dependency in the test's requirements.txt file.