20.5.0+ breaks existing setup.py files - UndefinedEnvironmentName #523

lnielsen · 2016-03-29T19:21:26Z

The latest 20.6.4 PyPI is breaking our setup.py files with the following exception:

Traceback (most recent call last):
  File "setup.py", line 163, in <module>
    cmdclass={'test': PyTest},
  File "/opt/python/2.7.9/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/opt/python/2.7.9/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/command/test.py", line 152, in run
    self.distribution.fetch_build_eggs(self.distribution.tests_require)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/__init__.py", line 814, in resolve
    if req.marker and not req.marker.evaluate():
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 273, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 198, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 180, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

The text was updated successfully, but these errors were encountered:

lnielsen · 2016-03-29T19:23:04Z

Seems to be related to this commit d25e2a5

jaraco · 2016-03-29T20:05:04Z

@s-t-e-v-e-n-k Can you look into this one?

@lnielsen How severe is the failure? Should I pull 20.5+ from PyPI?

lnielsen · 2016-03-29T20:16:41Z

Looks pretty serve. Just upgrading setuptools in a virtualenv breaks the code.

lnielsen · 2016-03-29T20:35:46Z

Just debugging now to provide some more information.

jaraco · 2016-03-29T20:39:31Z

I've removed 20.6.4 and 20.5.0 until the issue can be resolved.

msabramo · 2016-03-29T21:15:52Z

Also seeing this.

msabramo · 2016-03-29T21:22:58Z

My colleague has been looking into this and he thinks that it has to do with the fact that our tests are using a variable/parameter name called extra_environ. This is a parameter that is used by webtest (e.g.: see http://docs.pylonsproject.org/projects/webtest/en/latest/testapp.html#modifying-the-environment-simulating-authentication)

bszonye · 2016-03-29T21:23:43Z

I am seeing this in code that has variables and parameters named extra_environ that are ultimately passed to the webtest API. Renaming to extra_env_vars and extra_evars still hit the problem, but renaming to other_environ fixed it, so it looks like something is incorrectly identifying the extra_ pattern as an environment marker.

nakato · 2016-03-30T06:01:16Z

@lnielsen I can't reproduce this. How are you getting that traceback? What commands are you executing? What is the logging leading up to it? Is there anything that follows it?

jaraco · 2016-03-31T01:57:32Z

I can't replicate this either.

I inferred from the link that the command being executed is python setup.py test in the invenio-records-rest project, so I checked out that repo and ran setup.py test under Python 2.7 and 3.5 using setuptools 20.5+, and while some tests failed, the reported error did not occur.

I sense now that the failures are fairly isolated and only affecting tests, so I'm going to re-release the code in the latest version, but I am eager to see a fix for this issue, so do please provide more detail to describe how one can recreate the issue, preferably with a minimal use case.

bszonye · 2016-03-31T02:43:45Z

Our teams are encountering this when using the webtest API in a py.test fixture, so it affects all of their tests. I do not yet have a workaround that successfully builds against the webtest classes without triggering the setuptools bug, so those teams cannot even run commit tests to integrate new code.

jaraco · 2016-03-31T02:59:38Z

@bszonye Sorry. I don't mean to cause you trouble. Can you pin your environment to Setuptools 20.4 until the issue can be resolved?

bszonye · 2016-03-31T03:03:23Z

That’s what I recommended to the team for now, to pin setuptools<20.5 – tomorrow I will take a closer look to see if I can find a minimal test case.

bszonye · 2016-03-31T03:45:27Z

@jaraco Just confirmed with the team that version 20.6.6 does break their tests again, but they are able to work around by pinning to the 20.4 version. I suspect that something in packaging is misidentifying extra markers from code that happens to contain the string in certain identifiers.

Our traceback ends at the same place but starts with py.test instead of setup.py test:

[tests.python.functional.test_XXX.TestXXX.test_XXX] tests/python/functional/utils.py:44: in setUp
    self.testapp = get_webtest_testapp()
tests/python/functional/utils.py:98: in get_webtest_testapp
    app = get_app(join(conf_dir, 'test.ini'))
.tox/py27/local/lib/python2.7/site-packages/pyramid/paster.py:31: in get_app
    global_conf=options)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:247: in loadapp
    return loadobj(APP, uri, name=name, **kw)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:271: in loadobj
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:296: in loadcontext
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:320: in _loadconfig
    return loader.get_context(object_type, name, global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:450: in get_context
    global_additions=global_additions)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:559: in _pipeline_app_context
    APP, pipeline[-1], global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:454: in get_context
    section)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:476: in _context_from_use
    object_type, name=use, global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:406: in get_context
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:296: in loadcontext
    global_conf=global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:328: in _loadegg
    return loader.get_context(object_type, name, global_conf)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:620: in get_context
    object_type, name=name)
.tox/py27/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:640: in find_egg_entry_point
    pkg_resources.require(self.spec)
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py:947: in require
    needed = self.resolve(parse_requirements(requirements))
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py:814: in resolve
    if req.marker and not req.marker.evaluate():
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py:278: in evaluate
    return _evaluate_markers(self._markers, current_environment)
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py:203: in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
.tox/py27/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py:185: in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
E   UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

s-t-e-v-e-n-k · 2016-03-31T06:09:29Z

@bszonye I'd like to see which requirement is causing this issue, are you able to edit the virtual env's copy of local/lib/python2.7/site-packages/pkg_resources/init.py to log what 'req' is set to just before the call to req.marker.evaluate()? I'd personally use the q module.

warner · 2016-03-31T08:07:52Z

This is biting us too, on today's (brand-new) release of tahoe-lafs. To reproduce: pip install tahoe-lafs, then tahoe --version.

If I add a print req just before the req.marker.evaluate() call at pkg_resources/__init__.py line 814, then when I run our application, I see:

pyOpenSSL>=0.14
Nevow>=0.11.1
Twisted[tls]>=15.1.0
pyasn1-modules>=0.0.5
pyasn1>=0.1.8
characteristic>=14.0.0
service-identity
pycryptopp>=0.6.0
pycrypto!=2.2,!=2.4,>=2.1.0
foolscap>=0.10.1
zope.interface!=3.6.3,!=3.6.4,>=3.6.0
simplejson>=1.4
zfec>=1.1.0
setuptools>=11.3
six>=1.5.2
cryptography>=1.3
twisted>=13.0
pyopenssl>=0.13; extra == "tls"

followed by the same traceback as above.

Tahoe depends, among most of those other things, on twisted[tls] >= 15.1.0 and pyopenssl >= 0.14. (See https://tahoe-lafs.org/~warner/deps.png for a graph of the dependencies, created by https://github.com/tahoe-lafs/tahoe-lafs/blob/master/misc/coding_tools/graph-deps.py).

Nevow wants twisted >= 0.13, so I'm guessing that's where the next-to-last req comes from. Twisted[tls] wants pyopenssl >= 0.13. Is it normal that the req should have "extra == tls" on the pyopenssl line, when the extra is associated with Twisted?

Grepping through our code, we do have functions and variables with extra_ in them, in case it matters.

Please let me know if there's anything I can do to help track this one down.

warner · 2016-03-31T09:11:05Z

I don't know this code very well, but I think the requirement that it's tripping on is this one from Twisted's dist-info/METADATA file:

Requires-Dist: pyopenssl (>=0.13); extra == 'tls'

and I think the req.marker.evaluate() call in the traceback is treating this extra=='tls' as a marker (e.g. python_version>=2.7), rather than as an extra.

I know the line-based METADATA format necessarily merges extras and markers into the same syntax. Is this causing confusion? I see calls to e.g. req.marker.evaluate({'extra': extra}) elsewhere, and it seems like adding that kind of argument to our evaluate() call would help. I see req.extras is available, but I'm not sure how to deal with the fact that it's a list. Call it once with everything in req.extras and continue if any of them say True? Or if none of them say False?

s-t-e-v-e-n-k · 2016-03-31T09:14:21Z

extra == 'tls' is a valid environment marker. But yes, you're totally right, the evaluate() call needs to respect extras, which I'll be looking at tomorrow.

rbtcollins · 2016-03-31T10:21:10Z

For clarity, extra=='tls' is not a valid marker for setup.py scripts. Its valid in the syntax for distribution metadata only. Callers of the Python API must supply a extra mapping when evaluating requirements specifications if they are processing distribution metadata entries (like the one above) .

tl;dr: pkg_resources, when processing METADATA files can supply a value for extra. It may be wired up wrongly at the moment. extra can't be used from setup_requires or other such places though - its a gnarly thing in the distirbution metadata spec, not a user interface element.

I'll look more closely at the actual calling code tomorrow - this is mainly meant to hint to folk that have already been looking at it.

rbtcollins · 2016-03-31T10:22:48Z

Oh, and - 23:19 < warner> we've got a requirement on "twisted[tls]", and twisted"s got an extra that says [tls] needs pyopenssl
23:19 < warner> and the exception happens when it tries to do req.marker.evaluate() on a req for pyopenssl and a marker for "extra = 'tls'"
23:21 < lifeless> yeah
23:22 < lifeless> I think the problem here is that something has started reflecting markers back into requiremnets
23:22 < lifeless> but we can only do that for a subset of markers
23:22 < lifeless> the extra mapping needs to be translated when its mapped back

from IRC.

HTH

autopulated · 2016-03-31T13:55:02Z

Also just tripped over this, to reproduce:

pip install requests[security]
python -c "import pkg_resources; pkg_resources.require('requests[security]')"

Reverting back to an older setuptools with pip install setuptools==20.4 works.

jaraco · 2016-03-31T14:21:06Z

Thanks @autopulated - I was able to reproduce the issue with those commands. This definitely is a widespread issue. Rather than pull the latest releases, I'm going to disable this new functionality until a solution can be implemented.

robdennis · 2016-03-31T14:22:42Z

I'm seeing this also when running a console entry point:
leaving this stack trace in case it helps anyone. I got it on both versions 20.5 and 20.6.6 (we don't pin setuptools versions)

I happened when executing an entry point, here's the setup.py definition:

      entry_points={'console_scripts':
                    ['redacted_name = redacted_name.main:main'],
                    },

and the stack trace:

File "/opt/redacted_name/bin/redacted_name", line 8, in <module>
    from pkg_resources import load_entry_point
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2905, in <module>
    @_call_aside
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2891, in _call_aside
    f(*args, **kwargs)
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2918, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 642, in _build_master
    ws.require(__requires__)
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 947, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/__init__.py", line 814, in resolve
    if req.marker and not req.marker.evaluate():
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 273, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/opt/yamato/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 198, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/opt/redacted_name/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 180, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

…523.

jaraco · 2016-03-31T14:51:04Z

v20.6.7 is released bypassing the issue.

bszonye · 2016-03-31T17:50:10Z

Thanks for the troubleshooting tips, @s-t-e-v-e-n-k – I’ll see which requirement is causing the problem for us. I now suspect that it’s something dynamically loaded in Pyramid, and the extra_environ identifiers are a red herring.

gitrookie · 2016-05-17T10:24:39Z

I am getting this same issue.

pip install scikit-image
Collecting scikit-image
  Using cached scikit_image-0.12.3-cp35-cp35m-manylinux1_x86_64.whl
Requirement already satisfied (use --upgrade to upgrade): pillow>=2.1.0 in /usr/lib/python3/dist-packages (from scikit-image)
Requirement already satisfied (use --upgrade to upgrade): six>=1.7.3 in /usr/lib/python3/dist-packages (from scikit-image)
Collecting dask[array]>=0.5.0 (from scikit-image)
  Using cached dask-0.9.0-py2.py3-none-any.whl
Collecting networkx>=1.8 (from scikit-image)
  Using cached networkx-1.11-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): numpy; extra == "array" in /usr/lib/python3/dist-packages (from dask[array]>=0.5.0->scikit-image)
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 328, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 748, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 360, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 448, in _prepare_file
    req_to_install, finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 387, in _check_skip_installed
    req_to_install.check_if_exists()
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 997, in check_if_exists
    self.satisfied_by = pkg_resources.get_distribution(self.req)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 535, in get_distribution
    dist = get_provider(dist)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 415, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/gaurav/.virtualEnv/py3.5/share/python-wheels/pkg_resources-0.0.0-py2.py3-none-any.whl/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources.extern.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

How to circumvent it. Thanks

jaraco · 2016-08-02T02:41:31Z

@gitrookie: What version of setuptools do you have? Looks like you have a wheel called pkg_resources-0.0.0, which I think is an unreleased and unsupported version of pkg_resources.

vhakulin · 2016-09-28T16:31:45Z

This seems to be broken again in setuptools 28.0.0: I was generating message files with pybabel from jinja templates when I got this exception.

Downgrading setuptools to 27.3.1 fixed the issue.

zgoda · 2016-11-09T09:57:58Z

Still experiencing this with 28.8.0 when calling PyBabel (27.3.1 works fine):

$ pybabel extract -F ./babel.cfg -k lazy_gettext -o ./ecpapp/translations/messages.pot ./
[ ... ]
extracting messages from ecpapp/templates/base.html (extensions="jinja2.ext.autoescape,jinja2.ext.with_", encoding="utf-8")
Traceback (most recent call last):
  File "/home/jazg/v/emcp/bin/pybabel", line 11, in <module>
    sys.exit(main())
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 911, in main
    return CommandLineInterface().run(sys.argv)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 836, in run
    return cmdinst.run()
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 454, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 156, in extract_from_dir
    dirpath=absname,
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    strip_comment_tags=strip_comment_tags
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 236, in extract_from_file
    strip_comment_tags))
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/babel/messages/extract.py", line 298, in extract
    func = entry_point.load(require=True)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/jazg/v/emcp/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

jaraco · 2016-12-06T02:49:51Z

Downgrading setuptools to 27.3.1 fixed the issue.

So here's the diff from 27.3.1 to 28.0.0. I don't see anything there that appears relevant to the issue at hand. The one change in pkg_resources is only dealing with encoding issues.

Using the commands where I was able to replicate the issue earlier, I'm unable to replicate it now with Setuptools 28.0.0:

$ python -m venv ~/.envs/issue523
$ ~/.envs/issue523/bin/python -m pip install setuptools==28.0.0
Collecting setuptools==28.0.0
  Downloading setuptools-28.0.0-py2.py3-none-any.whl (467kB)
    100% |████████████████████████████████| 471kB 2.6MB/s 
Installing collected packages: setuptools
Successfully installed setuptools-28.8.0
$ ~/.envs/issue523/bin/python -m pip install 'requests[security]'
Collecting requests[security]
  Using cached requests-2.12.3-py2.py3-none-any.whl
Collecting cryptography>=1.3.4; extra == "security" (from requests[security])
  Using cached cryptography-1.6.tar.gz
Requirement already satisfied: idna>=2.0.0; extra == "security" in ./.envs/issue523/lib/python3.6/site-packages (from requests[security])
Collecting pyOpenSSL>=0.14; extra == "security" (from requests[security])
  Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.8 in ./.envs/issue523/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Requirement already satisfied: six>=1.4.1 in ./.envs/issue523/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Requirement already satisfied: setuptools>=11.3 in ./.envs/issue523/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "security"->requests[security])
Collecting cffi>=1.4.1 (from cryptography>=1.3.4; extra == "security"->requests[security])
  Using cached cffi-1.9.1.tar.gz
Requirement already satisfied: pycparser in ./.envs/issue523/lib/python3.6/site-packages (from cffi>=1.4.1->cryptography>=1.3.4; extra == "security"->requests[security])
Installing collected packages: cffi, cryptography, pyOpenSSL, requests
  Running setup.py install for cffi ... done
  Running setup.py install for cryptography ... done
Successfully installed cffi-1.9.1 cryptography-1.6 pyOpenSSL-16.2.0 requests-2.12.3
$ ~/.envs/issue523/bin/python -c "import pkg_resources; pkg_resources.require('requests[security]'); print('worked!')"
worked!

I also installed babel, but I don't have the state on my system to try to replicate the failure.

If you're running into this issue, please include steps to replicate the issue in a clean environment. Also, include the details from python -V and python -m pip --version and python -m easy_install --version (where python is the python from your env).

zgoda · 2016-12-07T08:45:47Z

Okay, more info. Clean virtualenv, installed pybabel and jinja2 with deps.

$ pip list --format=columns
Package       Version 
------------- --------
Babel         2.3.4   
Jinja2        2.8     
MarkupSafe    0.23    
pip           9.0.1   
pkg-resources 0.0.0   
pytz          2016.10 
setuptools    30.2.0  
wheel         0.30.0a0

$ python -V
Python 2.7.12

$ python -m pip --version
pip 9.0.1 from /home/jazg/v/stest/local/lib/python2.7/site-packages (python 2.7)

$ python -m easy_install --version
setuptools 30.2.0 from /home/jazg/v/stest/lib/python2.7/site-packages (Python 2.7)

Possibly relevant:

$ cat babel.cfg 
[python: **.py]

[jinja2: **/templates/**.html]
extensions=jinja2.ext.autoescape,jinja2.ext.with_
encoding=utf-8

[jinja2: **/templates/**.txt]
extensions=jinja2.ext.autoescape,jinja2.ext.with_
encoding=utf-8

And the result is:

$ pybabel extract -F ./babel.cfg -k lazy_gettext -o messages.pot ./
[... skipped OK lines from python files processed...]
extracting messages from ecpapp/templates/base.html (extensions="jinja2.ext.autoescape,jinja2.ext.with_", encoding="utf-8")
Traceback (most recent call last):
  File "/home/jazg/v/stest/bin/pybabel", line 11, in <module>
    sys.exit(main())
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 911, in main
    return CommandLineInterface().run(sys.argv)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 836, in run
    return cmdinst.run()
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/frontend.py", line 454, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 156, in extract_from_dir
    dirpath=absname,
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 207, in check_and_call_extract_file
    strip_comment_tags=strip_comment_tags
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 236, in extract_from_file
    strip_comment_tags))
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/babel/messages/extract.py", line 298, in extract
    func = entry_point.load(require=True)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/home/jazg/v/stest/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

rwillmer · 2016-12-12T14:16:14Z

I get this on a clean docker install based on ubuntu:xenial.

Virtualenv created with '$ virtualenv --no-site-packages '

Within the virtualenv, I have this:

$ python -V
Python 2.7.12

$ python -m pip --version
pip 9.0.1 from /usr/lib/ckan/default/local/lib/python2.7/site-packages (python 2.7)

$ python -m easy_install --version
setuptools 31.0.0 from /usr/lib/ckan/default/lib/python2.7/site-packages (Python 2.7)

And I get this:

Step 14 : RUN `pwd`/bin/paster make-config ckan /etc/ckan/default/development.ini
 ---> Running in 345382e0b9cf
/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/configuration.py:26: RuntimeWarning: Unable to load template engine entry point: 'pylonsmyghty = pylons.templating:MyghtyTemplatePlugin [myghty]': Traceback (most recent call last):
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pylons/templating.py", line 610, in <module>
    Engine = entry_point.load()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

  import pylons.templating

rwillmer · 2016-12-12T14:24:43Z

Here's a reproducible version of the bug.
Save this as a Dockerfile:

FROM ubuntu:xenial
RUN apt-get update && apt-get -y dist-upgrade && apt-get autoremove
RUN apt-get install -y python-dev python-pip python-virtualenv git-core apt-utils libpq-dev
RUN adduser --disabled-password --gecos '' ckan
RUN mkdir -p /usr/lib/ckan/default
RUN chown ckan: /usr/lib/ckan/default
RUN mkdir -p /etc/ckan/default
RUN chown -R ckan: /etc/ckan/
USER ckan
WORKDIR /usr/lib/ckan/default
RUN virtualenv --no-site-packages `pwd`
RUN bin/pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.6.0#egg=ckan'
RUN bin/pip install -r src/ckan/requirements.txt
# ERROR:
# https://github.com/ckan/ckan/issues/3274
# https://github.com/pypa/setuptools/issues/523
RUN bin/paster make-config ckan /etc/ckan/default/development.ini

and then try to build it:

$ docker build .

You'll get the error message from my previous comment.

rwillmer · 2016-12-12T14:35:36Z

And here's a workaround which I don't understand. Downgrading and then upgrading back to the same version and it works.
If you add these two lines before the 'RUN bin/paster' line, it works fine.

RUN bin/pip install setuptools==30.4.0
RUN bin/pip install setuptools==31.0.0

jaraco · 2016-12-12T14:50:51Z

@rwillmer: What do you get if you invoke bin/easy_install --version prior to the workaround?

rwillmer · 2016-12-12T16:07:13Z

@jaraco

$ bin/easy_install --version 
setuptools 31.0.0 from /usr/lib/ckan/default/lib/python2.7/site-packages (Python 2.7)

sirex · 2016-12-16T09:01:04Z

Same here:

venv/local/lib/python2.7/site-packages/pylons/configuration.py:26: RuntimeWarning: Unable to load template engine entry point: 'pylonsmyghty = pylons.templating:MyghtyTemplatePlugin [myghty]': Traceback (most recent call last):
  File "venv/local/lib/python2.7/site-packages/pylons/templating.py", line 610, in <module>
    Engine = entry_point.load()
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "venv/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "venv/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "venv/local/lib/python2.7/site-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

  import pylons.templating

setuptools version:

>>> setuptools.__version__
'32.0.0'

After downgrading and upgrading setuptools as suggested by @rwillmer the error is gone.

In order to set up CKAN I was using this Ansible script:

https://github.com/sirex/ckan-ivpk-import/blob/master/ckan.yml

xapple · 2017-01-26T20:03:54Z

Was also trying to install CKAN, and indeed @rwillmer's fix worked.

davidread · 2017-04-14T21:27:25Z

Simply reinstalling the same version of setuptools solves it too. e.g.

pip install --ignore-installed setuptools==34.4.1

zgoda · 2017-06-26T09:06:56Z

Still getting this on fresh virtualenv with 36.0.1:

$ python -m easy_install --version
setuptools 36.0.1 from /home/jazg/v/bwl/lib/python2.7/site-packages (Python 2.7)

zgoda · 2017-10-18T07:21:17Z

Same with 36.5.0, reinstalling the same version fixes the issue:

$ pip install --ignore-installed setuptools==36.5.0

Building the virtualenv as root caused us to run into [this][1] pip/setuptools issue. The suggested work around didn't help as the root owned virtualenv's pip didn't actually reinstall setuptools (for reasons I could not work out). Installing as the ubuntu user fixes this. [1]: pypa/setuptools#523

opfertunes · 2018-04-07T02:39:19Z

I had to manually downgrade setuptools to 34.00 to get around the "UndefinedEnvironmentName: 'extra' does not exist in evaluation environment." issue at runtime. (Ubuntu 16.04, python 2..7.12 virtualenv)

after running pip install -r ...

$ source ..../bin/activate
$ pip install setuptools==34.00

(34.00 is minimum required by my installed packages. The default, 39, triggers the error at runtime)

bmjjr · 2018-05-08T00:43:41Z

Hit this today trying to build and install from a github repo with pip install git+git://github.com/repo. Downgraded setuptools-38.5.1 to setuptools-34.00 and this resolved it so I could successfully install.

theladyjaye · 2018-10-18T05:19:38Z

I am getting this also when installing gevent

pip==18.1
setuptools==40.4.3
gevent==1.3.7

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 349, in run
    self._warn_about_conflicts(to_install)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 475, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 108, in check_install_conflicts
    package_set, should_ignore=lambda name: name not in whitelist
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 74, in check_package_set
    missed = req.marker.evaluate()
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 301, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 226, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/packaging/markers.py", line 208, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

However, if as @bmjjr stated, I downgrade to setuptools==34.0.0 it installs fine.

benoit-pierre · 2018-10-18T09:14:05Z

You'll have to give more information than that. I can't reproduce with a fresh virtualenv/venv with Python 2.7 / Python 3.7. I setuptools unvendored?

benoit-pierre · 2018-10-18T09:25:47Z

Additionally, this occurs in the "check for conflicts phase" of pip's install, which means any one of the other distributions currently installed could be the problem, so it would be worth checking with pypa/pip#5842 to find out which one.

lnielsen changed the title ~~20.6.4 breaks existing setup.py files - UndefinedEnvironmentName~~ 20.5.0+ breaks existing setup.py files - UndefinedEnvironmentName Mar 29, 2016

jacquerie mentioned this issue Mar 31, 2016

theme: MathJax fix inspirehep/inspire-next#1026

Merged

jaraco added a commit that referenced this issue Mar 31, 2016

Bypass environment marker evaluation in requirements resolution. Ref #…

04d10ff

…523.

jaraco added the bug label Mar 31, 2016

item4 mentioned this issue Mar 31, 2016

lektor command error with pkg_resources lektor/lektor#188

Closed

nepeat mentioned this issue Apr 25, 2016

Update Ubuntu in Dockerfile to the latest and add dockerignore file. MSPARP/newparp#96

Closed

ghost mentioned this issue Apr 26, 2016

Merge master to bootstrap to deploy ez_setup.py to https://bootstrap.pypa.io/ez_setup.py #560

Merged

amercader mentioned this issue Oct 21, 2016

Warning while initializing DB ckan/ckan#3274

Closed

rwillmer mentioned this issue Dec 12, 2016

"UndefinedEnvironmentName: 'extra' does not exist in evaluation environment." #887

Open

davidread mentioned this issue Apr 17, 2017

Tweak docs for Ubuntu 16.04 ckan/ckan#3544

Merged

brycepg mentioned this issue Nov 22, 2017

Reducing number of dependencies peterbrittain/asciimatics#95

Open

kushaldas mentioned this issue Jan 18, 2019

[xenial] resolve i18n related test failures freedomofpress/securedrop#4036

Closed

20.5.0+ breaks existing setup.py files - UndefinedEnvironmentName #523

20.5.0+ breaks existing setup.py files - UndefinedEnvironmentName #523

Comments

lnielsen commented Mar 29, 2016

lnielsen commented Mar 29, 2016

jaraco commented Mar 29, 2016

lnielsen commented Mar 29, 2016

lnielsen commented Mar 29, 2016

jaraco commented Mar 29, 2016

msabramo commented Mar 29, 2016

msabramo commented Mar 29, 2016

bszonye commented Mar 29, 2016

nakato commented Mar 30, 2016

jaraco commented Mar 31, 2016

bszonye commented Mar 31, 2016

jaraco commented Mar 31, 2016

bszonye commented Mar 31, 2016

bszonye commented Mar 31, 2016

s-t-e-v-e-n-k commented Mar 31, 2016

warner commented Mar 31, 2016

warner commented Mar 31, 2016

s-t-e-v-e-n-k commented Mar 31, 2016

rbtcollins commented Mar 31, 2016

rbtcollins commented Mar 31, 2016

autopulated commented Mar 31, 2016

jaraco commented Mar 31, 2016

robdennis commented Mar 31, 2016

jaraco commented Mar 31, 2016

bszonye commented Mar 31, 2016

gitrookie commented May 17, 2016

jaraco commented Aug 2, 2016

vhakulin commented Sep 28, 2016

zgoda commented Nov 9, 2016 • edited Loading

jaraco commented Dec 6, 2016

zgoda commented Dec 7, 2016

rwillmer commented Dec 12, 2016

rwillmer commented Dec 12, 2016

rwillmer commented Dec 12, 2016 • edited Loading

jaraco commented Dec 12, 2016

rwillmer commented Dec 12, 2016

sirex commented Dec 16, 2016 • edited Loading

xapple commented Jan 26, 2017

davidread commented Apr 14, 2017

zgoda commented Jun 26, 2017

zgoda commented Oct 18, 2017

opfertunes commented Apr 7, 2018

bmjjr commented May 8, 2018

theladyjaye commented Oct 18, 2018 • edited Loading

benoit-pierre commented Oct 18, 2018

benoit-pierre commented Oct 18, 2018

zgoda commented Nov 9, 2016 •

edited

Loading

rwillmer commented Dec 12, 2016 •

edited

Loading

sirex commented Dec 16, 2016 •

edited

Loading

theladyjaye commented Oct 18, 2018 •

edited

Loading