[xenial] resolve i18n related test failures

[redshiftzero]

Description

This is one of the Xenial-related application test failures.

Steps to Reproduce

1. make test-xenial

Expected Behavior

tests/test_i18n_tool.py::TestI18NTool::test_translate_messages_l10n and tests/test_i18n.py::test_i18n pass

Actual Behavior

Both tests fail

Comments

Not sure what is going on here, needs investigation:

__________________ TestI18NTool.test_translate_messages_l10n ___________________

self = <tests.test_i18n_tool.TestI18NTool object at 0x7fb082a71990>
tmpdir = local('/tmp/pytest-of-www-data/pytest-0/test_translate_messages_l10n0')

    def test_translate_messages_l10n(self, tmpdir):
        source = [
            join(self.dir, 'i18n/code.py'),
            join(self.dir, 'i18n/template.html'),
        ]
        args = [
            '--verbose',
            'translate-messages',
            '--translations-dir', str(tmpdir),
            '--mapping', join(self.dir, 'i18n/babel.cfg'),
            '--sources', ",".join(source),
            '--extract-update',
            '--compile',
        ]
>       i18n_tool.I18NTool().main(args)

tests/test_i18n_tool.py:122: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
i18n_tool.py:364: in main
    return args.func(args)
i18n_tool.py:78: in translate_messages
    *sources)
/usr/local/lib/python2.7/dist-packages/sh.py:1427: in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
/usr/local/lib/python2.7/dist-packages/sh.py:774: in __init__
    self.wait()
/usr/local/lib/python2.7/dist-packages/sh.py:792: in wait
    self.handle_command_exit_code(exit_code)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = , code = 1

    def handle_command_exit_code(self, code):
        """ here we determine if we had an exception, or an error code that we
            weren't expecting to see.  if we did, we create and raise an exception
            """
        ca = self.call_args
        exc_class = get_exc_exit_code_would_raise(code, ca["ok_code"],
                ca["piped"])
        if exc_class:
            exc = exc_class(self.ran, self.process.stdout, self.process.stderr,
                    ca["truncate_exc"])
>           raise exc
E           ErrorReturnCode_1: 
E           
E             RAN: /usr/local/bin/pybabel extract --charset=utf-8 --mapping /home/circleci/project/securedrop/tests/i18n/babel.cfg --output /tmp/pytest-of-www-data/pytest-0/test_translate_messages_l10n0/messages.pot --project=SecureDrop --version 0.12.0~rc1 --msgid-bugs-address=securedrop@freedom.press --copyright-holder=Freedom of the Press Foundation /home/circleci/project/securedrop/tests/i18n/code.py /home/circleci/project/securedrop/tests/i18n/template.html
E           
E             STDOUT:
E           
E           
E             STDERR:
E           extracting messages from /home/circleci/project/securedrop/tests/i18n/code.py
E           extracting messages from /home/circleci/project/securedrop/tests/i18n/template.html (extensions="jinja2.ext.autoescape,jinja2.ext.with_")
E           Traceback (most recent call last):
E             File "/usr/local/bin/pybabel", line 11, in <module>
E               sys.exit(main())
E             File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 908, in main
E               return CommandLineInterface().run(sys.argv)
E             File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 832, in run
E               return cmdinst.run()
E             File "/usr/local/lib/python2.7/dist-packages/babel/messages/frontend.py", line 467, in run
E               for filename, lineno, message, comments, context in extracted:
E             Fi... (1617 more, please see e.stderr)

/usr/local/lib/python2.7/dist-packages/sh.py:815: ErrorReturnCode_1

[kushaldas]

The issue is happening due to an older version of setuptools package.

root@app-staging:/var/www/securedrop# python                                                                                                         [105/433]
Python 2.7.12 (default, Nov 12 2018, 14:36:49) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.__version__
'20.7.0'

I can manually solve it by upgrading to the latest setuptools.

# pip install --upgrade setuptools
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.wh
l (573kB)
    100% |████████████████████████████████| 573kB 1.8MB/s 
Installing collected packages: setuptools
  Found existing installation: setuptools 20.7.0
    Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed setuptools-40.6.3
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

How to verify?

# /usr/local/bin/pybabel extract --charset=utf-8 --mapping tests/i18n/babel.cfg --output /tmp/pytest-of-root/pytest-0/tes$
_i18n0/data/tmp/messages.pot --project=SecureDrop --version 0.12.0~rc1 --msgid-bugs-address=securedrop@freedom.press --copyright-holder="Freedom of the Press 
Foundation" tests/i18n/code.py tests/i18n/template.html
extracting messages from tests/i18n/code.py
extracting messages from tests/i18n/template.html (extensions="jinja2.ext.autoescape,jinja2.ext.with_")
writing PO template file to /tmp/pytest-of-root/pytest-0/test_i18n0/data/tmp/messages.pot

# pytest -v tests/test_i18n.py 
==================================================================== test session starts =====================================================================
platform linux2 -- Python 2.7.12, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: tests/.cache
rootdir: /var/www/securedrop/tests, inifile: pytest.ini
plugins: mock-1.7.1, cov-2.5.1
collected 6 items                                                                                                                                            

tests/test_i18n.py::test_get_supported_locales PASSED                                                                                                  [ 16%]
tests/test_i18n.py::test_i18n PASSED                                                                                                                   [ 33%]
tests/test_i18n.py::test_verify_default_locale_en_us_if_not_defined_in_config PASSED                                                                   [ 50%]
tests/test_i18n.py::test_locale_to_rfc_5646 PASSED                                                                                                     [ 66%]
tests/test_i18n.py::test_html_en_lang_correct PASSED                                                                                                   [ 83%]
tests/test_i18n.py::test_html_fr_lang_correct PASSED                                                                                                   [100%]

Before creating any PR, I will try to see if anything else breaks due to this upgrade. @redshiftzero any thoughts?

[kushaldas]

The error is due to the following crash

  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2228, in load
    self.require(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2245, in require
    items = working_set.resolve(reqs, env, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 808, in resolve
    if not req_extras.markers_pass(req):
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 993, in markers_pass
    return not req.marker or any(extra_evals) or req.marker.evaluate()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/_vendor/packaging/markers.py", line 278, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/_vendor/packaging/markers.py", line 203, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/_vendor/packaging/markers.py", line 185, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
pkg_resources._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

[kushaldas]

Here is the related upstream issue: pypa/setuptools#523

[redshiftzero]

Looks like this is happening only with the call to pybabel.extract? If that's the case, then we could add to make translate an update of setuptools, such that the localization manager can update the POT files without hitting the upstream bug, and we don't make the change to dev/staging/prod.

[redshiftzero]

The issue on the babel side: python-babel/babel#501

Here are three options to proceed on this:

1. Update setuptools across all environments. This is dangerous due to [RFE] Isolate yourself from the system Python #3407.
2. Update setuptools in the Dockerfile (i.e. only for dev/test) which is what Fixes #4036 Updates setuptools for ci #4041 does. It is OK to only update in dev/test, because the failures due to this specific version of setuptools only occur in the dev environment (when we extract strings to be translated). It does mean that if there is an issue due to differences in behavior from the version of setuptools installed in staging/prod and in dev/test, that we would not detect it with the application test job, and instead would need to detect it in either: staging job running application tests, external server testing against staging server (planned, [functional testing] Support tor-browser-selenium tests against app-staging in CI #3661), or pre-release QA (which is what would catch the issue now).
3. Only upgrade setuptools for the translation-related tests and for make translate as I mention above. This involves having to add a test fixture to update just one dependency to get the tests to pass e.g. something like:

@pytest.fixture(scope='module')
def update_setuptools():
    import setuptools
    setuptools_version = setuptools.__version__
    subprocess.check_output(['sudo', 'pip', 'install', '--upgrade', 'setuptools'])
    imp.reload(setuptools)
    yield
    subprocess.check_output(['sudo', 'pip', 'install', 'setuptools=={}'.format(setuptools_version)])
    imp.reload(setuptools)

Given that this is pretty hacky and confusing for future maintainers, I think we should take option 2, and clearly mark in the Dockerfile why this was done.

I have verified via manual testing that other i18n/i10n-related functionality (e.g. babel/gettext) works without issue in the Xenial staging environment.