MemoryError when writing junitxml for large test suite

[pytestbot]

Originally reported by: Michael Howitz (BitBucket: icemac, GitHub: icemac)

---

In a large project (> 3500 Tests) we get a MemoryError when writing the junit xml file.
(The host is our Jenkins CI server with 4 GB of RAM and 2 GB of swap.)

#!python

bin/py.test --nosugar --junitxml=parts/tests.xml --cov-report html --cov-report xml --cov-report term --cov src
============================= test session starts ==============================
platform linux2 -- Python 2.7.6 -- py-1.4.25 -- pytest-2.6.4
plugins: gocept.pytestlayer, cache, cov, sugar, remove-stale-bytecode
collected 3643 items

… Many, many tests which ran fine …

Traceback (most recent call last):
  File "bin/py.test", line 308, in <module>
    sys.exit(pytest.main())
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/config.py", line 41, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 413, in __call__
    return self._docall(methods, kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 424, in _docall
    res = mc.execute()
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
    res = method(**kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/main.py", line 116, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/main.py", line 109, in wrap_session
    exitstatus=session.exitstatus)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 413, in __call__
    return self._docall(methods, kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 424, in _docall
    res = mc.execute()
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
    res = method(**kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/terminal.py", line 349, in pytest_sessionfinish
    __multicall__.execute()
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/core.py", line 315, in execute
    res = method(**kwargs)
  File "/srv/jenkins/eggs/pytest-2.6.4-py2.7.egg/_pytest/junitxml.py", line 215, in pytest_sessionfinish
    ).unicode(indent=0))
  File "/srv/jenkins/eggs/py-1.4.25-py2.7.egg/py/_xmlgen.py", line 54, in unicode
    return u("").join(l)
MemoryError

Any suggestions?

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/641

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

If you don't write the junitxml file py.test completes? How is memory usage evolving during the test run?

[pytestbot]

Original comment by Michael Howitz (BitBucket: icemac, GitHub: icemac):

---

Without the option --junitxml py.test completes. During the test run up to 1,5 GB of RAM is needed. I was not yet able to see the peek when writing the XML file.

When running the tests locally the resulting XML file is about 150 MB.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

ok, then it sounds like writing out the xml in a more iterative way should help. py._xmlgen.SimpleUnicodeVisitor might help to achieve that. Not going to try that myself soon but can review a PR.