py.test 2.1.0 causes new errors

[pytestbot]

Originally reported by: Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian)

---

After upgrading to 2.1.0, when running the same tests I normal run I get a bunch of new errors:

helpers/mysql.py:1: in <module>
>   import MySQLdb
build/bdist.macosx-10.3-fat/egg/MySQLdb/__init__.py:17: in <module>
>   ???
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/_pytest/assertion/rewrite.py:93: in find_module
>       py.path.local(cache_dir).ensure(dir=True)
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:413: in ensure
>           return p._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:395: in _ensuredirs
>           parent._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:395: in _ensuredirs
>           parent._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:398: in _ensuredirs
>               self.mkdir()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:370: in mkdir
>       py.error.checked_call(os.mkdir, str(p))
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_error.py:82: in checked_call
>           raise cls("%s%r" % (func.__name__, args))
E           EEXIST: [File exists]: mkdir('/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.3-fat.egg',)

I'm not sure what is causing these new errors.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/60

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

Hello, can you tell how you are getting this or find a small example, so I can reproduce it? Thanks

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

Here is another example. Basically i'm trying to collect my smoke tests and run them. It doesn't seem to be agreeing with my imports?

{{{

[boneill@boneills-MacBook-Pro apitest (dev)]$ py.test -k smoke
=============================================================================== test session starts ===============================================================================
platform darwin -- Python 2.6.6 -- pytest-2.1.0
collecting 0 items / 2 errors^C
===================================================================================== ERRORS ======================================================================================
________________________________________________________________________ ERROR collecting discount_new.py _________________________________________________________________________
discount_new.py:5: in

import simplejson
build/bdist.macosx-10.3-fat/egg/simplejson/init.py:109: in
???
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/_pytest/assertion/rewrite.py:93: in find_module
py.path.local(cache_dir).ensure(dir=True)
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:413: in ensure
return p._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:395: in _ensuredirs
parent._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:395: in _ensuredirs
parent._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:398: in _ensuredirs
self.mkdir()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:370: in mkdir
py.error.checked_call(os.mkdir, str(p))
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_error.py:82: in checked_call
raise cls("%s%r" % (func.name, args))
E EEXIST: [File exists]: mkdir('/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/simplejson-2.1.3-py2.6-macosx-10.3-fat.egg',)
_______________________________________________________________________ ERROR collecting discount_update.py _______________________________________________________________________
discount_update.py:5: in
import simplejson
build/bdist.macosx-10.3-fat/egg/simplejson/init.py:109: in
???
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/_pytest/assertion/rewrite.py:93: in find_module
py.path.local(cache_dir).ensure(dir=True)
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:413: in ensure
return p._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:395: in _ensuredirs
parent._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:395: in _ensuredirs
parent._ensuredirs()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:398: in _ensuredirs
self.mkdir()
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_path/local.py:370: in mkdir
py.error.checked_call(os.mkdir, str(p))
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/py/_error.py:82: in checked_call
raise cls("%s%r" % (func.name, args))
E EEXIST: [File exists]: mkdir('/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/simplejson-2.1.3-py2.6-macosx-10.3-fat.egg',)
_________________________________________________________________________ ERROR collecting event_copy.py _________________________________________________________________

}}}

I could try writing a standalone test that might cause this, but I think it's the import at the top that is causing issues.

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

Here is another example where it is running on a remote machine that doesn't have as much write access:

{{{
mytickets.py:1: in

import unittest, time, re
../../eventbrite-2245-dev-2d19765/eventbrite_python/lib/python2.6/site-packages/_pytest/assertion/rewrite.py:93: in find_module
py.path.local(cache_dir).ensure(dir=True)
../../eventbrite-2245-dev-2d19765/eventbrite_python/lib/python2.6/site-packages/py/_path/local.py:413: in ensure
return p._ensuredirs()
../../eventbrite-2245-dev-2d19765/eventbrite_python/lib/python2.6/site-packages/py/_path/local.py:398: in _ensuredirs
self.mkdir()
../../eventbrite-2245-dev-2d19765/eventbrite_python/lib/python2.6/site-packages/py/_path/local.py:370: in mkdir
py.error.checked_call(os.mkdir, str(p))
../../eventbrite-2245-dev-2d19765/eventbrite_python/lib/python2.6/site-packages/py/_error.py:82: in checked_call
raise cls("%s%r" % (func.name, args))
E EACCES: [Permission denied]: mkdir('/usr/lib/python2.6/pycache',)

}}}

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

fix assertion rewriting in read-only directories (refs #60)

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

Can you try with the Mercurial head?

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

Still getting the same "file exists" errors

{{{
E EEXIST: [File exists]: mkdir('/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.3-fat.egg',)
}}}

Why is pytest trying to create directories?

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

I'm unable to reproduce this on Linux mostly because I'm not exactly sure what you're running. This, for example, works for me:

{{{
benjamin@localhost ~/dev/repos/py $ virtualenv t
New python executable in t/bin/python2.7
benjamin@localhost ~/dev/repos/py $ cd t
benjamin@localhost ~/dev/repos/py/t $ . bin/activate
(t)benjamin@localhost ~/dev/repos/py/t $ easy_install simple_json
(t)benjamin@localhost ~/dev/repos/py/t $ mkdir w
(t)benjamin@localhost ~/dev/repos/py/t $ cd w
(t)benjamin@localhost ~/dev/repos/py/t/w $ cat > test_x.py
import simplejson

def test_x(): pass
(t)benjamin@localhost ~/dev/repos/py/t/w $ py.test
}}}

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

sfbrian: pytest-2.1 creates directories for caching pyc files (for test modules with rewritten assert statements). Are you by chance using a "python_files" setting or have test files in zip/egg files?

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

I don't think we are using python_files. Some of our tests manipulate a file here or there. I definitely know that our tests aren't in zip or eggs. It looks like pytest is trying to create a directory (and overwrite it) for my imports and not just the test files.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

can you retry with mercurial head or "pip install -i http://pypi.testrun.org -U pytest" and send a "pytestdebug.log" that you can create with a "py.test --debug ..." invocation?

from the trace we hopefully understand better what's going on - have added some more info there.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Brian, any chance you could provide today the pytestdebug.log as mentioned in my last post?

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

Sorry, was on vacation the past week. I will get on that today.

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

Attached the log. I took a brief look and this around here looks to be the problem:

{{{
find_module called for: helpers.mysql [assertion]
matched test file '/Users/boneill/Documents/Automation/qa/qa/selenium/helpers/helpers.mysql.py' [assertion]
rewriting '/Users/boneill/Documents/Automation/qa/qa/selenium/helpers/helpers.mysql.py' [assertion]
find_module called for: helpers.MySQLdb [assertion]
matched test file '/Users/boneill/Documents/Automation/qa/qa/selenium/helpers/helpers.MySQLdb.py' [assertion]
rewriting '/Users/boneill/Documents/Automation/qa/qa/selenium/helpers/helpers.MySQLdb.py' [assertion]
find_module called for: MySQLdb [assertion]
find_module called for: MySQLdb.release [assertion]
matched test file '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.3-fat.egg/MySQLdb/MySQLdb.release.py' [assertion]
failure to create directory: '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.3-fat.egg/MySQLdb' [assertion]
finish pytest_make_collect_report --> <CollectReport 'organizer_profile.py' lenresult=0 outcome='failed'> [hook]

}}}

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

It looks like py.test is matching absolutely everything as a test file. What sort of custom config is in effect?

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

The only thing I have in my config is
{{{
[pytest]
python_files=*.py
}}}

I have it in the same directory as my test files. It has worked fine with previous versions of py.test

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

only try to create the pycache dir (not a tree to it) fixes #60

Also, improve error handling surrounding pycache creation.

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

Tested it out and it works great! Thanks for the fix Benjamin. It is greatly appreciated!

[pytestbot]

Original comment by Benjamin Peterson (BitBucket: gutworth, GitHub: gutworth):

---

I'm glad that helped. By the way, you may want to reconsider using python_files = *.py, since that causes py.test to load every single python file as a test.

[pytestbot]

Original comment by Brian O'Neill (BitBucket: sfbrian, GitHub: sfbrian):

---

every python file in the current directory or everywhere? I assumed it was in the current directory (where all my tests exist).