check-manifest -p pythonX fails when check-manifest is installed on pythonY, and setup.py spawns additional Python subprocesses

[rgommers]

This failure on https://github.com/PyWavelets/pywt happens for a quite common way of invoking Cython. Normal python setup.py sdist -d sometmpdir works fine.

$ check-manifest -v
listing source files under version control: 140 files and directories
building an sdist
['/usr/bin/python', 'setup.py', 'sdist', '-d', '/tmp/check-manifest-19mzk0-sdist'] failed (status 1):
expanding template: /home/rgommers/Code/tmp/pywt/util/../pywt/src/wavelets.c.src -> /home/rgommers/Code/tmp/pywt/util/../pywt/src/wavelets.c
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/home/rgommers/.local/bin/cython", line 7, in <module>
    from Cython.Compiler.Main import main
  File "/home/rgommers/.local/lib/python2.7/site-packages/Cython/Compiler/Main.py", line 7, in <module>
    import os
  File "/usr/lib/python2.7/os.py", line 398, in <module>
    import UserDict
  File "/usr/lib/python2.7/UserDict.py", line 84, in <module>
    _abcoll.MutableMapping.register(IterableUserDict)
  File "/usr/lib/python2.7/abc.py", line 109, in register
    if issubclass(subclass, cls):
  File "/usr/lib/python2.7/abc.py", line 184, in __subclasscheck__
    cls._abc_negative_cache.add(subclass)
  File "/usr/lib/python2.7/_weakrefset.py", line 86, in add
    self.data.add(ref(item, self._remove))
TypeError: cannot create weak reference to 'classobj' object
[19624 refs]
Processing pywt/src/_pywt.pyx
Traceback (most recent call last):
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 188, in <module>
    main()
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 184, in main
    find_process_files(root_dir)
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 176, in find_process_files
    process(cur_dir, fromfile, tofile, function, hash_db)
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 153, in process
    processor_function(fromfile, tofile)
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 71, in process_pyx
    raise Exception('Cython failed')
Exception: Cython failed
Expanding templates
Cythonizing sources
Traceback (most recent call last):
  File "setup.py", line 221, in <module>
    setup_package()
  File "setup.py", line 213, in setup_package
    generate_cython()
  File "setup.py", line 122, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

[mgedmin]

I cannot reproduce:

$ git clone https://github.com/PyWavelets/pywt
$ cd pywt
$ check-manifest
lists of files in version control and sdist do not match!
missing from VCS:
  cythonize.dat
  pywt/src/_pywt.c
  pywt/src/_pywt.h
  pywt/src/wavelets.c
  pywt/version.py
missing from sdist:
  .coveragerc
  appveyor.yml
suggested MANIFEST.in rules:
  include *.yml
  include .coveragerc

Googling for "TypeError: cannot create weak reference to 'classobj' object" gives clues that this may have something to do with mixing incompatible Python versions, by, e.g., having a PYTHONPATH set, or maybe using bits installed by one build of Python in ~/.local/lib/pythonX.Y/ with a different build of Python.

BTW the [19624 refs] bit in your output indicates you're running with a debug build of Python. I wonder if that's important? I couldn't test with check-manifest -p python-dbg because my apt-get installed numpy is incompatible with the debug build.

[rgommers]

Thanks for checking. My Python install is the one in /usr/lib/python2.7/ says the above traceback, so that's the system-installed one and shouldn't be a debug build. I do have debug builds of Python 3.4/3.5 and a self-installed Cython though, so I'll check what gets mixed here. Will try in a clean venv as well.

[rgommers]

There's something weird about the Python version selection done by check-manifest. check-manifest -v fails, but this works:

check-manifest -v -p python        # is 2.7 for me
check-manifest -v -p python3.4   # same as documented default

I don't see anything odd in the use of python=sys.executable at https://github.com/mgedmin/check-manifest/blob/master/check_manifest.py#L669 though.

[mgedmin]

Can you run these commands for me and paste their output here?

which python
which check-manifest
head -n 1 $(which check-manifest)

[rgommers]

$ which python
/usr/bin/python
$ which check-manifest 
/home/rgommers/.local/bin/check-manifest
$ head -n 1 $(which check-manifest)
#!/usr/bin/python3.4

So if that's the issue, it can be worked around by installing check-manifest for my default Python version:

$ pip2.7 install -U check-manifest --user
$ head -n 1 $(which check-manifest)
#!/usr/bin/python

Now it works. Going back once more to the original situation:

$ pip3.4 uninstall check-manifest
$ pip3.4 install check-manifest -U --user
$ head -n 1 $(which check-manifest)
#!/usr/bin/python3.4

Now it fails again. Maybe this is related:

$ python3.4
>>> import check_manifest
>>> check_manifest.__file__
'/home/rgommers/.local/lib/python3.4/site-packages/check_manifest.py'
$ head -n 1 /home/rgommers/.local/lib/python3.4/site-packages/check_manifest.py
#!/usr/bin/env python

After changing the first line in check_manifest.py to #!/usr/bin/python3.4 I get another traceback which reveals that site.py from my Python 3.4 install is actually imported from Python 2.7 (it's valid 3.x code but gives a SyntaxError on 2.x):

$ check-manifest -v
listing source files under version control: 141 files and directories
building an sdist
['/usr/bin/python3.4', 'setup.py', 'sdist', '-d', '/tmp/check-manifest-ezy40qut-sdist'] failed (status 1):
expanding template: /home/rgommers/Code/tmp/pywt/util/../pywt/src/wavelets.c.src -> /home/rgommers/Code/tmp/pywt/util/../pywt/src/wavelets.c
  File "/usr/lib/python3.4/site.py", line 182
    file=sys.stderr)
        ^
SyntaxError: invalid syntax
Processing pywt/src/_pywt.pyx
Traceback (most recent call last):
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 188, in <module>
    main()
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 184, in main
    find_process_files(root_dir)
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 176, in find_process_files
    process(cur_dir, fromfile, tofile, function, hash_db)
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 153, in process
    processor_function(fromfile, tofile)
  File "/home/rgommers/Code/tmp/pywt/util/cythonize.py", line 71, in process_pyx
    raise Exception('Cython failed')
Exception: Cython failed
Expanding templates
Cythonizing sources
Traceback (most recent call last):
  File "setup.py", line 221, in <module>
    setup_package()
  File "setup.py", line 213, in setup_package
    generate_cython()
  File "setup.py", line 122, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

So somehow check-manifest manages to mix two Python versions.

[mgedmin]

I bet it's 396fda8 causing a mixup of Pythons here, but only when setup.py spawns additional Python subprocesses.

[mgedmin]

I can reproduce:

$ virtualenv -p python3.4 /tmp/py34
$ /tmp/py34/bin/pip install check-manifest numpy
$ git clone https://github.com/PyWavelets/pywt
$ cd pywt
$ /tmp/py34/check-manifest
['/tmp/py34/bin/python3.4', 'setup.py', 'sdist', '-d', '/tmp/check-manifest-jwqbnca0-sdist'] failed (status 1):
expanding template: /tmp/check-manifest-6ae1__rt-sources/util/../pywt/src/wavelets.c.src -> /tmp/check-manifest-6ae1__rt-sources/util/../pywt/src/wavelets.c
Traceback (most recent call last):
  File "/tmp/py34/lib/python3.4/site.py", line 67, in <module>
    import os
  File "/tmp/py34/lib/python3.4/os.py", line 379
    yield from walk(new_path, topdown, onerror, followlinks)
             ^
SyntaxError: invalid syntax
Processing pywt/src/_pywt.pyx
Traceback (most recent call last):
  File "/tmp/check-manifest-6ae1__rt-sources/util/cythonize.py", line 188, in <module>
    main()
  File "/tmp/check-manifest-6ae1__rt-sources/util/cythonize.py", line 184, in main
    find_process_files(root_dir)
  File "/tmp/check-manifest-6ae1__rt-sources/util/cythonize.py", line 176, in find_process_files
    process(cur_dir, fromfile, tofile, function, hash_db)
  File "/tmp/check-manifest-6ae1__rt-sources/util/cythonize.py", line 153, in process
    processor_function(fromfile, tofile)
  File "/tmp/check-manifest-6ae1__rt-sources/util/cythonize.py", line 71, in process_pyx
    raise Exception('Cython failed')
Exception: Cython failed
Expanding templates
Cythonizing sources
Traceback (most recent call last):
  File "setup.py", line 221, in <module>
    setup_package()
  File "setup.py", line 213, in setup_package
    generate_cython()
  File "setup.py", line 122, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

More info:

$ strace -tttfo trace.log /tmp/py34/bin/check-manifest
$ strace-process-tree trace.log
30549 execve("/tmp/py34/bin/check-manifest", ["/tmp/py34/bin/check-manifest"], [/* 71 vars */])
  ├─30550 execve("/usr/bin/git", ["git", "ls-files", "-z"], [/* 71 vars */])
  ├─30551 execve("/tmp/py34/bin/python3.4", ["/tmp/py34/bin/python3.4", "setup.py", "sdist", "-d", "/tmp/check-manifest-kof8bflg-sdi"...], [/* 72 vars */])
  │   ├─30552 execve("/usr/bin/git", ["git", "rev-parse", "HEAD"], [/* 4 vars */])
  │   ├─30553 execve("/usr/bin/file", ["file", "/tmp/py34/bin/python3.4"], [/* 72 vars */])
  │   ├─30554 execve("/usr/bin/gcc-5.real", ["/usr/bin/gcc-5.real", "-print-multiarch"], [/* 73 vars */])
  │   ├─30555 execve("/tmp/py34/bin/python3.4", ["/tmp/py34/bin/python3.4", "/tmp/pywt/util/templating_src.py", "pywt"], [/* 72 vars */])
  │   └─30556 execve("/tmp/py34/bin/python3.4", ["/tmp/py34/bin/python3.4", "/tmp/pywt/util/cythonize.py", "pywt"], [/* 72 vars */])
  └─30557 execve("/tmp/py34/bin/python3.4", ["/tmp/py34/bin/python3.4", "setup.py", "sdist", "-d", "/tmp/check-manifest-jwqbnca0-sdi"...], [/* 72 vars */])
      ├─30558 execve("/usr/bin/file", ["file", "/tmp/py34/bin/python3.4"], [/* 72 vars */])
      ├─30559 execve("/usr/bin/gcc-5.real", ["/usr/bin/gcc-5.real", "-print-multiarch"], [/* 73 vars */])
      ├─30561 execve("/tmp/py34/bin/python3.4", ["/tmp/py34/bin/python3.4", "/tmp/check-manifest-6ae1__rt-sou"..., "pywt"], [/* 72 vars */])
      └─30562 execve("/tmp/py34/bin/python3.4", ["/tmp/py34/bin/python3.4", "/tmp/check-manifest-6ae1__rt-sou"..., "pywt"], [/* 72 vars */])
          └─30563 execve("/usr/bin/cython", ["cython", "--fast-fail", "-o", "_pywt.c", "_pywt.pyx"], [/* 72 vars */])

(strace-process-tree is https://github.com/mgedmin/scripts/blob/master/strace-process-tree)

The execve calls with /* 72 vars */ include the extra PYTHONPATH. Looks like /usr/bin/cython is unhappy to find /tmp/py34/lib/python3.4/ on PYTHONPATH.

[rgommers]

Thanks, that revert fixes it. That was a pretty ugly hack:)