"Failed to find a python interpreter in the .data section"/"Failed to get line number"

[anntzer]

I'm getting the error in the title above when trying to profile a program that import the colorcet (https://github.com/pyviz/colorcet) package:

$ py-spy -- python -c 'import colorcet'

Error: Failed to find a python interpreter in the .data section

or occasionally

Error: Failed to get line number
Reason: Failed to copy line number table
Reason: Refusing to copy 42370 bytes

This is on fedora with a conda python 3.7, py-spy 0.1.11 or 0.2.0dev3 (and colorcet 2.0.2, fwiw).

I have found a similar issue in #1, but I'm not sure it's really the same thing? The OS is different, and you say in #1 that its root cause has been fixed.

Thanks for the great package :)

[benfred]

Hmm - I can't seem to replicate. Can you enable logging and paste the output here?

RUST_LOG=debug  py-spy -- python -c 'import colorcet; import time; time.sleep(10)'

Note that if the program immediately exits (like just importing a package and then returning) py-spy might not initialize in time to profile it - which is why I added the sleep call above. The errors returned in that case should be different though

[anntzer]

Output attached (with the sleep): log.txt

fwiw I cannot repro this either with an Arch Linux distro-packaged python.

[benfred]

Thanks for the bug report! I've managed to reproduce this and figure out what's happening here.

The problem is that the colorcet __init__.py is over 22K lines long, and it's mostly all at the module level rather than split into functions. This means the line number table is ~43K for the <module> scope since the table uses 2 bytes per line. The problem is that I had set a limit of 8K for the max here, assuming wrongly that no-one would have functions with more 4K lines (and not even thinking about the case of thousands of lines of constants being declared at the module scope).

This bug is also timing dependant - since as soon as the 'import colorcet' line we can initialize py-spy. The problem is that on the system I repro'ed on - this import takes > 500ms and we timeout the py-spy initialization process before then =(

I will have a fix in the next version of py-spy (this week? 0.2.0 is basically ready to go, I just need to update the documentation). The fix is trivial but the error logging really could be improved here.

[anntzer]

Thanks :)

[benfred]

fixed in v0.2.0