gh-86682: Adds sys._getframemodulename as a lightweight alternative to _getframe

[zooba]

Replaces _getframe calls in collections, doctest, enum, and typing modules

• Issue: Add function to get caller's name: sys._getframemodulename() #86682

[JelleZijlstra]

The typing change is fine, thanks for picking this up!

I might like your original version (sys._get_calling_module_name) better. It exposes less data, so it's easier to keep the interface if interpreter internals change.

[zooba]

Yeah, turns out we can't use __module__ reliably here, so I'm inclined to change it back and make it only return the module name (rather than encoding use of __globals__ on the caller's side).

[gvanrossum]

IMO the news file should mention the new function too.

I’d like a review by @markshannon .

I’m surprised the calling function isn’t obtainable, but the use cases are there, so no complaints about that.

[zooba]

I'd like a review from Mark, too.

The calling function is obtainable, even more easily than the module name. But in every case we already use this, we'd then have to go to func.__globals__['__name__'] and handle the cases where these are not consistent (__module__ is often <module> and not the actual module name).

If there's a use case for both, I'll happily add both, but right now, we don't need the function object for anything.

[gvanrossum]

Really need to get @markshannon to answer the question. But he’s on vacation this week.

[markshannon]

I'm thinking aloud here, so bear with me...

I don't think that _getframemodulename is a good name.
Not that _getframemodulename is a bad name for this API, maybe its just that the API is a bit odd?

It feels to me that this API is doing two things, getting the frame, then the module name. Why not split the parts?

frame = sys._getframe(n)
modulename = get_module_name(frame)

or, potentially

modulename = sys._getframe(n).f_modulename

Unfortunately this doesn't achieve the stated goal of removing (almost) all uses of sys._getframe(n)

Which leads us back to adding a function to get the caller function.
Functions already have a __module__ attribute, so we get

modulename = sys.getcallerfunc(n).__module__

Which seems cleaner to me.

[markshannon]

@zooba You state in an earlier comment that __module__ is not reliable.
It seems to me that we should fix that, rather than working around the bug.

[markshannon]

My cursory scan of the code suggests that __module__ == "<module>" only occurs in the repl, doctests and other source-less code.

[markshannon]

Even in the repl, it seems that functtions have __module__ == "main".

[zooba]

@zooba You state in an earlier comment that __module__ is not reliable.
It seems to me that we should fix that, rather than working around the bug.

I didn't trace down exactly why it wasn't being filled in correctly, but I'm willing to bet it's a much more complicated compatibility issue to change it. __module__ is essentially a public API, while _getframemodulename is not.

It's definitely a bit of an odd API though, I agree. That's why I happily tried returning the function object instead. Perhaps it was only doctests that didn't set it correctly?

Also keen on your thoughts on the FRAME_OWNED_BY_CSTACK flag in this comment, @markshannon. I think I've used it right, but couldn't convince myself that this is the right way to skip frames.

[markshannon]

You need to check for invalid frames when walking the frame stack.
You might want to factor out the walking the stack in sys._getframe() https://github.com/python/cpython/blob/main/Python/sysmodule.c#L1889

[markshannon]

OOI, what tests fail if you rely on __module__?

[zooba]

I've been looking into the __module__ issue, and it basically seems that <module> functions don't have __module__ set, which means anything at the top-level of a module won't work (e.g. virtually every typing, enum and dataclass definition). And this is the primary reason we use this, because anything that we'll be able to unpickle later on is going to have to be an importable name.

The only reliable workaround is to get it from f_globals, as the code currently does. Tracing through the code, I think it's probably best to get func_module first if it's available, as that should let code assign to func.__module__ and override what gets returned. Then we can fall back to f_globals['__module__'] if it's not on the function object (yet) and still match.

I added a better test to compare _getframe with _getframemodulename and updated the check to match what is used in _getframe (which looks like a fairly different condition, but I trust you).

[markshannon]

it basically seems that <module> functions don't have __module__ set

That sounds like a bug. Let's fix it rather than working around it.

[zooba]

Seems easy enough to fix, there's just a constructor that needs to pull __name__ out of globals.

Even so, I still don't see a reason why we'd return function objects. Our code either needs the module name, or else it needs actual frame information (e.g. current lineno). This is a private API, so if we need something different in the future we can always change it.

This is also why I made it not raise an error but return None, since everywhere we'd use it we would swap in a default value anyway (I was tempted to add a default argument, but we'd want to replace explicit None as well, so the semantics work better for us this way). We really never need the function object right now, at least not anywhere we don't also need the locals or lineno.

[markshannon]

Even so, I still don't see a reason why we'd return function objects.

Two main reasons:

• It is a simpler, cleaner API. It makes introspecting the frame stack for profilers and the like (that don't care about the values of local variables) more efficient, and less of an auditing issue.
• Because we can. Until 3.11, we couldn't implement it. I suspect that if we could have, some of use cases that you describe would have used it before now.

My feeling is that sys.getcallerfunc(n) would not need an audit hook. Would you agree?
Or is that an argument in favor of sys._getframemodulename()?

[zooba]

My feeling is that sys.getcallerfunc(n) would not need an audit hook. Would you agree?
Or is that an argument in favor of sys._getframemodulename()?

I started writing a reply earlier making the argument that only returning the name wouldn't require a red flag (I think it still deserves an event, but it's far less of a risk than if random code is unexpectedly doing _getframe). But I stopped because I don't actually have a strong argument, just a gut feel.

The most interesting thing you can do with a function object is mess with func_globals, which is the same as sys.modules[sys._getframemodulename()] anyway, and dict lookups don't get an audit event (though possibly we could special case sys.modules if needed... I'd do that as my own patch first before trying to upstream it).

But because we don't use it, it means instead of sys._getframemodulename() or 'default', we'd be writing getattr(sys._getframefunction(), '__module__', None) or 'default' everywhere. That, plus YAGNI, is my argument for only returning the name (or None).

[zooba]

If there are no other concerns raised I'd like to merge this sometime this week.

[serhiy-storchaka]

PyDict_GetItem is broken by design and should not be used. See #106033.