Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 558: Incorporate review comments from python-dev posting #2038

Merged
merged 4 commits into from
Jul 21, 2021
Merged

PEP 558: Incorporate review comments from python-dev posting #2038

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
770e2a5
PEP 558: Incorporate text review comments from python-dev posting
ncoghlan Jul 18, 2021
ec58e7f
Replace GetReturnsCopy with GetKind & other updates
ncoghlan Jul 21, 2021
098e6f8
Merge remote-tracking branch 'origin/master' into pep-558-july-2021-r…
ncoghlan Jul 21, 2021
e0ae83d
Fix typo in new text
ncoghlan Jul 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 57 additions & 29 deletions pep-0558.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,14 @@ presence or absence of tracing functions.
In addition, it proposes that the following functions be added to the stable
Python C API/ABI::

typedef enum {
PyLocals_UNDEFINED = -1,
PyLocals_DIRECT_REFERENCE = 0,
PyLocals_SHALLOW_COPY = 1
} PyLocals_Kind;

PyLocals_Kind PyLocals_GetKind();
PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
PyObject * PyLocals_GetCopy();
PyObject * PyLocals_GetView();

Expand Down Expand Up @@ -261,12 +267,12 @@ CPython Implementation Changes
Summary of proposed implementation-specific changes
---------------------------------------------------

* Changes are made as neccessary to provide the updated Python level semantics
* Changes are made as necessary to provide the updated Python level semantics
* Two new functions are added to the stable ABI to replicate the updated
behaviour of the Python ``locals()`` builtin::

PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
PyLocals_Kind PyLocals_GetKind();
* One new function is added to the stable ABI to efficiently get a snapshot of
the local namespace in the running frame::

Expand Down Expand Up @@ -307,10 +313,6 @@ The implementation of the ``locals()`` builtin is modified to return a distinct
copy of the local namespace rather than a direct reference to the internal
dynamically updated snapshot returned by ``PyEval_GetLocals()``.

At least for now, this copied snapshot will continue to include any extra
key/value pairs injected via the ``PyEval_GetLocals()`` API, but that could
potentially change in a future release if that API is ever fully deprecated.


Resolving the issues with tracing mode behaviour
------------------------------------------------
Expand Down Expand Up @@ -378,7 +380,7 @@ cache is consistent with the current frame state.
(if it is not already populated), and then either return the relevant value
(if the key is found in either the ``fast_refs`` mapping or the ``f_locals``
dynamic snapshot stored on the frame), or else raise ``KeyError``. Variables
that are defined, but not yet bound raise ``KeyError`` (just as they're
that are defined but not currently bound raise ``KeyError`` (just as they're
omitted from the result of ``locals()``).

As the frame storage is always accessed directly, the proxy will automatically
Expand Down Expand Up @@ -432,19 +434,27 @@ To enable mimicking the behaviour of Python code, the stable C ABI would gain
the following new functions::

PyObject * PyLocals_Get();
int PyLocals_GetReturnsCopy();
PyLocals_Kind PyLocals_GetKind();

``PyLocals_Get()`` is directly equivalent to the Python ``locals()`` builtin.
It returns a new reference to the local namespace mapping for the active
Python frame at module and class scope, and when using ``exec()`` or ``eval()``.
It returns a shallow copy of the active namespace at
function/coroutine/generator scope.

``PyLocals_GetReturnsCopy()`` returns zero if ``PyLocals_Get()`` returns a
direct reference to the local namespace mapping, and a non-zero value if it
returns a shallow copy. This allows extension module code to determine the
potential impact of mutating the mapping returned by ``PyLocals_Get()`` without
needing access to the details of the running frame object.
``PyLocals_GetKind()`` returns a value from the newly defined ``PyLocals_Kind``
enum, with the following options being available:

* ``PyLocals_DIRECT_REFERENCE``: ``PyLocals_Get()`` returns a direct reference
to the local namespace for the running frame.
* ``PyLocals_SHALLOW_COPY``: ``PyLocals_Get()`` returns a shallow copy of the
local namespace for the running frame.
* ``PyLocals_UNDEFINED``: an error occurred (e.g. no active Python thread
state). A Python exception will be set if this value is returned.

This query API allows extension module code to determine the potential impact
of mutating the mapping returned by ``PyLocals_Get()`` without needing access
to the details of the running frame object.

To allow extension module code to behave consistently regardless of the active
Python scope, the stable C ABI would gain the following new functions::
Expand All @@ -460,6 +470,9 @@ copy.
``PyLocals_GetView()`` returns a new read-only mapping proxy instance for the
current locals namespace. This view immediately reflects all local variable
changes, independently of whether the running frame is optimised or not.
However, some operations (e.g. length checking, iteration, mapping equality
comparisons) may be subject to frame cache consistency issues on optimised
frames (as noted above when describing the behaviour of the fast locals proxy).

The existing ``PyEval_GetLocals()`` API will retain its existing behaviour in
CPython (mutable locals at class and module scope, shared dynamic snapshot
Expand All @@ -477,13 +490,13 @@ for the use case:
frame.
* Use ``PyLocals_Get()`` to exactly match the semantics of the Python level
``locals()`` builtin.
* Query ``PyLocals_GetReturnsCopy()`` explicitly to implement custom handling
* Query ``PyLocals_GetKind()`` explicitly to implement custom handling
(e.g. raising a meaningful exception) for scopes where ``PyLocals_Get()``
would return a shallow copy rather than granting read/write access to the
locals namespace.
* Use implementation specific APIs (e.g. ``PyObject_GetAttrString(frame, "f_locals")``)
if read/write access to the frame is required and ``PyLocals_GetReturnsCopy()``
is true.
if read/write access to the frame is required and ``PyLocals_GetKind()``
returns something other than ``PyLocals_DIRECT_REFERENCE``.


Changes to the public CPython C API
Expand All @@ -502,7 +515,8 @@ will be updated only in the following circumstance:
* any call to ``PyFrame_GetLocals()``, ``PyFrame_GetLocalsCopy()``,
``_PyFrame_BorrowLocals()``, ``PyFrame_FastToLocals()``, or
``PyFrame_FastToLocalsWithError()`` for the frame
* any call to the ``sync_frame_cache()`` method on a fast locals object
* retrieving the ``f_locals`` attribute from a Python level frame object
* any call to the ``sync_frame_cache()`` method on a fast locals proxy
referencing that frame
* any operation on a fast locals proxy object that requires the shared
mapping to be up to date on the underlying frame. In the initial reference
Expand All @@ -521,16 +535,17 @@ interpreter implementation detail)
The additions to the public CPython C API are the frame level enhancements
needed to support the stable C API/ABI updates::

PyLocals_Kind PyFrame_GetLocalsKind(frame);
PyObject * PyFrame_GetLocals(frame);
int PyFrame_GetLocalsReturnsCopy(frame);
PyObject * PyFrame_GetLocalsCopy(frame);
PyObject * PyFrame_GetLocalsView(frame);
PyObject * _PyFrame_BorrowLocals(frame);

``PyFrame_GetLocals(frame)`` is the underlying API for ``PyLocals_Get()``.

``PyFrame_GetLocalsReturnsCopy(frame)`` is the underlying API for
``PyLocals_GetReturnsCopy()``.
``PyFrame_GetLocalsKind(frame)`` is the underlying API for
``PyLocals_GetKind()``.

``PyFrame_GetLocals(frame)`` is the underlying API for ``PyLocals_Get()``.

``PyFrame_GetLocalsCopy(frame)`` is the underlying API for
``PyLocals_GetCopy()``.
Expand Down Expand Up @@ -559,7 +574,7 @@ implementation also exposes the following undocumented interfaces::

This type is what the reference implementation actually returns from
``PyObject_GetAttrString(frame, "f_locals")`` for optimized frames (i.e.
when ``PyFrame_GetLocalsReturnsCopy()`` returns true).
when ``PyFrame_GetLocalsKind()`` returns ``PyLocals_SHALLOW_COPY``).


Reducing the runtime overhead of trace hooks
Expand All @@ -571,7 +586,7 @@ the frame proxy read values directly from the frame instead of getting them
from the mapping.

As the new frame locals proxy type doesn't require separate data refresh steps,
this PEP incorporate's Victor Stinner's proposal to no longer implicitly call
this PEP incorporates Victor Stinner's proposal to no longer implicitly call
``PyFrame_FastToLocalsWithError()`` before calling trace hooks implemented in
Python.

Expand Down Expand Up @@ -693,7 +708,7 @@ namespace when that is what ``locals()`` returns.

This behaviour will have potential performance implications, especially
for functions with large numbers of local variables (e.g. if these functions
are called in a loop, calling ``gloabls()`` and ``locals()`` once before the
are called in a loop, calling ``globals()`` and ``locals()`` once before the
loop and then passing the namespace into the function explicitly will give the
same semantics and performance characteristics as the status quo, whereas
relying on the implicit default would create a new shallow copy of the local
Expand Down Expand Up @@ -828,7 +843,7 @@ into the following cases:
operation. This is the ``PyLocals_Get()`` API.
* needing to behave differently depending on whether writes to the result of
``PyLocals_Get()`` will be visible to Python code or not. This is handled by
the ``PyLocals_GetReturnsCopy()`` query API.
the ``PyLocals_GetKind()`` query API.
* always wanting a mutable namespace that has been pre-populated from the
current Python ``locals()`` namespace, but *not* wanting any changes to
be visible to Python code. This is the ``PyLocals_GetCopy()`` API.
Expand Down Expand Up @@ -858,11 +873,12 @@ Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in
PEP that attempted to avoid introducing such a proxy.

Thanks to Steve Dower and Petr Viktorin for asking that more attention be paid
to the developer experience of the proposed C API additions [8]_.
to the developer experience of the proposed C API additions [8,13]_.

Thanks to Mark Shannon for pushing for further simplification of the C level
API and semantics (and restarting discussion on the PEP in early 2021 after a
few years of inactivity).
API and semantics, as well as significant clarification of the PEP text (and for
restarting discussion on the PEP in early 2021 after a further year of
inactivity) [10,11,12].


References
Expand Down Expand Up @@ -895,6 +911,18 @@ References
.. [9] Disable automatic update of frame locals during tracing
(https://bugs.python.org/issue42197)

.. [10] python-dev thread: Resurrecting PEP 558 (Defined semantics for locals())
(https://mail.python.org/archives/list/python-dev@python.org/thread/TUQOEWQSCQZPUDV2UFFKQ3C3I4WGFPAJ/)

.. [11] python-dev thread: Comments on PEP 558
(https://mail.python.org/archives/list/python-dev@python.org/thread/A3UN4DGBCOB45STE6AQBITJFW6UZE43O/)

.. [12] python-dev thread: More comments on PEP 558
(https://mail.python.org/archives/list/python-dev@python.org/thread/7TKPMD5LHCBXGFUIMKDAUZELRH6EX76S/)

.. [13] Petr Viktorin's suggestion to use an enum for ``PyLocals_Get``'s behaviour
(https://mail.python.org/archives/list/python-dev@python.org/message/BTQUBHIVE766RPIWLORC5ZYRCRC4CEBL/)

Copyright
=========

Expand Down