Merge remote-tracking branch 'upstream/main' into pr_19211

.github/workflows/jit.yml

@@ -150,3 +150,21 @@ jobs:
             ./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '--enable-optimizations ' }} --build=x86_64-linux-gnu --host="$HOST" --with-build-python=../build/bin/python3 --with-pkg-config=no ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptc=no ac_cv_file__dev_ptmx=yes
           make all --jobs 4
           ./python -m test --ignorefile=Tools/jit/ignore-tests-emulated-linux.txt --multiprocess 0 --timeout 4500 --verbose2 --verbose3
+
+  jit-with-disabled-gil:
+    name: Free-Threaded (Debug)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Build with JIT enabled and GIL disabled
+        run: |
+          sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" ./llvm.sh 18
+          export PATH="$(llvm-config-18 --bindir):$PATH"
+          ./configure --enable-experimental-jit --with-pydebug --disable-gil
+          make all --jobs 4
+      - name: Run tests
+        run: |
+          ./python -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3

.github/workflows/mypy.yml

@@ -34,6 +34,7 @@ concurrency:
 jobs:
   mypy:
     strategy:
+      fail-fast: false
       matrix:
         target: [
           "Lib/_pyrepl",

Doc/c-api/frame.rst

@@ -121,17 +121,18 @@ See also :ref:`Reflection <reflection>`.
 .. c:function:: PyObject* PyFrame_GetLocals(PyFrameObject *frame)
 
    Get the *frame*'s :attr:`~frame.f_locals` attribute.
-   If the frame refers to a function or comprehension, this returns
-   a write-through proxy object that allows modifying the locals.
-   In all other cases (classes, modules) it returns the :class:`dict`
-   representing the frame locals directly.
+   If the frame refers to an :term:`optimized scope`, this returns a
+   write-through proxy object that allows modifying the locals.
+   In all other cases (classes, modules, :func:`exec`, :func:`eval`) it returns
+   the mapping representing the frame locals directly (as described for
+   :func:`locals`).
 
    Return a :term:`strong reference`.
 
    .. versionadded:: 3.11
 
    .. versionchanged:: 3.13
-      Return a proxy object for functions and comprehensions.
+      As part of :pep:`667`, return a proxy object for optimized scopes.
 
 
 .. c:function:: int PyFrame_GetLineNumber(PyFrameObject *frame)

Doc/glossary.rst

@@ -889,6 +889,15 @@ Glossary
       (methods).  Also the ultimate base class of any :term:`new-style
       class`.
 
+   optimized scope
+      A scope where target local variable names are reliably known to the
+      compiler when the code is compiled, allowing optimization of read and
+      write access to these names. The local namespaces for functions,
+      generators, coroutines, comprehensions, and generator expressions are
+      optimized in this fashion. Note: most interpreter optimizations are
+      applied to all scopes, only those relying on a known set of local
+      and nonlocal variable names are restricted to optimized scopes.
+
    package
       A Python :term:`module` which can contain submodules or recursively,
       subpackages.  Technically, a package is a Python module with a

Doc/library/code.rst

@@ -18,9 +18,9 @@ build applications which provide an interactive interpreter prompt.
    This class deals with parsing and interpreter state (the user's namespace); it
    does not deal with input buffering or prompting or input file naming (the
    filename is always passed in explicitly). The optional *locals* argument
-   specifies the dictionary in which code will be executed; it defaults to a newly
-   created dictionary with key ``'__name__'`` set to ``'__console__'`` and key
-   ``'__doc__'`` set to ``None``.
+   specifies a mapping to use as the namespace in which code will be executed;
+   it defaults to a newly created dictionary with key ``'__name__'`` set to
+   ``'__console__'`` and key ``'__doc__'`` set to ``None``.
 
 
 .. class:: InteractiveConsole(locals=None, filename="<console>", local_exit=False)

Doc/library/dataclasses.rst

@@ -616,7 +616,8 @@ methods will raise a :exc:`FrozenInstanceError` when invoked.
 There is a tiny performance penalty when using ``frozen=True``:
 :meth:`~object.__init__` cannot use simple assignment to initialize fields, and
 must use :meth:`!object.__setattr__`.
-.. Make sure to not remove "object" from "object.__setattr__" in the above markup
+
+.. Make sure to not remove "object" from "object.__setattr__" in the above markup!
 
 .. _dataclasses-inheritance:
 

Doc/library/functions.rst

@@ -543,18 +543,19 @@ are always available.  They are listed here in alphabetical order.
 
    The *expression* argument is parsed and evaluated as a Python expression
    (technically speaking, a condition list) using the *globals* and *locals*
-   dictionaries as global and local namespace.  If the *globals* dictionary is
+   mappings as global and local namespace.  If the *globals* dictionary is
    present and does not contain a value for the key ``__builtins__``, a
    reference to the dictionary of the built-in module :mod:`builtins` is
    inserted under that key before *expression* is parsed.  That way you can
    control what builtins are available to the executed code by inserting your
    own ``__builtins__`` dictionary into *globals* before passing it to
-   :func:`eval`.  If the *locals* dictionary is omitted it defaults to the
-   *globals* dictionary.  If both dictionaries are omitted, the expression is
+   :func:`eval`.  If the *locals* mapping is omitted it defaults to the
+   *globals* dictionary.  If both mappings are omitted, the expression is
    executed with the *globals* and *locals* in the environment where
-   :func:`eval` is called.  Note, *eval()* does not have access to the
+   :func:`eval` is called.  Note, *eval()* will only have access to the
    :term:`nested scopes <nested scope>` (non-locals) in the enclosing
-   environment.
+   environment if they are already referenced in the scope that is calling
+   :func:`eval` (e.g. via a :keyword:`nonlocal` statement).
 
    Example:
 
@@ -587,6 +588,11 @@ are always available.  They are listed here in alphabetical order.
 
       The *globals* and *locals* arguments can now be passed as keywords.
 
+   .. versionchanged:: 3.13
+
+      The semantics of the default *locals* namespace have been adjusted as
+      described for the :func:`locals` builtin.
+
 .. index:: pair: built-in function; exec
 
 .. function:: exec(source, /, globals=None, locals=None, *, closure=None)
@@ -612,9 +618,15 @@ are always available.  They are listed here in alphabetical order.
 
    .. note::
 
-      Most users should just pass a *globals* argument and never *locals*.
-      If exec gets two separate objects as *globals* and *locals*, the code
-      will be executed as if it were embedded in a class definition.
+      When ``exec`` gets two separate objects as *globals* and *locals*, the
+      code will be executed as if it were embedded in a class definition. This
+      means functions and classes defined in the executed code will not be able
+      to access variables assigned at the top level (as the "top level"
+      variables are treated as class variables in a class definition).
+      Passing a :class:`collections.ChainMap` instance as *globals* allows name
+      lookups to be chained across multiple mappings without triggering this
+      behaviour. Values assigned to top level names in the executed code can be
+      retrieved by passing an empty dictionary as the first entry in the chain.
 
    If the *globals* dictionary does not contain a value for the key
    ``__builtins__``, a reference to the dictionary of the built-in module
@@ -635,7 +647,7 @@ are always available.  They are listed here in alphabetical order.
    .. note::
 
       The built-in functions :func:`globals` and :func:`locals` return the current
-      global and local dictionary, respectively, which may be useful to pass around
+      global and local namespace, respectively, which may be useful to pass around
       for use as the second and third argument to :func:`exec`.
 
    .. note::
@@ -651,6 +663,11 @@ are always available.  They are listed here in alphabetical order.
 
       The *globals* and *locals* arguments can now be passed as keywords.
 
+   .. versionchanged:: 3.13
+
+      The semantics of the default *locals* namespace have been adjusted as
+      described for the :func:`locals` builtin.
+
 
 .. function:: filter(function, iterable)
 
@@ -1056,39 +1073,51 @@ are always available.  They are listed here in alphabetical order.
     variable names as the keys, and their currently bound references as the
     values.
 
-    At module scope, as well as when using ``exec()`` or ``eval()`` with a
-    single namespace, this function returns the same namespace as ``globals()``.
+    At module scope, as well as when using :func:`exec` or :func:`eval` with
+    a single namespace, this function returns the same namespace as
+    :func:`globals`.
 
     At class scope, it returns the namespace that will be passed to the
     metaclass constructor.
 
     When using ``exec()`` or ``eval()`` with separate local and global
-    namespaces, it returns the local namespace passed in to the function call.
+    arguments, it returns the local namespace passed in to the function call.
 
     In all of the above cases, each call to ``locals()`` in a given frame of
     execution will return the *same* mapping object. Changes made through
-    the mapping object returned from ``locals()`` will be visible as bound,
-    rebound, or deleted local variables, and binding, rebinding, or deleting
-    local variables will immediately affect the contents of the returned mapping
-    object.
-
-    At function scope (including for generators and coroutines), each call to
-    ``locals()`` instead returns a fresh dictionary containing the current
-    bindings of the function's local variables and any nonlocal cell references.
-    In this case, name binding changes made via the returned dict are *not*
-    written back to the corresponding local variables or nonlocal cell
-    references, and binding, rebinding, or deleting local variables and nonlocal
-    cell references does *not* affect the contents of previously returned
-    dictionaries.
+    the mapping object returned from ``locals()`` will be visible as assigned,
+    reassigned, or deleted local variables, and assigning, reassigning, or
+    deleting local variables will immediately affect the contents of the
+    returned mapping object.
+
+    In an :term:`optimized scope` (including functions, generators, and
+    coroutines), each call to ``locals()`` instead returns a fresh dictionary
+    containing the current bindings of the function's local variables and any
+    nonlocal cell references. In this case, name binding changes made via the
+    returned dict are *not* written back to the corresponding local variables
+    or nonlocal cell references, and assigning, reassigning, or deleting local
+    variables and nonlocal cell references does *not* affect the contents
+    of previously returned dictionaries.
+
+    Calling ``locals()`` as part of a comprehension in a function, generator, or
+    coroutine is equivalent to calling it in the containing scope, except that
+    the comprehension's initialised iteration variables will be included. In
+    other scopes, it behaves as if the comprehension were running as a nested
+    function.
+
+    Calling ``locals()`` as part of a generator expression is equivalent to
+    calling it in a nested generator function.
+
+   .. versionchanged:: 3.12
+      The behaviour of ``locals()`` in a comprehension has been updated as
+      described in :pep:`709`.
 
    .. versionchanged:: 3.13
-      In previous versions, the semantics of mutating the mapping object
-      returned from this function were formally undefined. In CPython
-      specifically, the mapping returned at function scope could be
-      implicitly refreshed by other operations, such as calling ``locals()``
-      again. Obtaining the legacy CPython behaviour now requires explicit
-      calls to update the initially returned dictionary with the results
-      of subsequent calls to ``locals()``.
+      As part of :pep:`667`, the semantics of mutating the mapping objects
+      returned from this function are now defined. The behavior in
+      :term:`optimized scopes <optimized scope>` is now as described above.
+      Aside from being defined, the behaviour in other scopes remains
+      unchanged from previous versions.
 
 
 .. function:: map(function, iterable, *iterables)
@@ -1975,14 +2004,18 @@ are always available.  They are listed here in alphabetical order.
    :attr:`~object.__dict__` attributes (for example, classes use a
    :class:`types.MappingProxyType` to prevent direct dictionary updates).
 
-   Without an argument, :func:`vars` acts like :func:`locals`.  Note, the
-   locals dictionary is only useful for reads since updates to the locals
-   dictionary are ignored.
+   Without an argument, :func:`vars` acts like :func:`locals`.
 
    A :exc:`TypeError` exception is raised if an object is specified but
    it doesn't have a :attr:`~object.__dict__` attribute (for example, if
    its class defines the :attr:`~object.__slots__` attribute).
 
+   .. versionchanged:: 3.13
+
+      The result of calling this function without an argument has been
+      updated as described for the :func:`locals` builtin.
+
+
 .. function:: zip(*iterables, strict=False)
 
    Iterate over several iterables in parallel, producing tuples with an item

Doc/library/pdb.rst

@@ -123,6 +123,11 @@ The typical usage to inspect a crashed program is::
    0
    (Pdb)
 
+.. versionchanged:: 3.13
+   The implementation of :pep:`667` means that name assignments made via ``pdb``
+   will immediately affect the active scope, even when running inside an
+   :term:`optimized scope`.
+
 
 The module defines the following functions; each enters the debugger in a
 slightly different way:
@@ -579,18 +584,17 @@ can be overridden by the local file.
 
 .. pdbcommand:: interact
 
-   Start an interactive interpreter (using the :mod:`code` module) whose global
-   namespace contains all the (global and local) names found in the current
-   scope. Use ``exit()`` or ``quit()`` to exit the interpreter and return to
-   the debugger.
+   Start an interactive interpreter (using the :mod:`code` module) in a new
+   global namespace initialised from the local and global namespaces for the
+   current scope. Use ``exit()`` or ``quit()`` to exit the interpreter and
+   return to the debugger.
 
    .. note::
 
-      Because interact creates a new global namespace with the current global
-      and local namespace for execution, assignment to variables will not
-      affect the original namespaces.
-      However, modification to the mutable objects will be reflected in the
-      original namespaces.
+      As ``interact`` creates a new dedicated namespace for code execution,
+      assignments to variables will not affect the original namespaces.
+      However, modifications to any referenced mutable objects will be reflected
+      in the original namespaces as usual.
 
    .. versionadded:: 3.2
 

Doc/library/profile.rst

@@ -234,7 +234,7 @@ functions:
 .. function:: runctx(command, globals, locals, filename=None, sort=-1)
 
    This function is similar to :func:`run`, with added arguments to supply the
-   globals and locals dictionaries for the *command* string. This routine
+   globals and locals mappings for the *command* string. This routine
    executes::
 
       exec(command, globals, locals)

Doc/library/socket.rst

@@ -1589,7 +1589,8 @@ to sockets.
    Return a :term:`file object` associated with the socket.  The exact returned
    type depends on the arguments given to :meth:`makefile`.  These arguments are
    interpreted the same way as by the built-in :func:`open` function, except
-   the only supported *mode* values are ``'r'`` (default), ``'w'`` and ``'b'``.
+   the only supported *mode* values are ``'r'`` (default), ``'w'``, ``'b'``, or
+   a combination of those.
 
    The socket must be in blocking mode; it can have a timeout, but the file
    object's internal buffer may end up in an inconsistent state if a timeout

Doc/library/traceback.rst

@@ -473,7 +473,7 @@ in a :ref:`traceback <traceback-objects>`.
    attribute accessed (which also happens when casting it to a :class:`tuple`).
    :attr:`~FrameSummary.line` may be directly provided, and will prevent line
    lookups happening at all. *locals* is an optional local variable
-   dictionary, and if supplied the variable representations are stored in the
+   mapping, and if supplied the variable representations are stored in the
    summary for later display.
 
    :class:`!FrameSummary` instances have the following attributes:

Doc/reference/datamodel.rst

@@ -1349,7 +1349,7 @@ Special read-only attributes
    * - .. attribute:: frame.f_locals
      - The dictionary used by the frame to look up
        :ref:`local variables <naming>`.
-       If the frame refers to a function or comprehension,
+       If the frame refers to an :term:`optimized scope`,
        this may return a write-through proxy object.
 
        .. versionchanged:: 3.13

Doc/tutorial/inputoutput.rst

@@ -37,16 +37,23 @@ printing space-separated values. There are several ways to format output.
 * The :meth:`str.format` method of strings requires more manual
   effort.  You'll still use ``{`` and ``}`` to mark where a variable
   will be substituted and can provide detailed formatting directives,
-  but you'll also need to provide the information to be formatted.
+  but you'll also need to provide the information to be formatted. In the following code
+  block there are two examples of how to format variables:
+
 
   ::
 
      >>> yes_votes = 42_572_654
-     >>> no_votes = 43_132_495
-     >>> percentage = yes_votes / (yes_votes + no_votes)
+     >>> total_votes = 85_705_149
+     >>> percentage = yes_votes / total_votes
      >>> '{:-9} YES votes  {:2.2%}'.format(yes_votes, percentage)
      ' 42572654 YES votes  49.67%'
 
+  Notice how the ``yes_votes`` are padded with spaces and a negative sign only for negative numbers.
+  The example also prints ``percentage`` multiplied by 100, with 2 decimal
+  places and followed by a percent sign (see :ref:`formatspec` for details).
+
+
 * Finally, you can do all the string handling yourself by using string slicing and
   concatenation operations to create any layout you can imagine.  The
   string type has some methods that perform useful operations for padding
@@ -197,7 +204,12 @@ notation. ::
    Jack: 4098; Sjoerd: 4127; Dcab: 8637678
 
 This is particularly useful in combination with the built-in function
-:func:`vars`, which returns a dictionary containing all local variables.
+:func:`vars`, which returns a dictionary containing all local variables::
+
+   >>> table = {k: str(v) for k, v in vars().items()}
+   >>> message = " ".join([f'{k}: ' + '{' + k +'};' for k in table.keys()])
+   >>> print(message.format(**table))
+   __name__: __main__; __doc__: None; __package__: None; __loader__: ...
 
 As an example, the following lines produce a tidily aligned
 set of columns giving integers and their squares and cubes::