Improve the constructors of AST nodes #105858

JelleZijlstra · 2023-06-16T15:28:14Z

Currently, the constructors for AST nodes accept arbitrary keyword arguments and don't enforce any value:

>>> node=ast.FunctionDef(what="is this")
>>> node.what
'is this'
>>> node.name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'FunctionDef' object has no attribute 'name'

Problems with the current situation:

To make a correct AST node, you have to pass every attribute, including ones that could sensibly default to an empty list
You cannot rely on attributes like name being present
It's possible to create useless, broken AST nodes like the above
Introspection tools can't easily tell what the signature of the constructor is supposed to be
Adding new fields to an AST node causes various compatibility issues (PEP-695: Potentially breaking changes made to __match_args__ attributes of AST nodes #104799)

Proposed solution for 3.13:

Default all fields to some sensible value if they are not provided to the constructor: an empty list for list fields, and None for other fields.
Emit a DeprecationWarning if the constructor is passed arguments that it doesn't recognize (e.g., what above). In 3.15, this will raise an error.
Add a __text_signature__ to the AST classes indicating the expected signature.

Linked PRs

The text was updated successfully, but these errors were encountered:

furkanonder · 2023-06-16T20:19:17Z

CC: @isidentical

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0> Known problems: - Subclasses of AST nodes don't work properly, because we don't look up __annotations__ on the right class. - Unpickling throws DeprecationWarnings, probably because of how we construct the unpickled object. Need to think more about how to handle those cases.

brandtbucher · 2023-06-20T17:58:25Z

Yes please! This has been such a pain point for me.

One note: we should probably only default optional fields (?) to None, and require everything else.

JelleZijlstra · 2023-06-20T18:02:45Z

One note: we should probably only default optional fields (?) to None, and require everything else.

Agree, I ended up doing that in my draft PR. Omitting a required field (e.g. FunctionDef.name) will raise a DeprecationWarning, as will passing a bogus field.

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>

JelleZijlstra · 2024-02-28T02:14:03Z

Merged!

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>

JelleZijlstra · 2024-02-28T02:54:38Z

Reopening as I broke all the buildbots

We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g., NoneType) are already exposed as internal APIs.

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>

) We now use these in the AST parsing code after gh-105880. A few comparable types (e.g., NoneType) are already exposed as internal APIs.

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>

…ythonGH-116025) We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g., NoneType) are already exposed as internal APIs.

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>

…ythonGH-116025) We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g., NoneType) are already exposed as internal APIs.

hroncok · 2024-03-26T17:27:56Z

There might be a regression: #117266

Demonstration: >>> ast.FunctionDef.__annotations__ {'name': <class 'str'>, 'args': <class 'ast.arguments'>, 'body': list[ast.stmt], 'decorator_list': list[ast.expr], 'returns': ast.expr | None, 'type_comment': str | None, 'type_params': list[ast.type_param]} >>> ast.FunctionDef() <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'name'. This will become an error in Python 3.15. <stdin>:1: DeprecationWarning: FunctionDef.__init__ missing 1 required positional argument: 'args'. This will become an error in Python 3.15. <ast.FunctionDef object at 0x101959460> >>> node = ast.FunctionDef(name="foo", args=ast.arguments()) >>> node.decorator_list [] >>> ast.FunctionDef(whatever="you want", name="x", args=ast.arguments()) <stdin>:1: DeprecationWarning: FunctionDef.__init__ got an unexpected keyword argument 'whatever'. Support for arbitrary keyword arguments is deprecated and will be removed in Python 3.15. <ast.FunctionDef object at 0x1019581f0>

…ythonGH-116025) We now use these in the AST parsing code after pythongh-105880. A few comparable types (e.g., NoneType) are already exposed as internal APIs.

JelleZijlstra added type-feature A feature request or enhancement topic-parser labels Jun 16, 2023

This was referenced Jun 16, 2023

gh-104799: PEP 695 backward compatibility for ast.unparse #105846

Merged

PEP-695: Potentially breaking changes made to __match_args__ attributes of AST nodes #104799

Closed

bedevere-bot mentioned this issue Jun 17, 2023

gh-105858: Improve AST node constructors #105880

Merged

iritkatriel added the stdlib Python modules in the Lib dir label Nov 27, 2023

JelleZijlstra closed this as completed Feb 28, 2024

JelleZijlstra mentioned this issue Feb 28, 2024

Omit optional fields from ast.dump() #116023

Closed

JelleZijlstra reopened this Feb 28, 2024

bedevere-app bot mentioned this issue Feb 28, 2024

gh-105858: Expose some union-related objects as internal APIs #116025

Merged

encukou pushed a commit that referenced this issue Feb 28, 2024

gh-105858: Expose some union-related objects as internal APIs (GH-116025

d53560d

) We now use these in the AST parsing code after gh-105880. A few comparable types (e.g., NoneType) are already exposed as internal APIs.

JelleZijlstra closed this as completed Feb 28, 2024

JelleZijlstra mentioned this issue May 9, 2024

Default expr_context parameters in ast to Load() #118851

Closed

JelleZijlstra mentioned this issue May 19, 2024

Default values for AST Nodes #84162

Closed

picnixz mentioned this issue Jul 1, 2024

gh-121141: add support for copy.replace to AST nodes #121162

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the constructors of AST nodes #105858

Improve the constructors of AST nodes #105858

JelleZijlstra commented Jun 16, 2023 •

edited by bedevere-app bot

Loading

furkanonder commented Jun 16, 2023

brandtbucher commented Jun 20, 2023

JelleZijlstra commented Jun 20, 2023 •

edited

Loading

JelleZijlstra commented Feb 28, 2024

JelleZijlstra commented Feb 28, 2024

hroncok commented Mar 26, 2024

Improve the constructors of AST nodes #105858

Improve the constructors of AST nodes #105858

Comments

JelleZijlstra commented Jun 16, 2023 • edited by bedevere-app bot Loading

Linked PRs

furkanonder commented Jun 16, 2023

brandtbucher commented Jun 20, 2023

JelleZijlstra commented Jun 20, 2023 • edited Loading

JelleZijlstra commented Feb 28, 2024

JelleZijlstra commented Feb 28, 2024

hroncok commented Mar 26, 2024

JelleZijlstra commented Jun 16, 2023 •

edited by bedevere-app bot

Loading

JelleZijlstra commented Jun 20, 2023 •

edited

Loading