C API: Investigate how the PyTypeObject members can be removed from the public C API

[vstinner]

I propose to investigate an incompatible C API change: make the PyTypeObject and PyHeapTypeObject structures opaque, remove their members from the public C API (move them to the internal C API). We have to investigate how it's outside in 3rd party C extensions (ex: Cython, pybind11, etc.), and design a smooth migration plan.

The PyTypeObject structure is exposed as part of the public Python C API. For example, Py_TYPE(obj)->tp_name directly gets a type name (as a UTF-8 encoded byte string, char*).

The PyTypeObject members are NOT part of the limited C API (PEP 384).

---

In Python 3.9 (2020), I reworked the C API to avoid accessing directly PyTypeObject members at the ABI level: issue #84351. For example, Python 3.8 implements PyObject_IS_GC() as a macro:

/* Test if an object has a GC head */
#define PyObject_IS_GC(o) \
    (PyType_IS_GC(Py_TYPE(o)) \
     && (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))

whereas Python 3.9 only provides an opaque function call:

/* Test if an object implements the garbage collector protocol */
PyAPI_FUNC(int) PyObject_IS_GC(PyObject *obj);

At the ABI level, the direct access to the PyTypeObject.tp_is_gc member became an opaque function call.

---

Changing PyTypeObject API and ABI caused a lot of troubles in the past. Example:

• issue C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed #81431: Cython broken by PyTypeObject.tp_print removal -- related to PyTypeObject.tp_vectorcall_offset addition
• issue Remove cross-version binary compatibility #76569: ABI issue with the addition of PyTypeObject.tp_finalize (PEP 442). Is Py_TPFLAGS_HAVE_FINALIZE needed for ABI compatibility? See also the python-dev thread.
• issue Remove Py_TPFLAGS_HAVE_VERSION_TAG flag? #86913: remove Py_TPFLAGS_HAVE_VERSION_TAG flag (ABI compatibility)
• issue Documentation inconsistency with the stable ABI #91271: Documentation inconsistency with the stable ABI (PyTypeObject)
• The PEP 620 uses PyTypeObject incompatible changes in its rationale.

---

For many years, there is a work-in-progress to convert all Python built-in types and types of stdlib extensions from static types to heap types. See for example issue #84258 and PEP 630.

The API and ABI for heap type was also enhanced over the years. Examples:

• Python 3.12 adds PyType_FromMetaclass()
• Add the buffer protocol to the limited C API (to PyType_FromSpec()): issue Limited API support for Py_buffer #89622
• issue [C API] PyType_GetSlot cannot get tp_name #86201: get tp_name with PyType_GetSlot()

---

In the past, other structure members were removed:

• PyInterpreterState: Python 3.8
• PyGC_Head: Python 3.9
• PyFrameObject: Python 3.11

The work was also prepared for:

• PyObject: issue [C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type #83754, Python 3.10 and Python 3.11, add Py_SET_REFCNT() and Py_SET_TYPE()
• PyTheadState: issue [C API] Prepare the C API to make PyThreadState opaque: add getter functions #84128, Python 3.11

[vstinner]

For Py_TYPE(obj)->tp_name pattern, I proposed adding %T format to PyUnicode_FromFormat(): issue GH-78776. See related python-dev discussion. The change was reverted.

[ronaldoussoren]

One thing I don't know how to do with the spec-based API is adding additional C fields to a type without exposing them to Python. That is something I have a use-case for in PyObjC and can currently do by embedding a PyTypeObject (or rather PyHeapTypeObject in a larger struct). Storing the data out of line, for example in some kind of hash table, likely results in additional overhead in what is in the end fairly hot code.

I want to try moving PyObjC to the spec-based API during the 3.13 release cycle if I have enough free time, but that's a fairly big "if". The longer term goal with that is to support sub interpreters.

[vstinner]

@ronaldoussoren:

One thing I don't know how to do with the spec-based API is adding additional C fields to a type without exposing them to Python

Isn't it the purpose of PEP 697 – Limited C API for Extending Opaque Types? It's said to be fully implemented in Python 3.12.

cc @encukou

[vstinner]

Changing PyTypeObject and PyHeapTypeObject API is complex because they have 112 members in total.

In general, most members can be get with PyType_GetSlot() and many can be get and set with PyObject_GetAttrString() and PyObject_SetAttrString(). I listed members with more specialized way to access them.

Attributes (read-only):

• tp_base: get "__base__" attribute
• tp_basicsize: _PyObject_SIZE()
• tp_doc: get/set "__doc__" attribute
• tp_flags: PyType_GetFlags(), PyType_HasFeature()
• tp_itemsize: _PyObject_VAR_SIZE()
• tp_vectorcall_offset: PyVectorcall_Function(), PyObject_Vectorcall()

Attributes:

• tp_bases: get/set "__bases__" attribute, _PyType_GetBases()
• tp_name: PyType_GetName(), PyType_GetQualName(), set "__name__" attribute

Internal functions for the GC:

• tp_clear: used indirectly by Py_Finalize()
• tp_traverse: call gc.get_objects() function

Internal attributes and methods:

• tp_dealloc: _Py_Dealloc(), used by Py_DECREF()
• tp_members
• tp_mro (read-only): PyObject_GetAttrString(type, "__mro__")
• tp_subclasses: call __subclasses__() method, _PyType_GetSubclasses()
• tp_version_tag: method cache, PyUnstable_Type_AssignVersionTag(), _PyType_Lookup()
• tp_watched: PyType_Watch(), PyType_Unwatch()
• tp_weaklist: used by tp_weaklistoffset
• tp_weaklistoffset: get "__weakrefoffset__" attribute
• tp_vectorcall: used by tp_vectorcall_offset

Async methods:

• tp_as_async.am_await: call __await__() method, _PyCoro_GetAwaitableIter()
• tp_as_async.am_aiter: call __aiter__() method
• tp_as_async.am_anext: call __anext__() method
• tp_as_async.am_send: PyIter_Send()

Buffer methods:

• tp_as_buffer.bf_getbuffer: PyObject_GetBuffer()``
• tp_as_buffer.bf_releasebuffer: PyBuffer_Release()

Mapping methods:

• tp_as_mapping.mp_length: PyMapping_Size()
• tp_as_mapping.mp_subscript: PyObject_GetItem()
• tp_as_mapping.mp_ass_subscript: PyObject_SetItem()

Number methods:

• tp_as_number.nb_add: PyNumber_Add()
• tp_as_number.nb_subtract: PyNumber_Subtract()
• tp_as_number.nb_multiply: PyNumber_Multiply()
• tp_as_number.nb_remainder: PyNumber_Remainder()
• tp_as_number.nb_divmod: PyNumber_Divmod()
• tp_as_number.nb_power: PyNumber_Power()
• tp_as_number.nb_negative: PyNumber_Negative()
• tp_as_number.nb_positive: PyNumber_Positive()
• tp_as_number.nb_absolute: PyNumber_Absolute()
• tp_as_number.nb_bool: PyObject_IsTrue()
• tp_as_number.nb_invert: PyNumber_Invert()
• tp_as_number.nb_lshift: PyNumber_Lshift()
• tp_as_number.nb_rshift: PyNumber_Rshift()
• tp_as_number.nb_and: PyNumber_And()
• tp_as_number.nb_xor: PyNumber_Xor()
• tp_as_number.nb_or: PyNumber_Or()
• tp_as_number.nb_int: PyNumber_Long()
• tp_as_number.nb_float: PyNumber_Float()
• tp_as_number.nb_inplace_add: PyNumber_InPlaceAdd()
• tp_as_number.nb_inplace_subtract: PyNumber_InPlaceSubtract()
• tp_as_number.nb_inplace_multiply: PyNumber_InPlaceMultiply()
• tp_as_number.nb_inplace_remainder: PyNumber_InPlaceRemainder()
• tp_as_number.nb_inplace_power: PyNumber_InPlacePower()
• tp_as_number.nb_inplace_lshift: PyNumber_InPlaceLshift()
• tp_as_number.nb_inplace_rshift: PyNumber_InPlaceRshift()
• tp_as_number.nb_inplace_and: PyNumber_InPlaceAnd()
• tp_as_number.nb_inplace_xor: PyNumber_InPlaceXor()
• tp_as_number.nb_inplace_or: PyNumber_InPlaceOr()
• tp_as_number.nb_floor_divide: PyNumber_FloorDivide()
• tp_as_number.nb_true_divide: PyNumber_TrueDivide()
• tp_as_number.nb_inplace_floor_divide: PyNumber_InPlaceFloorDivide()
• tp_as_number.nb_inplace_true_divide: PyNumber_InPlaceTrueDivide()
• tp_as_number.nb_index: PyNumber_Index()
• tp_as_number.nb_matrix_multiply: PyNumber_MatrixMultiply()
• tp_as_number.nb_inplace_matrix_multiply: PyNumber_InPlaceMatrixMultiply()

Sequence methods:

• tp_as_sequence.sq_length: PySequence_Size()
• tp_as_sequence.sq_concat: PySequence_Concat()
• tp_as_sequence.sq_repeat: PySequence_Repeat()
• tp_as_sequence.sq_item: PySequence_GetItem()
• tp_as_sequence.sq_ass_item: PySequence_SetItem()
• tp_as_sequence.sq_contains: PySequence_Contains()
• tp_as_sequence.sq_inplace_concat: PySequence_InPlaceConcat()
• tp_as_sequence.sq_inplace_repeat: PySequence_InPlaceRepeat()

Attribute lookup, get/attr instance attributes:

• tp_descr_get
• tp_descr_set
• tp_dict: get "__dict__" attribute, PyObject_GenericGetAttr(), PyObject_GenericSetDict(), _PyObject_GetDictPtr(). See also Python 3.10 Py_TPFLAGS_MANAGED_DICT.
• tp_dictoffset: get "__dictoffset__" attribute
• tp_getattr: PyObject_GetAttr()
• tp_getattro: PyObject_GetAttr()
• tp_setattr: PyObject_SetAttr()
• tp_setattro: PyObject_SetAttr()
• tp_getset
• tp_methods

Methods and how to call them:

• tp_alloc: PyType_GenericNew()
• tp_call: PyObject_Call()
• tp_del: call __del__() method (?), subtype_dealloc()
• tp_finalize: call PyObject_CallFinalizer()
• tp_free: get PyType_GetSlot(Py_tp_free)
• tp_hash: PyObject_Hash()
• tp_init: get PyType_GetSlot(Py_tp_init), call a type to create an instance, type_call()
• tp_is_gc: PyObject_IS_GC()
• tp_iter: PyObject_GetIter()
• tp_iternext: PyIter_Next()
• tp_new: get PyType_GetSlot(Py_tp_new), call a type to create an instance, type_call()
• tp_repr: PyObject_Repr()
• tp_richcompare: PyObject_RichCompare()
• tp_str: PyObject_Str()

PyHeapTypeObject internal members:

• _ht_tpname
• _spec_cache
• ht_cached_keys
• ht_slots

PyHeapTypeObject members:

• ht_type: ?
• as_async: copy of PyTypeObject.tp_as_async
• as_number: copy of PyTypeObject.tp_as_number
• as_mapping: copy of PyTypeObject.tp_as_mapping
• as_sequence: copy of PyTypeObject.tp_as_sequence
• as_buffer: copy of PyTypeObject.tp_as_buffer
• ht_name: PyType_GetName(), set "__name__" attribute
• ht_qualname: PyType_GetQualName()
• ht_module: PyType_GetModule(), PyType_GetModuleByDef()

Unused members:

• tp_cache
• tp_as_number.nb_reserved
• tp_as_sequence.was_sq_slice
• tp_as_sequence.was_sq_ass_slice

[vstinner]

Creating a static type PyTypeObject my_type = {...}; requires accessing directly all PyTypeObject members. This API, like PyType_Ready(), should be deprecated in favor of heap types.

Creating a heap type can be done with the PyType_Spec structure and PyType_FromSpec() function. This API doesn't access PyTypeObject or PyHeapTypeObject members on purpose. It helps the ABI backward/forward compatibility: the stable ABI used it for example.

The Python stdlib is a bad example, it still has many static types in Python 3.13: issue #84258. Moreover, some static types may remain implemented as static types since they are exposed in the C API, such as &PyUnicode_Type (issue #84781).

What changed recently is that many data are now "static": generated at build type and constant. "Static" types means more and more something really "static", like it's no longer needed to release their memory at Python exit (issue #103276).

[vstinner]

• tp_dict: get "dict" attribute, PyObject_GenericGetAttr(), PyObject_GenericSetDict(), _PyObject_GetDictPtr(). See also Python 3.10 Py_TPFLAGS_MANAGED_DICT.
• tp_dictoffset: get "dictoffset" attribute

PyType_GetDict() is excluded from the limited C API, and _PyObject_GetDictPtr() is part of the internal API.

[vstinner]

Using tp_name to format an error message is a very common code pattern. I created issue gh-111696 for that.

[vstinner]

Let's revisit that once PEP 737 is accepted and implementated, it's one of the most common consumer of the PyTypeObject.