[C API] Prepare PyTypeObject structure for a stable ABI: avoid accessing members in the public API #84351
Comments
|
Leaking the PyTypeObject structure in the C API indirectly causes ABI issue (especially for statically allocated types), cause practical issues when old fields are removed and new fields are added (ex: tp_vectorcall addition and tp_print removal caused a lot of troubles with C code generated by Cython: see bpo-37250), prevents us to add feature and experiment optimization. I don't expect that we will be able to make PyTypeObject opaque soon. The purpose of this issue is to track the work done towards this goal. I propose to slowly prepare the Python code base, the C API and third party code (especially Cython) to make PyTypeObject structure opaque. We have to identify most common code patterns which access directly PyTypeObject fields, provide helper functions, and ease the migration to solutions which don't access directly PyTypeObject. See also bpo-39573: "Make PyObject an opaque structure in the limited C API" and bpo-39947 "Make the PyThreadState structure opaque (move it to the internal C API)". Longer rationale about making structures of the C API opaque:
-- Multiple practical issues are preventing us to make PyTypeObject opaque right now. (*) Many C extension modules are still using statically allocated types: there is an on-going effort in bpo-40077 to convert C extension modules one by one to PyType_FromSpec(). (*) Py_TYPE(obj)->tp_name is commonly accessed to format an error message. Example: PyErr_Format(PyExc_TypeError, "exec() globals must be a dict, not %.100s",
Py_TYPE(globals)->tp_name);I worked on bpo-34595 and started a discussion on python-dev to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format(): Sadly, we failed to reach a consensus and I gave up on this idea. We should reconsider this idea. We need to agree on how types should be formatted:
There is also the question of breaking applications which rely on the current exact error message. And the question of removing legacy "%.100s" which was used before Python was able to allocate a buffer large enough to arbitrary string length. When an error is formatted in pure Python, names are never truncated. (*) Call the function of the parent type when a method is overriden in a subclass. Example with PyTypeObject.tp_free called in a deallocator: static void
abc_data_dealloc(_abc_data *self)
{
PyTypeObject *tp = Py_TYPE(self);
...
tp->tp_free(self);
Py_DECREF(tp);
}() The PEP-384 provides the most generic PyType_GetSlot() but it's not convenient to use: need to handle error (NULL), need to cast the void into the expected type (error prone cast), etc. We should slowly add more and more helper functions for most common use cases. We can try to convert a few C extension modules of the Python stdlib to see which use cases are the most common. Hopefully, many use cases are already abstracted by widely used functions like PyNumber_Add(), PySequence_Size(), etc. (*) Likely other issues that I forgot. |
|
Macros and static inline functions of the public C API which access directly PyTypeObject fields. There may be more. #define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize )static inline vectorcallfunc #define PyObject_CheckBuffer(obj) \
((Py_TYPE(obj)->tp_as_buffer != NULL) && \
(Py_TYPE(obj)->tp_as_buffer->bf_getbuffer != NULL))
#define PyIndex_Check(obj) \
(Py_TYPE(obj)->tp_as_number != NULL && \
Py_TYPE(obj)->tp_as_number->nb_index != NULL)
#define PyObject_GET_WEAKREFS_LISTPTR(o) \
((PyObject **) (((char *) (o)) + Py_TYPE(o)->tp_weaklistoffset))
static inline int
PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
#ifdef Py_LIMITED_API
return ((PyType_GetFlags(type) & feature) != 0);
#else
return ((type->tp_flags & feature) != 0);
#endif
}
#define _PyObject_SIZE(typeobj) ( (typeobj)->tp_basicsize ) |
|
Just manually verified that PR19377, when compiled against xlc - crashes during make: Objects/genobject.c:127: _PyObject_GC_TRACK: Assertion "!(((PyGC_Head *)(op)-1)->_gc_next != 0)" failed: object already tracked by the garbage collector FYI: about two hours ago I verified that xlc and 08050e9 : bpo-40147: Fix a compiler warning on Windows in Python/compile.c (GH-19389) all was green. |
|
Just checked - seems to be SPECIFIC to xlc-v16 as neither xlv-v11 nor xlc-v13 have any issues building. |
|
Py_TRASHCAN_BEGIN() access directly PyTypeObject.tp_dealloc: #define Py_TRASHCAN_BEGIN(op, dealloc) \
Py_TRASHCAN_BEGIN_CONDITION(op, \
Py_TYPE(op)->tp_dealloc == (destructor)(dealloc))It should use PyType_GetSlot() or a new getter function (to read PyTypeObject.tp_dealloc) should be added. |
Oh. It seems like currently, PyType_GetSlot() can only be used on a heap allocated types :-( The function starts with: if (!PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE) || slot < 0) {
PyErr_BadInternalCall();
return NULL;
} |
That sounds like an AIX specific issue. Please open a separated issue. |
|
Hum, once most changes will land, maybe it would be worth it to document them at: |
/* Test if an object has a GC head */
#define PyObject_IS_GC(o) \
(PyType_IS_GC(Py_TYPE(o)) \
&& (Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))This macro should be converted to an opaque function. |
|
|
On python-dev, Ronald Oussoren asked to add support for the buffer protocol in PyType_FromSpec() and PyTypeSpec API: Ronald:
See also PyType_FromSpec() issue with opaque PyObject: |
|
Something else that probably needs attention with the TypeSpec API is subclassing type in an extension when that subclass adds fields to the type object. I use this in PyObjC to (dynamically) create types that have some additional data. I could probably work around this issue by adding a level of indirection (basically storing the extra data in a WeakKeyDictionary), but haven't looked into this yet. |
Looks like this macro not recorded in docs. Do we need using function to replace this macro? |
It never prevented anyone to use a function of the C API :-) |
|
I created bpo-41073: [C API] PyType_GetSlot() should accept static types. |
No. The internal C API access directly to structure members on purpose, for best performances. |
For macOS which doesn't use LTO compiler optimization, we added private static inline functions of some "Check" functions. But I don't think that it's worth it here (I don't think that the function is commonly called in "hot code"). |
|
For PyExceptionClass_Name: Is it ok to just remove the macro version (like with #68736)? |
Yes, I think so. |
|
Thanks, Victor. For PySequence_ITEM, I guess adding a private C version (for example _PySequence_Item) and redirecting the macro to the C version would be acceptable. Ditto for PyHeapType_GET_MEMBERS and PyType_SUPPORTS_WEAKREFS. |
This can introduce a performance slowdown and so should wait until the PEP-620 is accepted. |
Noted. |
|
TODO:
Other TODO tasks which can be addressed in follow-up issues: |
|
I searched for "_PyGC_FINALIZED" in top 5000 PyPI projects. It seems like only Cython is impacted. ddtrace and guppy3 use directly the internal C API. == Cython 0.29.26 ==
== ddtrace 0.57.3 == In ddtrace/profiling/collector/stack.pyx: == guppy3-3.1.2 == In src/heapy/hv.c: #if PY_MAJOR_VERSION >= 3 && PY_MINOR_VERSION >= 9
# define Py_BUILD_CORE
/* PyGC_Head */
# undef _PyGC_FINALIZED
# include <internal/pycore_gc.h>
# undef Py_BUILD_CORE
#endif |
|
I searched for "_PyObject_DebugMallocStats" in top 5000 PyPI projects. There is a single project using it: guppy3. Extract of guppy3 src/heapy/xmemstats.c: ...
dlptr__PyObject_DebugMallocStats = addr_of_symbol("_PyObject_DebugMallocStats");
...
static PyObject *
hp_xmemstats(PyObject *self, PyObject *args)
{
if (dlptr__PyObject_DebugMallocStats) {
fprintf(stderr, "======================================================================\n");
fprintf(stderr, "Output from _PyObject_DebugMallocStats()\n\n");
dlptr__PyObject_DebugMallocStats(stderr);
}
...
}addr_of_symbol() is implemented with dlsym() or GetModuleHandle(NULL)+GetProcAddress(): it searchs for the symbol in the current process. |
|
In the top 5000 PyPI projects, the _PyObject_SIZE() and _PyObject_VAR_SIZE() functions are used by 7 projects:
|
|
Changes already done:
PyType_HasFeature() is left unchanged since the change caused performance issue on macOS, whereas Python is not built with LTO. |
vstinner
changed the title
vstinner
changed the title
|
I close the issue. While this issue is not fully fixed, it's a milestone of my "stable ABI" goal. I prefer to address remaining issues in new separted issues. This issues is not fully fixed, there are are still a 4 macros which access directly PyTypeObject members:
PySequence_ITEM() and PyType_HasFeature() are important for performance, I prefer to have a PEP before changing these two functions. _PyObject_SIZE() and _PyObject_VAR_SIZE() should be made public with a better name like PyObject_SizeOf() and PyVarObject_SizeOf(). But I prefer to do that in a separated issue. IMO it would be interesting to merge the PyHeapTypeObject structure into the PyTypeObject structure: And make the PyTypeObject opaque: deprecate static types and promote the usage of heap types in C extensions. That's a big project which may be splitted into multiple issues and the final change may need its own PEP. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: