PyType_FromSpec should take metaclass as an argument #60074
Comments
|
PyType_FromSpec() is a convenient function to create types dynamically in C extension modules, but its usefulness is limited by the fact that it creates new types using the default metaclass. I suggest adding a new C API function PyObject *PyType_FromSpecEx(PyObject *meta, PyType_Spec *spec) and redefine PyType_FromSpec() as PyType_FromSpecEx((PyObject *)&PyType_Type, spec) This functionality cannot be implemented by user because PyType_FromSpec requires access to private slotoffsets table. A (trivial) patch attached. |
|
The patch is a bit light: see how type_new also computes the metaclass from the base classes. |
|
On Thu, Sep 6, 2012 at 12:44 PM, Amaury Forgeot d'Arc
This was intentional. I was looking for a lightweight facility to |
As you can see from my first message, I originally considered PyType_FromSpecEx(PyObject *meta, PyType_Spec *spec) without bases. (In fact I was unaware of the recent addition of PyType_FromSpecWithBases.) Maybe the original signature makes more sense than the one in the patch. Explicitly setting a metaclass is most useful for the most basic type. On the other hand, a fully general function may eventually replace both PyType_FromSpec and PyType_FromSpecWithBases for most uses. |
|
What is your use case for this API? |
|
On Sep 6, 2012, at 5:10 PM, Martin v. Löwis <report@bugs.python.org> wrote:
I can describe my use case, but it is somewhat similar to ctypes. I searched the tracker for a PEP-3121 refactoring applied to ctypes and could not find any. I'll try to come up with a PEP-3121 patch for ctypes using the proposed API. |
|
If it's very special, I'm -0 on this addition. This sounds like this is something very few people would ever need, and they can learn to write more complicated code to achieve the same effect. Convenience API exists to make the common case convenient. I'm -1 on calling it PyType_FromSpecEx. |
|
On Sep 6, 2012, at 6:25 PM, Martin v. Löwis <report@bugs.python.org> wrote:
I find it encouraging that you commented on the choice of name. :-) I can live with PyType_FromMetatypeAndSpec and leave out bases. PyType_FromTypeAndSpec is fine too. On the substance, I don't think this API is just convenience. In my application I have to replace meta type after my type is created with PyType_FromSpec. This is fragile and works only for very simple metatypes. Let's get back to this discussion once I have a ctypes patch. I there will be a work-around for ctypes it will probably work for my case. (My case is a little bit more complicated because I extend the size of my type objects to store custom metadata. Ctypes fudge this issue by hiding extra data in a custom tp_dict. ) |
|
This API may make it easier to declare ABCs in C. |
|
As for declaring ABCs: I don't think the API is necessary, or even helps. An ABC is best created by *calling* ABCMeta, with the appropriate name, a possibly-empty bases tuple, and a dict. What FromSpec could do is to fill out slots with custom functions, which won't be necessary or desirable for ABCs. The really tedious part may be to put all the abstract methods into the ABC, for which having a TypeSpec doesn't help at all. (But I would certainly agree that simplifying creation of ABCs in extension modules is a worthwhile reason for an API addition) For the case that Alexander apparently envisions, i.e. metaclasses where the resulting type objects extend the layout of heap types: it should be possible for an extension module to fill out the entire type "from scratch". This will require knowledge of the layout of heap types, so it can't use just the stable ABI - however, doing this through the stable ABI won't be possible, anyway, since the extended layout needs to know how large a HeapType structure is. If filling out a type with all slots one-by-one is considered too tedious, and patching ob_type too hacky - here is another approach: Use FromSpec to create a type with all slots filled out, then call the metatype to create a subtype of that. I.e. the type which is based on a metatype would actually be a derived class of the type which has the slots defined. |
As a matter of fact, this is what the io module is doing (except that |
|
I know this is quite an old bug that was closed almost 10 years ago. But I am wishing this had been accepted; it would have been quite useful for my case. I'm working on a new iteration of the protobuf extension for Python. At runtime we create types dynamically, one for each message defined in a .proto file, eg. from "message Foo" we dynamically construct a "class Foo". I need to support class variables like Foo.BAR_FIELD_NUMBER, but I don't want to put all these class variables into tp_dict because there are a lot of them and they are rarely used. So I want to implement __getattr__ for the class, which requires having a metaclass. This is where the proposed PyType_FromSpecEx() would have come in very handy. The existing protobuf extension gets around this by directly calling PyType_Type.tp_new() to create a type with a given metaclass: It's unclear to me if PyType_Type.tp_new() is intended to be a supported/public API. But in any case, it's not available in the limited API, and I am trying to restrict myself to the limited API. (I also can't use PyType_GetSlot(PyType_Type, Py_tp_new) because PyType_Type is not a heap type.) Put more succinctly, I do not see any way to use a metaclass from the limited C API. Possible solutions I see:
|
You can also call (PyObject_Call*) the metaclass with (name, bases, namespace); this should produce a class. Or not: >>> class Foo(metaclass=print):
... def foo(self): pass
...
Foo () {'__module__': '__main__', '__qualname__': 'Foo', 'foo': <function Foo.foo at 0x7f6e9ddd9e50>}PyType_FromSpecEx will surely need to limit the metaclass to subtypes of type. What other limitations are there? How closely can we approach the behavior of the
I wouldn't recommend doing that after PyType_Ready is called. Including indirectly, which the type-creation functions in the stable ABI do. |
But won't that just call my metaclass's tp_new? I'm trying to do this from my metaclass's tp_new, so I can customize the class creation process. Then Python code can use my metaclass to construct classes normally.
Why not? What bad things will happen? It seems to be working so far. Setting ob_type directly actually solves another problem that I had been having with the limited API. I want to implement tp_getattro on the metaclass, but I want to first delegate to PyType.tp_getattro to return any entry that may be present in the type's tp_dict. With the full API I could call self->ob_type->tp_base->tp_getattro() do to the equivalent of super(), but with the limited API I can't access type->tp_getattro (and PyType_GetSlot() can't be used on non-heap types). I find that this does what I want: PyTypeObject *saved_type = self->ob_type;
self->ob_type = &PyType_Type;
PyObject *ret = PyObject_GetAttr(self, name);
self->ob_type = saved_type;Previously I had tried: PyObject *super = PyObject_CallFunction((PyObject *)&PySuper_Type, "OO",
self->ob_type, self);
PyObject *ret = PyObject_GetAttr(super, name);
Py_DECREF(super);But for some reason this didn't work. |
It breaks the unwritten contract that "once PyType_Ready is called, the C struct will not be modified". This is implemented in PyPy, since calling PyType_Ready creates the PyPy object in the interpreter based on the C structure. Any further changes will not be reflected in the PyPy interpreter object, so now the python-level and c-level objects do not agree what type(obj) is. We have discussed this in the PyPy team, and would like to propose relaxing the contract to state that "if the c-level contents of an object are modified, PyType_Modified must be called to re-synce the python level and c-level objects" |
|
Since PyPy does not use the Limited API, PySide can quite easily work around the limitations by directly working with the type object. But the usage of PyType_Modified() would make very much sense for PySide‘s new switchable features. That would work immediately without any change, because we already use that function to invalidate Python 3.10‘s type cache. |
|
I found a way to use metaclasses with the limited API. I found that I can access PyType_Type.tp_new by creating a heap type derived from PyType_Type: static PyType_Slot dummy_slots[] = { static PyType_Spec dummy_spec = { PyObject *bases = Py_BuildValue("(O)", &PyType_Type);
PyObject *type = PyType_FromSpecWithBases(&dummy_spec, bases);
Py_DECREF(bases);
type_new = PyType_GetSlot((PyTypeObject*)type, Py_tp_new);
Py_DECREF(type);
#ifndef Py_LIMITED_API
assert(type_new == PyType_Type.tp_new);
#endif
// Creates a type using a metaclass.
PyObject *uses_metaclass = type_new(metaclass, args, NULL);PyType_GetSlot() can't be used on PyType_Type directly, since it is not a heap type. But a heap type derived from PyType_Type will inherit tp_new, and we can call PyType_GetSlot() on that. Once we have PyType_Type.tp_new, we can use it to create a new type using a metaclass. This avoids any of the class-switching tricks I was trying before. We can also get other slots of PyType_Type like tp_getattro to do the equivalent of super(). The PyType_FromSpecEx() function proposed in this bug would still be a nicer solution to my problem. Calling type_new() doesn't let you specify object size or slots. To work around this, I derive from a type I created with PyType_FromSpec(), relying on the fact that the size and slots will be inherited. This works, but it introduces an extra class into the hierarchy that ideally could be avoided. But I do have a workaround that appears to work, and avoids the problems associated with setting ob_type directly (like PyPy incompatibility). |
stephaniegilbert944
mannequin
added
build
Added a new stable API function ``PyType_FromMetaclass``, which mirrors the behavior of ``PyType_FromModuleAndSpec`` except that it takes an additional metaclass argument. This is, e.g., useful for language binding tools that need to store additional information in the type object.
Added a new stable API function ``PyType_FromMetaclass``, which mirrors the behavior of ``PyType_FromModuleAndSpec`` except that it takes an additional metaclass argument. This is, e.g., useful for language binding tools that need to store additional information in the type object.
erlend-aasland
added
the
pending
|
Yes, with a note that #89546 follows up on this. |
|
Hi @wjakob I am contacting you as a PySide developer: Since I tried Python 3.12.0a4 on PySide6 today, we can no longer build PySide.
We use the PyType_FromSpec function extensively in PySide to pass our own tp_new function to the metaclass. All the best -- Chris |
|
@ctismer I think you should be using |
|
here is the work-around in nanobind for versions before 3.12 |
|
@mattip Versions before 3.12 are no problem. And whatever I do gets redirected to PyType_FromMetaclass in 3.12, which then forbids tp_new. |
|
There was more discussion on the newer PR I think, but also here: gh-89546. The issue is that the |
|
@seberg Thanks a lot, now I understand the problem better. And as I read all the discussions, the tp_new problem was not solved in the end, although there was a plan. On first glance I was happy that this hackery around PyType_* would come to an end, but with the redirections leading always into PyType_FromMetaclass, there is effectively a new restriction that was not present before. |
|
If there is a |
|
It should simply do nothing but accept it as-is. Before the new function existed, we supplied the metatype after type crestion. |
|
Before the new function existed PySide worked by pure luck because Python always allocated one slot extra which happened to be enough for its use-case. (It worked for PySide, but not really anyone else, took me weeks to fully understand probably at the time.) In NumPy I currently also have a tp_new, all it does is reject creation, though. IIRC there was a point that it should be OK to do without/remove (I don't remember the actual point, but trusted it at the time). (NumPy's hacks are deep enough they should keep working in 3.12 for the time being.) So, I agree we need to find a solution. But we should also ask the question if you actually need the |
That means it would create an object without calling its C-level initialization function. That sounds pretty dangerous -- most
But does that “C new” need to be a slot? It probably needs a custom signature -- if you can fit in |
No, of course not. In the old issue you had brought up a Now, I am not convinced we need this, because all I currently need in NumPy is rejecting creation in Python. And if I would want more, I would just create a factory function anyway, probably: which is more than enough probably. |
Is doing that in
A good first step would be |
PySide has a meta type Basically, we add some extra attributes to the meta type. But in 2021, supporting PyPy forced me to This is done. The much simpler solution is below: |
|
@seberg I have a crude fix: As soon as a proper "C new" function is available, I will remove that hack. But it serves me well, Hoping for a wise decision on this weird issue 😄 So long -- Chris |
|
Sorry for dropping the ball here. The behavior change is backwards incompatible. If refusing to create classes with I assume this won't really help you since PySide will want to avoid deprecation warnings, though... |
|
@encukou Whatever the outcome, it's fine with me. The most important thing is a stable status that will not change for the time being. |
|
More discussion is in #103968. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: