shared_ptr clarification

[brettc]

Thanks for nanobind, this looks very interesting (and thanks for pybind11 -- which I've been using for a while now).

I'm trying to understand the details of the new way that shared_ptr interacts with nanobind, especially with respect to the disallowed shared_from_this, which I use in a library I'd like to rewrap in nanobind. I've read the docs, and tried a bit of debugging, and I think I get it, but I wanted to check.

As I understand it, there are two situations:

1. shared_ptr<T> is created by C++, and returned to python. nanobind creates a python object that binds a T *, and creates a shared_ptr<T> that has the same lifetime as the python object.

In this case, there seems to no issue with their ever being multiple shared_ptr control blocks. In fact, if you could guarantee this was the only way for python to get hold of a T, then the shared_from_this mechanism would actually be fine.

2. A python object is created that binds a T (not a pointer) using placement new. This object is then passed into C++ to a function requiring a shared_ptr<T>. nanobind handles this by increasing the python refcount and creating a shared_ptr<T> with a custom deleter that reduces the python refcount. This ensures python does not delete the object if that shared_ptr is still held somewhere in the C++ code.

This problem is case 2 is that the shared_ptr<T> does not reflect the right number of references to the T held in the python object, as it does not know about the python refcount.

So that is my (likely incomplete) understanding.

A couple of questions:

1. How can multiple shared_ptr<T> control blocks be created? (other than by bad C++ habits). I see the problem as the reference counting being shared across python refcount and shared_ptr so I must be missing something.

2. The docs seem to miss a pertinent case. They suggest that "When calling a C++ function from python [ ... ]" nanobind ends ups "creating a shared_ptr<T>". But if the python object was one that was already returned from C++ function, then this does not happen, right?

In any case, my thought was that I can actually use the shared_from_this mechanism, by implementing it directly in my class (adding a weak_ptr that I manually initialise), and then ensuring that creation is only possible via C++. (ie. a private constructor, and a static function returning a shared_ptr<T>.

[wjakob]

Sorry, I missed this question when you first posted it. Is it still relevant?

[brettc]

Hi @wjakob, I have a much better understanding of it now. I'm still not clear on the answer to Q1, if there is an easy response.

(also, I'll try to get back to the #56 soon)

[wjakob]

Having multiple shared blocks associated with a pointer is indeed an unusual situation—normally it would indicate a bug. But there is another important aspect to this: what happens when the last reference of the shared_ptr goes to zero and the deleter is invoked?

Normally, the deleter is simply operator delete, and then having multiple control blocks is of course a bug. But what nanobind does is that it installs a custom deleter that performs Py_DECREF on an underlying Python object. In this case, having multiple control blocks is fine — lifetime is actually tracked by Python, and the control block simply "owns" a single reference that it gives up upon deletion. Does that clarify things?

[brettc]

Yes, thanks!

I'm excited about the new intrusive pointer support -- it will fix some long standing problems of mine (and others).