Clarifying nanobind positioning with respect to sub interpreters (related to PEP 684)

[stellaraccident]

Hi - I've really been enjoying nanobind. Clean, fast, and has the right granularity of features for what I want.

I was recently going down the rabbit hole of seeing whether there was any hope for PEP 684 style per-interpreter GIL with nanobind based extensions. I didn't find a concrete answer to this question but did some research and decided to post specifically to ask.

The big thing I found (in the porting guide):

"○ Multi-intepreter, Embedding: The ability to embed Python in an executable or run several independent Python interpreters in the same process is unsupported. Nanobind caters to bindings only. Multi-interpreter support would require TLS lookups for nanobind data structures, which is undesirable."

I found this somewhat ambiguous and possibly worded this way because of specific experience with pybind11 (which I know has a lot of features related to this vs just aiming for peaceful co-existence). This is how I read the "Nanobind caters to bindings only" and nodded in agreement. But I found it somewhat peculiar to have a hard line against being able to build C extensions with nanobind that cannot function with multiple interpreters. I further didn't know if this was referring to some setup involving multiple top level interpreters vs sub interpreters. Sorry to get all language-lawyer like: my intent is not to challenge so much but to express confusion and raise the question.

My Findings

The first thing I looked at was whether nanobind does multi-phase initialization (PEP 489), which (from what I read) is the primary indicator that an extension module is compatible with PEP 684. I don't write this level of code every day, but the definition of the NB_MODULE macro appears to do legacy init (i.e. return a fully constructed module vs a module def).

Further, nanobind::detail::init(NB_DOMAIN_STR); does process global initialization of a extern nb_internals *internals. The nb_internals struct is inherently tied to a single interpreter-extension-module pair and is stored in a global, so that's a pretty hard stop.

While there are some entities in nanobind that capture a local nb_internals* instance and are scoped to that, there are (from a simple grep) 61 locations that directly dereference the global. I couldn't find anything "clever" being done to try to swap the global struct or anything.

While I could certainly imagine ways to change this so that it is sub-interpreter/multi-phase-init tolerant, a casual reading of the code leads me to the interpretation of the above statement being that the author decided explicitly that this use case will never be supported by the bindings.

I wanted to clarify whether this is actually the intent/goal at this point before drawing further conclusions. Like I said, I could imagine various surgical ways to refactor to a compatible state, but am not going to spend the time to flesh that out if a design non starter. But I would have more questions if this is indeed the hard line being taken.

If nothing else comes of this discussion, I would at least like to save the next person searching for an answer by having it be answered here.

[wjakob]

My experience with pybind11 was that people who use embedding/sub-interpreters pursue highly unusual projects that invariably run into various limitations and corner cases. Those users then open issues that clog up the project issue tracker. Quite likely those issues aren't specific to pybind11 but rather Python and various used packages, but it's difficult to investigate those failures just because the use cases requiring embeding and sub-interpreters are inherently already on the complicated end of the spectrum. This being a hobby project with little time for support, the restriction to the base case of bindings in a standard Python interpreter is therefore intentional. You are also correct in your observation about the multi-phase initialization.

The issue with TLS mentioned in the documentation is that nanobind employs an internals data structure to keep track of its state, and this is a global variable (i.e. accessed efficiently via the shared library's global offset table). If you have multiple interpreters, each one would need its own nanobind state, and that would require some modificiations of the library that make this lookup (which is on the hot path) slower.

Probably it's not hard to maintain some patches to nanobind out of band that extend it to enable embedding and sub-interpreters, but I am not enthusiastic about supporting this feature set here for the reasons explained above.

[stellaraccident]

Thank you for taking the time to answer -- very helpful to know where you are coming from.

Perhaps if PEP 684 style sub interpreters catches on, this will be less of an all-over-the-place thing to support. But there are pretty substantial headwinds to that happening since afaict hardly any of the C extension ecosystem has trivial work to do to support it.

I had some small hopes of trying out that concurrency model with a nanobind based extension I was working on, but I think I'm going to pursue a different option (probably just lean in on gil-less).

It would have saved me a few hours of looking into things if I had connected that the line in the porting guide as related to PEP 684. I've been using nanobind for a while and haven't consulted the "porting guide" for a long time since all of my new work is just natively based on nanobind, and the regular " non porting" docs are excellent and comprehensive.

But I can't think of a better place to call it out. Maybe this thread helps the next person.