Extension dependency conflicts

[gswifort]

As mentioned in #146, it is assumed that pyrx can be used to create modular CAD extensions. These extensions can be plugged in as independent plugins. There is a big risk of conflicting dependencies and possibly conflicts at the interpreter level. I think this issue should be looked into https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreterFromConfig
Maybe this will also help to solve, as we have discussed many times, module reloading (pyreload) problems.

[CEXT-Dan]

Indeed, though, it’s not limited to python, clashes can occur throughout the system.
Maybe this would add a false sense of security that clashes can’t happen?

Autodesk made valiant attempt to solve this, with the now discontinued, registered developer symbol (RDS)
A four-character prefix, I had registered ‘CEXT’

The idea is all your paths, module names, command names, CUIX, etc., should be prefixed with your RDS
Even though the registration is discontinued, it’s still recommended in the docs
https://help.autodesk.com/view/OARX/2023/ENU/?guid=GUID-F150D6E2-4335-4284-ACAC-DC23D45DEB26

If you look at Lee Mac’s amazing work for AutoLISP, https://www.lee-mac.com/programs.html
He prefixes everything with LM: to avoid clashes

Looking at the documentation for Py_NewInterpreterFromConfig, it seems very risky, very hard.
https://docs.python.org/3/c-api/init.html#a-per-interpreter-gil
https://docs.python.org/3/c-api/init.html#bugs-and-caveats

seems I would have to manage what interpreter gets called for every function on the C++ side.
Garbage collection for shared objects that derive from PyDbObject would be difficult to manage

[gswifort]

What problems do you see?

[CEXT-Dan]

1, how would this be registered?
2, each interpreter has its own lock, how would I know which interpreter to lock for methods that run unattended? Reactors, Events, Point monitor, on idle etc.
3, How does garbage collection work, will the GC in one interpreter nuke an AcDbObject in another?
4, Pyload command, how do I know which interpreter to load into
5, other behaviors, are modules like wxPython and boost::python compatible

That way I read it is each interpreter is its own context and I would need to switch to some current context for every C++ call back into Python. Or am I read it wrong?

[gswifort]

First of all, let me remind you that I have zero knowledge of C++ and everything I write here is based on the Python documentation.

1. I think there should be a function provided in the Ap module that allows the main interpreter to create a new interpreter for a given extension (something like a subprocess)
2. The GIL can (and for this project should) be shared among all interpreters
   https://docs.python.org/3/c-api/init.html#c.PyInterpreterConfig_SHARED_GIL
3. I don't think so, but I don't see why it would be necessary.
4. PYLOAD should work on the main interpreter. Any "ready" extensions should not be loaded by PYLOAD in my opinion but by pyrx_onload or by RxLoader.
5. Also my main question mark was the issue of event handling by wxpython, my intuition (supported by very modest knowledge in this area) tells me that the event will be passed on to subsequent interpreters until it is handled.

I don't know if what I wrote makes any sense, we would have to test it...

[CEXT-Dan]

1, interesting

2, I don’t know which interpreter boost::python will use to allocate memory in the converters , example
https://github.com/CEXT-Dan/PyRx/blob/01380662f56001c671c2d7af8244ac759c8939e3/PyRxCore/StdAfx.h#L401-L524\

PyAutoLockGIL lock;
boost::python::list list;  

boost does all the allocations somewhere its internal workings, from what I read, it’s not advised to share objects. So, in the case of Item 1, the sub modules would never be able to use the AutoCAD wrappers, maybe Ax stuff would work

python/cpython#123488

Right, mixing them is unsupported. You still need to have Python initialized, and then you can create subinterpreters with the main interpreter's GIL -- once the sub interpreter is initialized, it will have its own GIL.

I read this as I would have some lock switching context

3, it’s not clear how reference counting works, if two interpreters have a handle to a Database Object and we have two separate objects with a ref count of one, then the garbage collector of one will delete the C++ pointer, the object in the other interpreter will have undefined behavior

4, pyload registers commands, commands hold a handle to a python function. How can the command know which interpreter to exec into?

5, I don’t think wxpython will traverse interpreters, I explicitly load wxPython, wxWidgets and assign it to AutoCAD’s main frame.
I wrapped palettes in wxWidgets and use the GIL to lock calls into wxPython

Applications are usually forward compatible, i.e. if a user loads modules A & B that reference pandas, B requires the latest version, it’s very likely A will still run just fine even though it was tested on an older version.

Maybe there's some best practices that could be documented to avoid clashes
For now, I think It’s interesting, but too risky, too much development time gone if it doesn't work

[gswifort]

Yes, let's leave this open. Maybe some better solutions will come up.