Rewrite packaging/setup to work better with pip

[sassy-asjp]

Describe your use case
I'd like to be able to use pip to manage the setup, install, and packaging process. In particular, I'd like to be able to build and use a bdist_wheel of Basilisk for easy and quick installation for non-developers (e.g., a hardware technician setting up a HILS test platform that uses Basilisk to run the dynamics and environment).

Right now, conan calls pip to register the package with python. However, I'd like to flip this such that pip calls conan to build the C/C++ components.

Additional context
I'd kinda like to know background on why the current solution with conan calling pip was chosen. While Basilisk is mostly written in C/C++, its primary external interface and user interface is python, so is more intuitive as a python/pip package than a C++/conan package. Was pip packaging considered already and rejected for some reason?

[schaubh]

We tried to do this several years ago. We got the BSK wheel to work, but only if the computers were using the same Python version. The user can use system python, home-brew python, python from python.org, and the wheel with pre-compiled binaries only worked if this matched. We are currently looking at providing docker images for tagged BSK releases to make for easier use of BSK.

[sassy-asjp]

@schaubh

Thanks for the reply. Just to clarify, was it a Python ABI compatibility issue? Or was it that even Python of the same version e.g. 3.9, from different sources were breaking things?

[schaubh]

It was looking for the specific version of python that was used to create BSK on the developer computer. There might be better ways to package this, I'm not a pip wheel expert. Just sharing our experience :-)

[sassy-asjp]

Understood. That is a fairly expected, though unfortunate, behavior of pip wheel builds. There are tools like cibuildwheel which automate the process of building on many platforms/versions. If you still have any of the work from when you tried to build wheels lying around, I'd love to see it.

I am also looking at packaging via docker images, and that may be better. However, my main development environment has been docker on macOS, however it is annoying to work with graphs outside the context of the Jupyter notebook environment, and connecting Vizard to a running simulation is pretty flaky, so I think there is value in packing BSK in a way that can be installed "normally" for macOS/Windows users.

[sassy-asjp]

@schaubh

We are exploring a packaging system that is able to build Python wheels and it seems to work. However, we have had to hardcode our "external modules" path into it, since pip doesn't have an easy way of specifying the external modules like conan does.

We'd like to rewrite packaging in a way that can be adopted by the open source project, so I was wondering if any of these options sounded appealing:

• External module path specified by modifying one or two lines in the setup.py (or pyproject.toml)?
• External module path specified as system environment variable instead of an argument (e.g., BSK_EXTERNAL_MODULE=/path/to/my/ext/module pip install -e .)?
• External module path specified by creating a specific file that the system looks for to enable/disable external module

[schaubh]

Howdy @sassy-asjp , thanks for the idea. I would prefer the BSK user not have to modify any of the core BSK repo files to make an external folder be included in the build. Modifying such files makes pulling the latest changes more challenging (not impossible of course :-).

Of the three options, I think setting the environment variable is an interesting option for this reason. We could leave the current setup as is, and add this as an alternate way to specify the external folder? @patkenneally , I know you have been thinking a lot of BSK packaging, any thoughts on this idea?