Reticulate doesn't detect libpython.so from binary

[beew]

I have python3.7.9 installed in a local folder: $HOME/opt/python37, but reticulate is not able to infer the location of libpython from the path to the python binary

>Sys.setenv(RETICULATE_PYTHON="/home/bee/opt/python37/bin/python")
>library(reticulate)

>py_discover_config()
Error in system2(command = python, args = paste0("\"", config_script,  : 
  error in running command
>py_config()
Error in system2(command = python, args = paste0("\"", config_script,  : 
  error in running command

In order to fix this I have to set LD_LIBRARY_PATH explicitly either in .profle or in R session using Sys.setenv() like so

>Sys.setenv(RETICULATE_PYTHON="/home/bee/opt/python37/bin/python")
>Sys.setenv(LD_LIBRARY_PATH = paste("/home/bee/opt/python37/lib", Sys.getenv("LD_LIBRARY_PATH"), sep = ":"))
>library(reticulate)
> py_config()
python:         /home/bee/opt/python37/bin/python
libpython:      /home/bee/opt/python37/lib/libpython3.7m.so
pythonhome:     /home/bee/opt/python37:/home/bee/opt/python37
version:        3.7.9 (default, Aug 24 2020, 16:50:14)  [GCC 9.3.0]
numpy:          /home/bee/opt/python37/lib/python3.7/site-packages/numpy
numpy_version:  1.19.1

NOTE: Python version was forced by RETICULATE_PYTHON
>py_discover_config()
python:         /home/bee/opt/python37/bin/python
libpython:      /home/bee/opt/python37/lib/libpython3.7m.so
pythonhome:     /home/bee/opt/python37:/home/bee/opt/python37
version:        3.7.9 (default, Aug 24 2020, 16:50:14)  [GCC 9.3.0]
numpy:          /home/bee/opt/python37/lib/python3.7/site-packages/numpy
numpy_version:  1.19.1

NOTE: Python version was forced by RETICULATE_PYTHON

According to the documentation reticulate is able to infer the location of libpython from the binary with use_python() or setting RETICULATE_PYTHON (using use_python("/home/bee/opt/python37/bin/python") instead of Sys.setenv(RETICULATE_PYTHON=..) above without setting LD_LIBRARY_PATH gave the same uninformative error,--I figured out it is about missing libpython by running /usr/lib/R/site-library/reticulate/config/config.py with this version of python.)

OS is ubuntu 20.04,
R 4.0.2,
reticulate-1.16
Same behaviour in rstudio 1.3.1073 and running R in the terminal
I have the same setup in Ubuntu 16.04 and python3.8.5 and python3.7.8 installed locally but reticulate is able to detect libpython from the python binaries (can't check the detailed configuration and versions there since I have no access to that machine at the moment)

[kevinushey]

Does this work with the development version of reticulate?

[beew]

@kevinushey
No, the developmental version still doesn't find libpython without explicitly specifying LD_LIBRARY_PATH

[jds-amplify]

I have the identical problem on Mac OS X 10.15.6. The trouble here is that on a stock system we can't even set DYLD_LIBRARY_PATH, so reticulate doesn't work at all.

% export RETICULATE_PYTHON=~/.pyenv/versions/3.8.5/bin/python3.8   
% R

R version 4.0.2 (2020-06-22) -- "Taking Off Again"

> library(reticulate)

> py_discover_config()
python:         /Users/jds/.pyenv/versions/3.8.5/bin/python
libpython:      [NOT FOUND]
pythonhome:     /Users/jds/.pyenv/versions/3.8.5:/Users/jds/.pyenv/versions/3.8.5
version:        3.8.5 (default, Sep 22 2020, 20:49:29)  [Clang 12.0.0 (clang-1200.0.31.1)]
numpy:          /Users/jds/.pyenv/versions/3.8.5/lib/python3.8/site-packages/numpy
numpy_version:  1.19.2

python versions found: 
 /Users/jds/.pyenv/versions/3.8.5/bin/python
 /Users/jds/.pyenv/shims/python3
 /usr/bin/python3
 /usr/bin/python

> py_config()
Error: Python shared library not found, Python bindings not loaded.
Use reticulate::install_miniconda() if you'd like to install a Miniconda Python environment.

[heavywatal]

Currently python_config() function is trying to find libpython by scanning candidate directories. How about asking the given python executable about its library?

library(reticulate)
reticulate::use_python("~/.pyenv/versions/3.9.4/bin/python3", required = TRUE)
script = "
from distutils.sysconfig import get_config_var
libdir = get_config_var('LIBDIR')
instsoname = get_config_var('INSTSONAME')
print(f'{libdir}/{instsoname}')
"
python = path.expand(reticulate:::.globals$use_python_versions[[1]])
system2(python, args = "-", input = script)
# /Users/watal/.pyenv/versions/3.9.4/lib/libpython3.9.a

(of course this will not be the best implementation, though.)

[kevinushey]

@heavywatal That seems like a good idea. We could probably add it to our Python config script:

https://github.com/rstudio/reticulate/blob/master/inst/config/config.py

If that succeeds in finding the shared library explicitly, then we could use that. (Note: would need to verify this works for regular Python installs, virtual environments, and Conda environments)