PyCall does not use virtualenv

[langesimmons]

I'm running Python 3.7.3 and Julia 1.1.1 (2019-05-16). I have a virtual environment set up for python and PyCall is unable to import the packages installed there. In addition, when I run

py"""import site
 print(site.getsitepackages())"""

I get the site packages folder of my main install and not the environment one. I can also import packages if they are installed on my system's Python.

When adding PyCall I have my environment activated. I have also tried setting the ENV["PYTHON"] and running Pkg.build("PyCall") as well with no change.

I am on Windows 10.

[stevengj]

Which Python is PyCall configured to use? i.e. what is PyCall.python?

[dabidgs3]

What was the solution to this issue?

[ghost]

Hi,
I have the same issue. Setting ENV["PYTHON"] = ".env\\Scripts\\python.exe" (which is the python executable of the virtualenv) and re-building PyCall seems to have no effect.

The executable is set right:

julia> PyCall.python
"C:\\DATA\\Spine\\Toolbox projects\\CS A5\\.env\\Scripts\\python.exe"

but Python’s sys.path does not seem to contain right site-packages dir

julia> pyimport("sys").path
10-element Array{String,1}:
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\python37.zip"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\DLLs"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\lib"
 "C:\\Users\\...\\AppData\\Local\\Julia-1.2.0\\bin"
 "C:\\Users\\...\\AppData\\Roaming\\Python\\Python37\\site-packages"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\lib\\site-packages"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\lib\\site-packages\\win32"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\lib\\site-packages\\win32\\lib"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\lib\\site-packages\\Pythonwin"

since there are only the site-packages of the main Python installation.

I have Win10 too.

[langesimmons]

Sorry for the delay. I eventually changed to using WSL after not being able to figure this out.

[ghost]

So on Linux, this virtualenv thing works fine, right? I guess it’s because PyCall is not using Conda.jl. I wonder how to bypass Conda on Windows too.

[stevengj]

You need to set ENV["PYTHON"] to an absolute path. Using ".env\\Scripts\\python.exe" will not work.

[ghost]

Thanks. Actually I used

julia> ENV["PYTHON"] = Sys.which("python")
"C:\\<full path here>\\.env\\Scripts\\python.exe"

inside the virtualenv which does give the absolute path. It still did not work as expected.

[ghost]

A work-around solution for me is to add the site-packages dir of the virtualenv to environment variable PYTHONPATH:

ENV["PYTHONPATH"] = abspath(".env/lib/site-packages") 

[ghost]

I’m guessing this behaviour is because ‘PyCall does not use the python program per se: it links directly to the libpython library’ (from README). In the case of a virtualenv, the libpython library still is at the main Python install directory, and that’s why only the site-packages of that install are in sys.path.

As the README says that virtualenvs are supported, I’d assume that includes the site-packages too (which is probably the biggest reason to use virtualenvs). Could the sys.path manipulation happen in e.g. src/pyinit.jl?

[ghost]

Seem that using virtualenv instead of Python standard library venv, resolves this. I did:

> virtualenv venv
> venv\Scripts\activate
(venv) > julia
julia> ENV["PYTHON"] = Sys.which("python")
julia> using Pkg
julia> Pkg.add("PyCall")
julia> Pkg.build("PyCall")
julia> using PyCall
julia> PyCall.python
"C:\\...\\venv\\Scripts\\python.exe"
julia> pyimport("sys").path
8-element Array{String,1}:
 "C:\\...\\venv\\Scripts\\python37.zip"
 "C:\\...\\venv\\DLLs"
 "C:\\...\\venv\\lib"
 "C:\\Users\\...\\AppData\\Local\\Julia-1.2.0\\bin"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\Lib"
 "C:\\Users\\...\\AppData\\Local\\Continuum\\miniconda3\\DLLs"
 "C:\\...\\venv"
 "C:\\...\\venv\\lib\\site-packages"

Note the site-packages dir of the virtual environment venv on the last line.

[nnhjy]

@ererkka
The virtualenv approach seems to fail on Julia-1.5.3 again.

[bicycle1885]

I also tried but a virtual environment created using venv didn't work on Windows, even if I specified PYCALL_JL_RUNTIME_PYTHON as python.exe in the virtual environment.
Python itself can detect the virtual environment and loads the environment-specific packages as expected.
It seems that PyCall.jl initializes the Python library in a different way:

(workspace) PS C:\Users\Julius\workspace> python
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix  # points to the home directory of my virtual environment where pyvenv.cfg is placed
'C:\\Users\\Julius\\workspace'
>>> exit()
(workspace) PS C:\Users\Julius\workspace> julia -q
julia> ENV["PYCALL_JL_RUNTIME_PYTHON"] = Sys.which("python")
"C:\\Users\\Julius\\workspace\\Scripts\\python.exe"

julia> using PyCall

julia> pyimport("sys").prefix  # PyCall ignores my virtual environment
"C:\\Users\\Julius\\AppData\\Local\\Programs\\Python\\Python39"

(@v1.6) pkg> st
      Status `C:\Users\Julius\.julia\environments\v1.6\Project.toml`
  [438e738f] PyCall v1.92.3

julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake-avx512)

This happens only on Windows. It doesn't reproduce on my Linux environment.