bug: installing packages from PyPi is broken upon upgrading from v7.1.0 to v7.2.2

[ahmadomira]

Overview

After updating our OT-2 robot's software from v7.1.0 to v7.2.2, installing a Python package (from outside Python's standard library) using pip doesn't work. The installed packages do exist in Python's site-packages directory, yet importing them inside the Python interpreter results a Module Not Found error.

Please note that this problem goes away when downgrading the software to v7.1.0.

Steps to reproduce

Note: the following problem can be reproduced by installing any PyPi package. The packages are installed following this Opentrons tutorial.

After completing the initial SSH-setup:

1. connect the robot to our local network
2. open a Powershell window, and do:

ssh -i ot2_ssh_key root@OUR_ROBOT_ID

3. after successful login to the robot:

pip3 install openpyxl

4. after a successful package installation, restart the robot.
5. login again into the robot over SSH (repeat Step 1)
6. doing, for example:

python3
>>> import openpyxl

Current behavior

Doing, for example, this:

python3
>>> import openpyxl

outputs: Module Not Found Error

or simply attempting to uninstall the same package, with no further steps in between (as shown in the screenshot below):

pip3 uninstall openpyxl

outputs: WARNING: Skipping openpyxl as it is not installed.

We have tried to install packages from PyPi into Python virtual environemts (venv), but the problem persists.

Problem Screenshot

The installed packages do exist in Python's system directory

Navigating to Python's site-packages directory, after package installation, shows that the packages exist within the Python directory. Doing in the robots terminal, for example:

which python3

outputs: /usr/bin/python3

Now navigating to /usr/bin/python3/.../site-packages shows that the installed package (and its submodules) exists, as shown in these screenshots:

Expected behavior

Before upgrading the software, which we had to do because of other problem, we used to install our Python packages from within our Jupyter notebooks by doing:

!pip install openpyxl

then import openpyxl in next cells with no issues. Now, doing the same results Module Not Found errors

Operating system

Windows

System and robot setup or anything else?

• Opentrons software version: v7.2.2
• The robot is connected to the PC via Ethernet
• Note: the PYTHONPATH environment variable seems to be empty. We were not able to set it otherwise

[aschleck]

Experiencing this as well on an OT-3 at v7.3.1. Presumably a breakage caused on the OT-2 by Opentrons/buildroot@4d580d8 / Opentrons/buildroot#231 but I don't really understand the mechanism and I know the OT-3 is pulling from eg Opentrons/oe-core@9eef29d .

Anyway you can set PYTHONPATH to make this work-ish. Opentrons should still fix this.

root@41523d:~# cd /home/root
root@41523d:~# python3 -m venv venv
root@41523d:~# source venv/bin/activate
(venv) root@41523d:~# pip install -r requirements.txt --verbose                                       
Using pip 22.3.1 from /home/root/venv/lib/python3.10/site-packages/pip (python 3.10)
Requirement already satisfied: opentrons==7.3.1 in /opt/opentrons-robot-server (from -r requirements.txt (line 1)) (7.3.1)
Collecting pandas==1.5.3
  Using cached pandas-1.5.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.4 MB)
Requirement already satisfied: opentrons-shared-data==7.3.1 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (7.3.1)
Requirement already satisfied: aionotify==0.2.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (0.2.0)
Requirement already satisfied: anyio<4.0.0,>=3.6.1 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (3.7.1)
Requirement already satisfied: jsonschema<4.18.0,>=3.0.1 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (4.17.3)
Collecting numpy<2,>=1.20.0
  Using cached numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.2 MB)
Requirement already satisfied: pydantic<2.0.0,>=1.10.9 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (1.10.12)
Requirement already satisfied: pyserial>=3.5 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (3.5)
Requirement already satisfied: typing-extensions<5,>=4.0.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (4.11.0)
Requirement already satisfied: click<9,>=8.0.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (8.1.2)
Requirement already satisfied: packaging>=21.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (24.0)
Collecting pytz>=2020.1
  Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB)
Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Requirement already satisfied: idna>=2.8 in /opt/opentrons-robot-server (from anyio<4.0.0,>=3.6.1->opentrons==7.3.1->-r requirements.txt (line 1)) (3.3)
Requirement already satisfied: sniffio>=1.1 in /opt/opentrons-robot-server (from anyio<4.0.0,>=3.6.1->opentrons==7.3.1->-r requirements.txt (line 1)) (1.3.1)
Requirement already satisfied: exceptiongroup in /opt/opentrons-robot-server (from anyio<4.0.0,>=3.6.1->opentrons==7.3.1->-r requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/opentrons-robot-server (from jsonschema<4.18.0,>=3.0.1->opentrons==7.3.1->-r requirements.txt (line 1)) (23.2.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /opt/opentrons-robot-server (from jsonschema<4.18.0,>=3.0.1->opentrons==7.3.1->-r requirements.txt (line 1)) (0.20.0)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, six, numpy, python-dateutil, pandas
  Creating /var/user-packages/home/root/venv/bin
  changing mode of /var/user-packages/home/root/venv/bin/f2py to 755
  WARNING: The script f2py is installed in '/var/user-packages/home/root/venv/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.26.4 pandas-1.5.3 python-dateutil-2.9.0.post0 pytz-2024.1 six-1.16.0

[notice] A new release of pip available: 22.3.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(venv) root@41523d:~# ls venv/lib/python3.10/site-packages/
_distutils_hack  distutils-precedence.pth  pip	pip-22.3.1.dist-info  pkg_resources  setuptools  setuptools-65.5.0.dist-info
(venv) root@41523d:~# ls /var/user-packages/home/root/venv/lib/python3.10/site-packages/
__pycache__  dateutil  numpy  numpy-1.26.4.dist-info  numpy.libs  pandas  pandas-1.5.3.dist-info  python_dateutil-2.9.0.post0.dist-info  pytz  pytz-2024.1.dist-info  six-1.16.0.dist-info  six.py
(venv) root@41523d:~# PYTHONPATH=/var/user-packages/home/root/venv/lib/python3.10/site-packages/:/opt/opentrons-robot-server:/usr/lib/python3.10/site-packages python3          
Python 3.10.9 (main, Dec  6 2022, 18:44:57) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> import opentrons
>>>

[aschleck]

Better solution: run the venv's pip with PIP_CONFIG_FILE=/dev/null like PIP_CONFIG_FILE=/dev/null pip install pandas.

The breakage on Flex is caused by https://github.com/Opentrons/oe-core/blob/2ad1ac910d393c84b7f363ae3cca23786d290935/layers/meta-opentrons/recipes-devtools/python3-pip/files/pip.conf#L1

[ahmadomira]

Better solution: run the venv's pip with PIP_CONFIG_FILE=/dev/null like PIP_CONFIG_FILE=/dev/null pip install pandas.

The breakage on Flex is caused by https://github.com/Opentrons/oe-core/blob/2ad1ac910d393c84b7f363ae3cca23786d290935/layers/meta-opentrons/recipes-devtools/python3-pip/files/pip.conf#L1

oh wow, this seems to work. Thank you! I don't understand the purpose of this pip config file :/ how is this supposed to work? it has also broke the venv mechanism.

[aschleck]

@vegano1 any thoughts on this?

My guess is that OT doesn't run the Jupyter notebook inside of a venv, so if the user does !pip install pandas then it would by default try to install into the read-only root. So then this change forces pip to install into this particular path.

Personally I think a better way to support !pip install pandas in the notebook would have been to just run Jupyter inside of a venv and not force every pip invocation to use this path. But I may not understand the problem or goal fully.

One reason my approach, using venvs, is a better solution for Jupyter is because with venvs you can install the latest version of Pandas. Whereas in the current approach, even with this pip.conf hack, pandas will still try to install into /usr/bin and fail because it's RO. If you use pip install with venvs then Pandas will correctly install the binary into a bin folder in the venv instead.