Remote API for Python: Missing typing-extensions dependency in pyproject.toml

[WeeknightMVP]

pyproject.toml is missing a dependency on typing-extensions, which is required for from typing_extensions import Literal in cryptol/__init__.py. Adding typing-extensions = "^4.1.1" (or an earlier version that correctly defines typing_extensions.Literal) to [tool.poetry.dependencies] in pyproject.toml (or running poetry add typing-extensions from cryptol-remote-api/python) would account for this dependency.

[RyanGlScott]

Good catch. Out of curiosity, is this causing build issues for you? I ask since our test suite hasn't encountered difficulties due to the lack of an explicit typing-extensions dependency, but this may simply be because mypy (a different dependency) already depends on typing-extensions. (I'm a bit alarmed that the test suite would work without declaring an explicit dependency, but perhaps this is simply a quirk of how Python's package management works.)

[WeeknightMVP]

@RyanGIScott

I encountered this when starting a fresh Python 3.10 Poetry project from a base Python 3.10 installation, something like:

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt-get update
$ sudo apt-get install python3.10 python3.10-venv
$ curl -sSL https://install.python-poetry.org | python3.10 -
$ poetry new MyExample
$ cd MyExample
$ poetry add cryptol
$ echo "import cryptol" > myexample.py

As you point out, mypy depends on typing-extensions. mypy is included among pyproject.toml's dev-dependencies, but not dependencies. poetry add pulls development dependencies when instructed (e.g. poetry add --dev cryptol), but not by default. Any packages installed by Poetry will of course appear in poetry.lock, and be available for poetry run, poetry shell, etc., after poetry install or poetry update. These commands install dev-dependencies by default, but offer a --no-dev option to omit them. Somewhere in here is an insight into why cryptol-remote-api unit tests work, and how to configure CI/CD to distinguish usage and development tests.

[pnwamk]

Ah yes, that nicely explains why the typing-extensions entry in the updated poetry.lock swaps the category from dev to main: https://github.com/GaloisInc/cryptol/pull/1336/files

[RyanGlScott]

Tricky. I suppose if we really wanted to be thorough, we'd have first have to run poetry install before running mypy, then run poetry install --no-dev before running the Python unit tests. (Assuming that the poetry install --no-dev invocation removes development dependencies from the current scope, that is.)

[WeeknightMVP]

@RyanGIScott I tested poetry update --no-dev, which uninstalls dev-dependencies (except as needed for dependencies, having added typing-extensions to dependencies):

$ cd cryptol-remote-api/python
$ poetry install
...
$ poetry update --no-dev
Updating dependencies
Resolving dependencies... (15.6s)

Package operations: 0 installs, 0 updates, 3 removals

  • Removing mypy (0.812): Pending...
  • Removing mypy (0.812): Removing...
  • Removing mypy (0.812)
  • Removing mypy-extensions (0.4.3): Pending...
  • Removing mypy-extensions (0.4.3): Removing...
  • Removing mypy-extensions (0.4.3)
  • Removing typed-ast (1.4.3): Pending...
  • Removing typed-ast (1.4.3): Removing...
  • Removing typed-ast (1.4.3)

(Edited to reflect typing-extensions in main dependencies.)

[RyanGlScott]

#1336 adds an explicit typing-extensions dependency, which fixes the immediate problem. I'll leave this issue open to track the idea of only installing the non-developer dependencies in CI to catch mistakes like this in the future.