Detect pyproject.toml changes and automatically run poetry install if needed

[mtkennerly]

When using Stack (Haskell) and Cargo (Rust), if you manually add a new dependency in package.yaml or Cargo.toml respectively, then the next invocation of stack test or cargo test will detect the change and install the new dependency, whereas poetry run pytest could fail on the dependency missing. It would be great if Poetry could auto-detect the change instead of having to manually run poetry install again. Of course, for an individual user, it's easy to go through the poetry add interface instead, but it's more problematic when working on a team since other people won't immediately know after git pull whether there were updates requiring a poetry install.

[cauebs]

Maybe warn the user instead of automatically installing. But I agree that detecting changes would be useful.

[jacebrowning]

I've achieved this behavior with a Makefile:

.PHONY: all
all: install

.PHONY: install
install: .venv/flag
.venv/flag: pyproject.lock
	@ poetry config settings.virtualenvs.in-project true
	poetry develop
	@ touch $@

pyproject.lock: pyproject.toml
	poetry lock

###############################################################################

.PHONY: test
test: install
	poetry run pytest

but I would be in favor of poetry doing this automatically.

[cauebs]

This is more important than I initially thought, as it's causing some very confusing things, e.g.: #292.

• How do we detect changes in pyproject.toml? Check the file's metadata for last modified time? Hash its contents?
• Where do we store these stamps? In pyproject.lock
• Then, whenever the lock file is used, that should be checked, right?

I can give this one a go, but I'll definitely need some pointers from @sdispater.

[ranisalt]

Check the file's metadata for last modified time?

This is faster and less resource-consuming, but there are known issues when depending on modification times.

I think that just as pyproject contains hashes to deps, a hash to pyproject.toml content is fine too.

[sdispater]

Poetry already tracks changes and discrepancies between pyproject.toml and pyproject.lock.
That's why you can sometimes see a warning when executing install.

Here is the corresponding line: https://github.com/sdispater/poetry/blob/master/poetry/installation/installer.py#L164

A similar approach could be used elsewhere if needed.

And I think displaying a warning (like in install) is preferable instead of installing automatically.

[cauebs]

Well, then why didn't he get a warning in #292?

[sdispater]

It seems like a bug to me. We do no track extras to generate the pyproject.toml hash.

See https://github.com/sdispater/poetry/blob/master/poetry/packages/locker.py#L15

We should add extras to this list

[cauebs]

While we're at it, what's the point of the lock command? How is it different to update? Only that it doesn't install anything?

[PavlosMelissinos]

locker.py seems to include extras now, is this issue resolved?

EDIT: nevermind, I mixed this up with #292. Anyway, what is the state of this?

[mtkennerly]

@PavlosMelissinos, the behavior is still the same as of v0.12.17 - no warning or automatic install:

$ poetry new issue-75
Created package issue-75 in issue-75

$ cd issue-75/

$ poetry install
Creating virtualenv issue-75-py3.5 in C:\tmp\issue-75\.venv
Installing dependencies from lock file


Package operations: 11 installs, 0 updates, 0 removals

  - Installing more-itertools (7.2.0)
  - Installing zipp (0.6.0)
  - Installing importlib-metadata (0.23)
  - Installing six (1.12.0)
  - Installing atomicwrites (1.3.0)
  - Installing attrs (19.3.0)
  - Installing colorama (0.4.1)
  - Installing pathlib2 (2.3.5)
  - Installing pluggy (0.13.0)
  - Installing py (1.8.0)
  - Installing pytest (3.10.1)
  - Installing issue-75 (0.1.0)

$ poetry run pytest
============================= test session starts =============================
platform win32 -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\tmp\issue-75, inifile:
collected 1 item

tests\test_issue_75.py .                                                 [100%]

========================== 1 passed in 0.05 seconds ===========================

Then edit files to add a dependency:

$ cat pyproject.toml
[tool.poetry]
name = "issue-75"
version = "0.1.0"
description = ""
authors = ["mtkennerly <mtkennerly@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.5"

[tool.poetry.dev-dependencies]
pytest = "^3.0"
requests = "^2.22"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

$ cat tests/test_issue_75.py
from issue_75 import __version__
import requests


def test_version():
    assert __version__ == '0.1.0'

$ poetry run pytest
============================= test session starts =============================
platform win32 -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\tmp\issue-75, inifile:
collected 0 items / 1 errors

=================================== ERRORS ====================================
___________________ ERROR collecting tests/test_issue_75.py ___________________
ImportError while importing test module 'C:\tmp\issue-75\tests\test_issue_75.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_issue_75.py:2: in <module>
    import requests
E   ImportError: No module named 'requests'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.15 seconds ===========================

Compare with Cargo for Rust:

$ cargo new issue-75-rs
     Created binary (application) `issue-75-rs` package

$ cd issue-75-rs/

$ cargo test
   Compiling issue-75-rs v0.1.0 (C:\tmp\issue-75-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 0.61s
     Running target\debug\deps\issue_75_rs-067f4b5baf064cb5.exe

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Then edit to add a dependency:

$ cat Cargo.toml
[package]
name = "issue-75-rs"
version = "0.1.0"
authors = ["mtkennerly <mtkennerly@gmail.com>"]
edition = "2018"

[dependencies]
derive-error = "0.0.4"

$ cargo test
    Updating crates.io index
   Compiling unicode-xid v0.0.4
   Compiling quote v0.3.15
   Compiling case v0.1.0
   Compiling synom v0.11.3
   Compiling syn v0.11.11
   Compiling derive-error v0.0.4
   Compiling issue-75-rs v0.1.0 (C:\tmp\issue-75-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 10s
     Running target\debug\deps\issue_75_rs-b9dbf4a0aa18a11e.exe

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[wagnerluis1982]

I wonder about the state of this issue. Is anyone working on it?

Anyway, I would like to point out some random ideas:

• By default, every subcommand (except self, help, search and maybe others) should have a hook to check lock state.
• So if I run poetry run or poetry build, for example, this hook would execute and act accordingly.
• By default, the action would be to notify the user, something like the existing message Warning: poetry.lock is not consistent with pyproject.toml...
• Optionally, this hook could be configurable as:
  • NOTIFY: to show the warning message
  • UPDATE: to run poetry update automatically
  • LOCK: to run poetry lock automatically

[neersighted]

Generally an open PR is the only sign somebody is working on a feature -- if you have a comprehensive proposal you want to examine/get feedback on before working on it in code, I'd suggest opening a new issue with the design proposal and a link back here.

[social-anthrax]

For people coming here looking for a solution to make sure their team's local lock file is always consistent with the project's pyproject.toml; you can set up a .pre-commit hook that is triggered on checkout, merge and rewrite. I've included the bash script and the optional pre-commit config to do so.

lock_and_update.sh:

#! /usr/bin/env bash
if ! poetry lock --check --quiet; then 
    poetry lock --no-update && poetry install
fi

Pre-commit config:

- repo: local
  hooks: 
    - id: poetry-lock-and-update
      name: "Update poetry lock and install"
      pass_filenames: false
      language: script
      stages:
        - "post-commit"
        - "post-checkout"
        - "post-merge"
        - "post-rewrite"
        - "manual"
      entry: lock_and_update.sh
      always_run: True

[social-anthrax]

I've created a pre-commit hook to do this for you. Can be found at https://github.com/social-anthrax/Poetry-Sync