New and faster installer implementation

[sdispater]

Pull Request Check List

• Added tests for changed code.
• Updated documentation for changed code.

This PR introduces a new installation experience with a brand new installer that supports parallel operations. This was made possible by splitting the different parts of the installation process into several new classes:

• The Executor class is now the main point of entry and will be used by the existing Installer class if opted in via the use_executor method (note that the new installer will be opt-out by default since I want to gather as much feedback as possible). The Executor is responsible for executing a batch of operations (install, uninstall , update), in parallel by default. Note that not all operations will be executed in parallel but only those with the same depth in the dependency tree.
• The Chooser class is responsible for picking the appropriate distributions (sdist or wheel) for a given package and environment.
• The Chef class does not do much at the moment, it only manages the distributions cache, but will eventually be responsible for building wheels before package installation, this will be implemented in the 1.2 release.
• The Authenticator class is reponsible for handling credentials when downloading distributions from custom indices. Most of the ideas have been taken from the cache system of pip.

This is the first step towards the removal of calls to pip for installation of packages. We now use it exclusively to install distributions available locally.

The installer can be opted out via the experimental.new-installer setting:

poetry config experimental.new-installer false

Here is an example of the new installer output:

[abn]

A few recommendations. Haven't gone through the test cases yet and have largely skimmed the UI part of the executor.

[abn]

Should we consider moving this into a configuration constants section?

[pradyunsg]

Some, not 100% related to this PR, comments. Lemme know if this is less than useful. :)

[davidszotten]

hi. excited to try this out!

when i tried it i get a KeyError: interpreter_name in poetry/installation/chef.py:97 (key_parts["interpreter_name"] = self._env.marker_env["interpreter_name"])

i wonder if it's because we have virtualenvs.create false and self._env == SystemEnv("/usr/local")

[sdispater]

@davidszotten Thanks for reporting this!

There were actually missing elements for SystemEnv. It should be fixed now.

[sdispater]

@abn Your various points have been addressed and/or fixed.

[davidszotten]

could we not just (cancel tasks and) shutdown the executor and just return here? if i'm reading correctly nothing else happens if shutdown becomes True anyway

[sdispater]

It's just to have the same code path whether we have a KeyboardInterrupt or an actual error, since an error will also set _shutdown.

[davidszotten]

ah, missed that the tasks can also set _shutdown. thanks replied to the wrong comment

[davidszotten]

hm, something else has gone wrong in 35225a7. before, the status message (Pending, Downloading, Installing) would disappear after each package finished installing. Now it remains next to each package (so i can no longer tell what is still installing from what is done)

[sdispater]

@davidszotten Yes, I noticed that as well but it should be fixed now.

[MichaelAquilina]

minor, but you missed the type annotation here :) have you considered turning on disallow_untyped_defs?

[sdispater]

Thanks to everyone who reviewed this and especially @davidszotten for testing it thoroughly.

I am pretty confident that this is in a stable enough state to be released in the next alpha release of the 1.1.0 version so I'll go ahead and merge it.

If further bugs and issues are discovered we will address them in the next alpha (which should be the last) and beta releases.

[davidszotten]

🎉
thanks so much for your work on poetry!

[danihodovic]

@sdispater is it possible to track what makes an installation slower?

I'd like to debug the below install time to see if any package is causing problems.

$ poetry add watchdog@latest -D
Using version ^0.10.3 for watchdog
Updating dependencies
Resolving dependencies... (959.6s)

[abn]

@danihodovic I am unable to reproduce that in a new project.

$ poetry@1.1.0rc1 add watchdog 
Using version ^0.10.3 for watchdog

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing pathtools (0.1.2)
  • Installing watchdog (0.10.3)

Just to make sure we are not missing anything, what version of poetry are you using? If you are using 1.0.10 and the lock file was generated by a 1.1 release, resolving will not complete. Alternatively, it could be a reslult of some other dependencies in your pyproject.toml file. If the issue persists, please raise an issue, this PR is closed.

[danihodovic]

I'm not interested in opening a new bug on the slow dependency resolution as there are plenty open already. I want to know how to debug it when the dependency resolution takes an unreasonable amount of time.

[abn]

Use poetry add -D -vvv watchdog@latest.

[Vozf]

I've got some problems in gitlab ci using new poetry parallel installation.
the poetry install fails with

  • Installing holoviews (1.13.4)
  • Installing comet-ml (3.2.3)
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 0.5 seconds.

  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.8/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811


  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.8/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811

I don't even know which package causes this. It is likely not related to comet_ml as it is different package everytime. but it can be fixed with poetry config experimental.new-installer false

[earshinov]

We also face occasional errors when using new version of Poetry:

Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file
Package operations: 119 installs, 13 updates, 0 removals
  • Installing colorama (0.4.1)
  • Installing markupsafe (1.1.1)
  • Updating six (1.11.0 /usr/lib/python3/dist-packages -> 1.11.0)
  EnvCommandError
  Command ['/usr/bin/python3.8', '-m', 'pip', 'install', '--no-deps', '/root/.cache/pypoetry/artifacts/cf/46/78/4a92e28748cc243c738fb10b6dcaae48ea8ddc2684dfff839e0ce9f6c4/MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/cf/46/78/4a92e28748cc243c738fb10b6dcaae48ea8ddc2684dfff839e0ce9f6c4/MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl
  Installing collected packages: MarkupSafe
  ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/~ix-1.15.0.dist-info'
  
  
  at /usr/local/lib/python3.8/dist-packages/poetry/utils/env.py:948 in _run
       944│                 output = subprocess.check_output(
       945│                     cmd, stderr=subprocess.STDOUT, **kwargs
       946│                 )
       947│         except CalledProcessError as e:
    →  948│             raise EnvCommandError(e, input=input_)
       949│ 
       950│         return decode(output)
       951│ 
       952│     def execute(self, bin, *args, **kwargs):

For now we decided to run poetry install in CI in a loop like this:

RUN poetry_succeeded=;                 \
    for i in $(seq 3);                 \
    do                                 \
        if poetry install --no-root;   \
        then                           \
            poetry_succeeded=true;     \
            break;                     \
        fi;                            \
    done;                              \
    if [ -z "$poetry_succeeded" ];     \
    then                               \
        exit 1;                        \
    fi;                                \

[abn]

@earshinov you can disable the new installer using poetry config experimental.new-installer false for now.

[abn]

Locking this thread as issues reported here might be missed and get lost.

If there are issues with the installer, please first search existing issues. If an issue does not exist, please create one with details on how to reproduce it, -vvv logs and if possible a pyproject.toml file to use. This helps us in identifying the root cause of he issue and improving the installer in upcoming versions.

For issues related to the new installer, if a fix has not already been made for your issue, you may want to disable the new installer like so until the issue has been resolved.

poetry config experimental.new-installer false