Poetry 1.1.0a2 Breaks Support for File Dependencies on Networked Drives

[KyleKing]

• I am on the latest {*pre-release} Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Win10 1909

• Poetry version: 1.1.0a2

• Link of a Gist with the contents of your pyproject.toml file: See example toml snippet at bottom

Issue

Poetry 1.1.0a2 attempts to determine the relative path to a file dependency, but this fails (os.path.relpath raises a ValueError) if the dependency is not on the same drive. In my use case, we deploy wheel/tar files to a folder on a networked drive (\\company.com\dfsroot\etc...) so that the files are accessible to all internal users.

Poetry 1.1.0a1 does not have this issue, so I have downgraded to 1.1.0a1 for now

poetry/poetry/puzzle/provider.py

Lines 680 to 690 in feef53b

	if (package.dependency.is_directory() or package.dependency.is_file()) and (
	dep.is_directory() or dep.is_file()
	):
	relative_path = Path(
	os.path.relpath(
	dep.full_path.as_posix(), package.root_dir.as_posix()
	)
	)
	# TODO: Improve the way we set the correct relative path for dependencies
	dep._path = relative_path

Logs

poetry update -vvv

Using virtualenv: C:\Users\me\project_b\.venv
Updating dependencies
Resolving dependencies...
   1: fact: project_b is 0.0.0
   1: derived: project_b
   1: fact: project_b depends on project_a (0.1.1 \\company.com\dfsroot\project_a-0.1.1-py3-none-any.whl)
   1: selecting project_b (0.0.0)
   1: derived: project_a (0.1.1 \\company.com\dfsroot\project_a-0.1.1-py3-none-any.whl)

   1: Version solving took 0.540 seconds.
   1: Tried 1 solutions.

  Stack trace:

  13  ~\.poetry\lib\poetry\_vendor\py3.7\clikit\console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│ 
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  12  ~\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

  11  ~\.poetry\lib\poetry\_vendor\py3.7\clikit\api\command\command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│ 
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│
       173│     def __repr__(self):  # type: () -> str

  10  ~\.poetry\lib\poetry\_vendor\py3.7\cleo\commands\command.py:92 in wrap_handle
        90│         self._command = command
        91│
     →  92│         return self.handle()
        93│
        94│     def handle(self):  # type: () -> Optional[int]

   9  ~\.poetry\lib\poetry\console\commands\update.py:49 in handle
       47│         installer.update(True)
       48│ 
     → 49│         return installer.run()
       50│

   8  ~\.poetry\lib\poetry\installation\installer.py:74 in run
        72│ 
        73│         local_repo = Repository()
     →  74│         self._do_install(local_repo)
        75│
        76│         return 0

   7  ~\.poetry\lib\poetry\installation\installer.py:170 in _do_install
       168│             )
       169│
     → 170│             ops = solver.solve(use_latest=self._whitelist)
       171│         else:
       172│             self._io.write_line("Installing dependencies from lock file")

   6  ~\.poetry\lib\poetry\puzzle\solver.py:59 in solve
        57│         with self._provider.progress():
        58│             start = time.time()
     →  59│             packages, depths = self._solve(use_latest=use_latest)
        60│             end = time.time()
        61│

   5  ~\.poetry\lib\poetry\puzzle\solver.py:213 in _solve
       211│         try:
       212│             result = resolve_version(
     → 213│                 self._package, self._provider, locked=locked, use_latest=use_latest
       214│             )
       215│

   4  ~\.poetry\lib\poetry\mixology\__init__.py:7 in resolve_version
       5│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       6│
     → 7│     return solver.solve()
       8│

   3  ~\.poetry\lib\poetry\mixology\version_solver.py:84 in solve
        82│             while next is not None:
        83│                 self._propagate(next)
     →  84│                 next = self._choose_package_version()
        85│
        86│             return self._result()

   2  ~\.poetry\lib\poetry\mixology\version_solver.py:392 in _choose_package_version
       390│             return dependency.name
       391│
     → 392│         version = self._provider.complete_package(version)
       393│
       394│         conflict = False

   1  ~\.poetry\lib\poetry\puzzle\provider.py:685 in complete_package
       683│                 relative_path = Path(
       684│                     os.path.relpath(
     → 685│                         dep.full_path.as_posix(), package.root_dir.as_posix()
       686│                     )
       687│                 )

  ValueError

  path is on mount '\\\\company.com\\dfsroot', start on mount 'C:'

  at ~\AppData\Local\Continuum\anaconda3\lib\ntpath.py:562 in relpath
      558│         start_drive, start_rest = splitdrive(start_abs)
      559│         path_drive, path_rest = splitdrive(path_abs)
      560│         if normcase(start_drive) != normcase(path_drive):
      561│             raise ValueError("path is on mount %r, start on mount %r" % (
    → 562│                 path_drive, start_drive))
      563│
      564│         start_list = [x for x in start_rest.split(sep) if x]
      565│         path_list = [x for x in path_rest.split(sep) if x]
      566│         # Work out how much of the filepath is shared by start and path.

Example TOML

[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "project_b"
version = "0.0.0"
description = "Example toml with DFS-linked wheel file"
authors = [""]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies.project_a]
path = "\\\\company.com\\dfsroot\\project_a-0.1.1-py3-none-any.whl"

[KyleKing]

Just following up. This issue is still present in the latest Poetry 1.1.0b2

Could this just be dep._path = dep.full_path.as_posix()? Why is the relative path to package.root_dir.as_posix() necessary?

poetry/poetry/puzzle/provider.py

Lines 686 to 696 in 4769f72

	if (package.dependency.is_directory() or package.dependency.is_file()) and (
	dep.is_directory() or dep.is_file()
	):
	relative_path = Path(
	os.path.relpath(
	dep.full_path.as_posix(), package.root_dir.as_posix()
	)
	)
	# TODO: Improve the way we set the correct relative path for dependencies
	dep._path = relative_path

[abn]

@KyleKing can you try using a url dependency with direct-reference instead here? The issue with using a path dependency is that we expect it to be on the local machine. For network mounts, you will have to use file: direct refernces.

That said the specific code path you have pointed out needs to be fixed too, since we should really be testing if the path already is absolute and if so deal without using a relative path. But I also suspect this will have some downstream implications. Eitherway, for your use case I feel a url dependency is more appropriate.

We did make an improvement in python-poetry/poetry-core#22 to support this properly. Would be good to verify that it works.

[KyleKing]

Okay, that makes sense. I just have a couple of quick questions about modifying the toml file and how can I test pip @ file://... separate of poetry

1. I looked through the tests from Improve support for PEP-440 direct references poetry-core#22 and the poetry website and couldn't determine what the TOML file should look like to specify a direct-reference (url=file:///, right?). Based on PEP440 and the MDN note (i.e. file://laptop/My%20Documents/FileSchemeURIs.doc), I think the URL should be: file:///company.com/dfsroot/project_a-0.1.1-py3-none-any.whl

2. With pip 19.0.3 (version with poetry), how can I test direct-references directly? I tried pip @ file:///company.com/dfsroot/project_a-0.1.1-py3-none-any.whl but get an error that pip doesn't recognize the "@" command

---

For the hypothetical TOML file for this ticket (old way):

[tool.poetry.dev-dependencies.project_a]
path = "\\\\company.com\\dfsroot\\project_a-0.1.1-py3-none-any.whl"

Should this be correct?

[tool.poetry.dev-dependencies.project_a]
url = "file:///company.com/dfsroot/project_a-0.1.1-py3-none-any.whl"

> poetry --version
Poetry version 1.1.0b2

When I try the latter, I get a No connection adapters were found for ... when trying to URL parse the file (full stack trace in details below)

>poetry update -vvv
Stack trace:
11  ~.poetry\lib\poetry_vendor\py3.7\clikit\console_application.py:131 in run

129│             parsed_args = resolved_command.args

130│

→ 131│             status_code = command.handle(parsed_args, io)

132│         except KeyboardInterrupt:

133│             status_code = 1
10  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\command\command.py:120 in handle

118│     def handle(self, args, io):  # type: (Args, IO) -> int

119│         try:

→ 120│             status_code = self._do_handle(args, io)

121│         except KeyboardInterrupt:

122│             if io.is_debug():
9  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\command\command.py:163 in _do_handle

161│         if self._dispatcher and self._dispatcher.has_listeners(PRE_HANDLE):

162│             event = PreHandleEvent(args, io, self)

→ 163│             self._dispatcher.dispatch(PRE_HANDLE, event)

164│

165│             if event.is_handled():
8  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\event\event_dispatcher.py:22 in dispatch

20│

21│         if listeners:

→  22│             self._do_dispatch(listeners, event_name, event)

23│

24│         return event
7  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\event\event_dispatcher.py:89 in _do_dispatch

87│                 break

88│

→  89│             listener(event, event_name, self)

90│

91│     def _sort_listeners(self, event_name):  # type: (str) -> None
6  ~.poetry\lib\poetry\console\config\application_config.py:116 in set_env

114│

115│         io = event.io

→ 116│         poetry = command.poetry

117│

118│         env_manager = EnvManager(poetry)
5  ~.poetry\lib\poetry\console\commands\command.py:10 in poetry

8│     @property

9│     def poetry(self):

→ 10│         return self.application.poetry

11│

12│     def reset_poetry(self):  # type: () -> None
4  ~.poetry\lib\poetry\console\application.py:69 in poetry

67│             return self._poetry

68│

→  69│         self._poetry = Factory().create_poetry(Path.cwd())

70│

71│         return self._poetry
3  ~.poetry\lib\poetry\factory.py:33 in create_poetry

31│             io = NullIO()

32│

→  33│         base_poetry = super(Factory, self).create_poetry(cwd)

34│

35│         locker = Locker(
2  ~.poetry\lib\poetry_vendor\py3.7\poetry\core\factory.py:84 in create_poetry

82│                     continue

83│

→  84│                 package.add_dependency(name, constraint)

85│

86│         if "dev-dependencies" in local_config:
1  ~.poetry\lib\poetry_vendor\py3.7\poetry\core\packages\package.py:359 in add_dependency

357│             elif "url" in constraint:

358│                 dependency = URLDependency(

→ 359│                     name, constraint["url"], category=category, optional=optional

360│                 )

361│             else:
ValueError
file:///company.com/dfsroot/project_a-0.1.1-py3-none-any.whl does not seem like a valid url
at ~.poetry\lib\poetry_vendor\py3.7\poetry\core\packages\url_dependency.py:18 in init

14│         self._url = url

15│

16│         parsed = urlparse.urlparse(url)

17│         if not parsed.scheme or not parsed.netloc:

→ 18│             raise ValueError("{} does not seem like a valid url".format(url))

19│

20│         super(URLDependency, self).init(

21│             name, "*", category=category, optional=optional, allows_prereleases=True

22│         )

Alternatively, swapping url with path, I get a [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:\\Users\\user\\project_folder\\file:\\company.com\\dfsroot\\project_a-0.1.1-py3-none-any.whl' because of the relative path combination mentioned above (full stack trace in details below)

>poetry update -vvv
Stack trace:
12  ~.poetry\lib\poetry_vendor\py3.7\clikit\console_application.py:131 in run

129│             parsed_args = resolved_command.args

130│

→ 131│             status_code = command.handle(parsed_args, io)

132│         except KeyboardInterrupt:

133│             status_code = 1
11  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\command\command.py:120 in handle

118│     def handle(self, args, io):  # type: (Args, IO) -> int

119│         try:

→ 120│             status_code = self._do_handle(args, io)

121│         except KeyboardInterrupt:

122│             if io.is_debug():
10  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\command\command.py:163 in _do_handle

161│         if self._dispatcher and self._dispatcher.has_listeners(PRE_HANDLE):

162│             event = PreHandleEvent(args, io, self)

→ 163│             self._dispatcher.dispatch(PRE_HANDLE, event)

164│

165│             if event.is_handled():
9  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\event\event_dispatcher.py:22 in dispatch

20│

21│         if listeners:

→  22│             self._do_dispatch(listeners, event_name, event)

23│

24│         return event
8  ~.poetry\lib\poetry_vendor\py3.7\clikit\api\event\event_dispatcher.py:89 in _do_dispatch

87│                 break

88│

→  89│             listener(event, event_name, self)

90│

91│     def _sort_listeners(self, event_name):  # type: (str) -> None
7  ~.poetry\lib\poetry\console\config\application_config.py:116 in set_env

114│

115│         io = event.io

→ 116│         poetry = command.poetry

117│

118│         env_manager = EnvManager(poetry)
6  ~.poetry\lib\poetry\console\commands\command.py:10 in poetry

8│     @property

9│     def poetry(self):

→ 10│         return self.application.poetry

11│

12│     def reset_poetry(self):  # type: () -> None
5  ~.poetry\lib\poetry\console\application.py:69 in poetry

67│             return self._poetry

68│

→  69│         self._poetry = Factory().create_poetry(Path.cwd())

70│

71│         return self._poetry
4  ~.poetry\lib\poetry\factory.py:33 in create_poetry

31│             io = NullIO()

32│

→  33│         base_poetry = super(Factory, self).create_poetry(cwd)

34│

35│         locker = Locker(
3  ~.poetry\lib\poetry_vendor\py3.7\poetry\core\factory.py:84 in create_poetry

82│                     continue

83│

→  84│                 package.add_dependency(name, constraint)

85│

86│         if "dev-dependencies" in local_config:
2  ~.poetry\lib\poetry_vendor\py3.7\poetry\core\packages\package.py:336 in add_dependency

334│

335│                 if self.root_dir:

→ 336│                     is_file = (self.root_dir / path).is_file()

337│                 else:

338│                     is_file = path.is_file()
1  ~\AppData\Local\Continuum\anaconda3\lib\pathlib.py:1372 in is_file

1370│         """

1371│         try:

→ 1372│             return S_ISREG(self.stat().st_mode)

1373│         except OSError as e:

1374│             if not _ignore_error(e):
OSError
[WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:\Users\user\project_folder\file:\company.com\dfsroot\project_a-0.1.1-py3-none-any.whl'
at ~\AppData\Local\Continuum\anaconda3\lib\pathlib.py:1168 in stat

1164│

1165│         Return the result of the stat() system call on this path, like

1166│         os.stat() does.

1167│         """

→ 1168│         return self._accessor.stat(self)

1169│

1170│     def owner(self):

1171│

1172│         Return the login name of the file owner.

[abn]

Hmm, thanks for the details @KyleKing much appreciated. It might be better to handle this as a file dependency (as you originally did) I reckon. I forgot that poetry handles file uri's as file dependnecies as opposed to url dependency, so my original recommendation is moot

As for the the issue itself, I think adding an "is absolute" test might work. Assuming that pathlib correcly recognises the mount path as an absolute path. Would be great if you can test that solution in your environment. I do not readily have access to a viable environment to test this.

Regarding the URI, you are correct that file:////company.com/dfsroot/project_a-0.1.1-py3-none-any.whl is what would be expected. For testing this you can just do poetry run pip install -U pip in your current venv before trying, or recreate the venv since poetry-1.1.0b1 now uses virtualenv to create a new environment with pip-20.x.

[KyleKing]

Oh okay, I tested pathlib with this snippet:

from pathlib import Path

full_path = Path('\\\\company.com\\dfsroot\\project_a-0.1.1-py3-none-any.whl')
full_path.is_file()  # True
full_path.is_absolute()  # True

Do you want me to submit a PR with this change? Something like:

 if (package.dependency.is_directory() or package.dependency.is_file()) and ( 
     dep.is_directory() or dep.is_file() 
 ): 
     if (dep.full_path.is_absolute()) {
        dep._path = dep.full_path.as_posix()
     } else {
        relative_path = Path( 
            os.path.relpath( 
                dep.full_path.as_posix(), package.root_dir.as_posix() 
            ) 
        ) 
    
        # TODO: Improve the way we set the correct relative path for dependencies 
        dep._path = relative_path
     }

Although I think this change may be better grouped in a PR with changes that resolve the relative path TODO item

[KyleKing]

Just following up, the latest poetry 1.1.1 (and 1.1.4) version won't let me use dependencies on mounted drive locations. Can I help make code changes? What would be the desired behaviour of poetry in these cases?

[KyleKing]

@python-poetry/triage I submitted PR (#3466 ) that resolves this issue. Is there any additional input that I can provide?

[KyleKing]

Still an issue, but not worth nudging maintainers until reviewed

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.