pipenv lock fails when a dependent package depends on an another package served on non-default pypi server with pipenv 2022.4.8

[ysk24ok]

Issue description

We have a Pipfile like this, which installs mypackage from our private pypi server.

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "private"
url = "private pypi URL"
verify_ssl = true

[packages]
mypackage = {version = "==X.Y.Z", index = "private"}

[requires]
python_version = "3.8"

In setup.py of mypacakge, it depends on an another package named myanotherpackage and served on the same private pypi server.

setup(
    name="mypackage",
    version="X.Y.Z",
    install_requires=["myanotherpackage==A.B.C"],
    dependency_links=["private pypi URL"],
)

We'd like to update the dependencies by running pipenv lock.

Expected result

pipenv lock works fine and updates Pipfile.lock.

Actual result

With pipenv 2022.4.8, pipenv lock shows an error like this. Note that the package names and their versions are manually edited.

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement myanotherpackage==A.B.C (from mypackage) (from versions: 0.0.0)
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 822, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 771, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1066, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 862, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 663, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for myanotherpackage==A.B.C

WIth 2022.3.28, it works fine.

Steps to replicate

Just follow the steps described above if you have an alternative pypi server.

---

I'm sorry but I cannot paste the result of $ pipenv --support here as it contains the real name of the packages.

[matteius]

@ysk24ok This is because of a limitation of the way that index restrictions are implemented, that dependent packages get the default source. I realize based on your report that it worked in 2022.3.28 and I believe the change that caused it to be different is to apply the index restrictions to also the ignore_compatibility_finder as well since it had the same security issues as the regular finder. https://github.com/pypa/pipenv/pull/5024/files#diff-62e85156fc3b274046cc03dabfe6e3ceead503f330b98fb3611645216b4b6e1fR563-R582

If you can update the order of your sources so that the private index comes first, and your use case should work, though if you are not mirroring the pypi dependencies there you may need to pin your pypi packages by name to that index. If you did not wish to pin pypi packages to that index you would need to mirror them to your private package server, as recommended by pip maintainers: “To correctly make a private project installable is to point –index-url to an index that contains both PyPI and their private projects—which is our recommended best practice.” The pipenv documentation has been updated to describe this pattern in more detail, but please report back what your experience is like trying to update to follow the given advice: https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

Just noting that I am not familiar with the dependency_links part of the setup metdata -- perhaps that could be useful to extending the feature to know where to look for sub-dependencies, but that pipenv code can be super complicated and no guarantees it will be possible, and in the short term it likely will not be.

[ysk24ok]

Thank you for you quick response.

I realize based on your report that it worked in 2022.3.28 and I believe the change that caused it to be different is to apply the index restrictions to also the ignore_compatibility_finder as well since it had the same security issues as the regular finder.

So this behavior I reported is expected one, not a bug?

If you can update the order of your sources so that the private index comes first, and your use case should work,

In fact, mypackage depends on packages served on both our internal and public pypi server, so I guess flipping the order around wouldn't solve the problem.

To correctly make a private project installable is to point –index-url to an index that contains both PyPI and their private projects

Does that mean we have to put packages served on the public pypi server to our internal one? Not sure if it's possible for us...

I'm afraid I'm gonna deviate from the original question, but do you know how to specify an index for each dependency of a package?

If something like this is possible, it should be fine.

setup(
    name="mypackage",
    version="X.Y.Z",
    install_requires=[
        "myanotherpackage==A.B.C @ our internal pypi server",
        "requests @ public pypi server"
    ],
)

[matteius]

In fact, mypackage depends on packages served on both our internal and public pypi server, so I guess flipping the order wouldn't solve the problem.

@ysk24ok I think that it may still work for the ones that depend on pypi, at least I think that was my experience in testing but I also didn't have packages that depended on other packages in my own local pypi to test with and did most of my testing using a third-party torch repository. Maybe you could try it out and let me know if that for sure doesn't work?

I'm afraid I'm gonna deviate from the original question, but do you know how to specify an index for each dependency of a package?

This wouldn't be possible from the setup.py configuration that setuptools uses because it is a layer below pip and that is a layer below pipenv. I think you would need to add both packages to your top-level Pipfile, but there are other issue reports about not being able to specify the order of dependencies being installed, but they are old reports and I don't suspect that should be an issue for the current pip resolver to at least lock.

Also going to note for your case you may be interested in a feature option we are working on that would allow the installation to succeed from any of the sources based on an already locked Pipfile.lock, but the resolution of packages would still need to be possible to create the lock file for that to work, so it may not be of immediate use. This basically makes the sync step behave like before and search all indexes for packages that match that version and the allowed hashes, That PR is here: #5039

[matteius]

@ysk24ok If you find that it doesn't work to just flip the order of the indexes, you could at least add the other dependent pypi packages to your top-level Pipfile with the pinned index name for pypi, and that should definitely work to resolve everything, I believe.

[ysk24ok]

Flipping the order of indices around and specifying indices for public pypi explicitly didn't solve the problem.

[[source]]
name = "private"
url = "private pypi URL"
verify_ssl = true

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
mypackage = {version = "==X.Y.Z"}
requests = {version = "*", index = "pypi"}

[requires]
python_version = "3.8"

It shows the following error. I guess this is because mypackage depends on both internal/public pypi indices and pipenv couldn't find the packages for internal pypi.

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Cannot install -r /tmp/pipenv4grg0_z_requirements/pipenv-kcricj1v-constraints.txt (line 3) because these package versions have conflicting dependencies.
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 822, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 771, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1066, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 862, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 663, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Instead, listing all packages in our internal pypi in top-level Pipfile worked.

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "private"
url = "private pypi URL"
verify_ssl = true

[packages]
mypackage = {version = "==X.Y.Z", index = "private"}
myanotherpackage = {version = "==A.B.C", index = "private"}
requests = {version = "*"}

[requires]
python_version = "3.8"

Ideally it would be great that setuptools allows us to specify an index for each dependency, but as a workaround I think this is fine.
Thank you for your support! Let me close this issue.

[JacobHayes]

I just ran into this today too after upgrading. I'd rather not duplicate dependencies in both setup.py and Pipfile as we then have to maintain both files, eg: when bumping minimum versions, adding new deps, or removing deps (we have a lot of internal packages + transitive deps, so it'd be quite noisy).

This seems like a regression (I may have missed things in release notes) - any chance this could be reopened until a fix is in place? Thanks for the hard work on pipenv!

note: interestingly, this does seem to work ok on 2022.3.28, even though that's when the behavior seems to have changed, as per https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes:

Starting in release 2022.3.23 all packages are mapped only to a single package index for security reasons. All unspecified packages are resolved using the default index source; the default package index is PyPI.

[matteius]

@JacobHayes It is not a regression, but intentional to prevent package confusion attacks. I had missed part of the resolver code when the initial patch rolled out in March. The recommendation from pip is to mirror your pypi dependencies to your private package server so that you can just specify that as your default index and resolution will work the way you want it without extra specifiers. If you don't want to manage that pypi mirroring, then you would have to pin to the non-default indexes. Without further enhancements to how to know which index to search the sub-dependency for, and then actually search only that index its not currently possible and there is desire to remove --extra-index-url option from pip itself. Otherwise you end up with the possibility for the package confusion attack again. The documentation you link to was recent updated to explain these concerns and recommendations.

[matteius]

@JacobHayes I just thought of something else that might be helpful or could work. Since you mention that the setup.py requires the package, which calls out the version, and you don't want to manage the version in your Pipfile ... but you could add that other requirement to the Pipfile with {version: "*", index: "my-index"} and I think the resolver would combine that requirement with the setup.py and hopefully always resolve that transitive dependency to the right index. It would at least keep edits to when you add/remove a dependency and not when version bumps happen.

[juanitosvq]

I got here from #5021 - @matteius, the docs seem to describe pretty accurately what the new behaviour is, so thanks for that!

I would agree with @JacobHayes that having to maintain all those internal dependencies in the Pipfile of your application seems wrong. Even if we only had to do it whenever a dependency is added/removed.

We've temporarily pinned our pipenv version, but we will have to look into mirroring the pypi dependencies into our internal repo. It would be great to see, however, if it would be possible to specify the index in the setup.py of the library and have the resolver know in which index to find those internal dependencies, avoiding this way having to mirror pypi.