pip: allow locking circular dependencies

[yajo]

Right now, pip module forbids doing that:

dream2nix/v1/nix/pkgs/fetchPipMetadata/src/fetch_pip_metadata/lock_file_from_report.py

Lines 136 to 141 in 548ced9

	if root_name in seen:
	print(
	f"fatal: cycle detected: {root_name} ({' '.join(seen)})",
	file=sys.stderr, # noqa: E501
	)
	sys.exit(1)

Although it seems a good idea, the fact is that one could overcome that circle later, using overrides. However, since you cannot pass the lock step, you can't get to the point where your overrides fix the problem.

My proposal is that there should be a cycle whitelist, so some packages can be fetched without stopping the locking if found in the whitelist. This makes the problem skipping explicit, which is good imho.

Context: trying to install manifestoo.

[phaer]

Hello!

Yes, the reason is that this way of locking was/is still quite experimental and while I didn't run into cycles myself while testing, I anticipated that those would cause problems with the way environment handlers are evaluated and so I put in a stop-gap for now.

Thanks for the example, I am going to look into what would be needed for manifestoo.

Please note that the main problem we are still trying to solve with fetchPipMetadata is to enable cross-platform generation of lock-files. As we are afraid we might not be able to support that without patching pip, I am considering switching the lock file generation away from pip to resolvelib, packaging + custom code entirely.

[yajo]

Ah cool, thanks!

Regarding cross-platform, maybe it's an overkill but... what about running pip through qemu? 😁

[phaer]

Regarding cross-platform, maybe it's an overkill but... what about running pip through qemu? 😁

Too resource-intensive, and can't legally cover macOS afaik

[yajo]

Did you know about these pip flags?

• https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-platform (see https://peps.python.org/pep-0599/ too)
• https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-python-version
• https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-implementation
• https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-abi

IIUC you can use a local-system pip to generate an appropriate lock file for any other system.

[phaer]

Yes, we know about pip installs flags, but those only affect the wheel selection process, not the requirement resolution (and ofc don't handle sdists at all), see e.g. pypa/pip#11664

[phaer]

@yajo You might also take this upstream to manifestoo, as PEP 517 recommends not to support circular dependencies:

Front ends SHOULD check explicitly for requirement cycles, and terminate the build with an informative message if one is found.

[yajo]

That PEP is talking about build dependencies, not runtime ones. Currently it's a nixpkgs weakness that both are treated equally. Refs:

• https://discourse.nixos.org/t/python-propagatedbuildinputs-could-we-do-better/29035?u=yajo
• Python: build wheels in separate derivation NixOS/nixpkgs#170577

[phaer]

That PEP is talking about build dependencies, not runtime ones.

Right, I got confused about that one, thanks for the clarification & sorry for the noise!

[jsirois]

@phaer I just noticed all this. Pex supports generating lock files for Linux and Mac and it supports cycles. It uses Pip under the covers. It also supports locking for a foreign platform. So, for example, locking Naked 0.1.31 which has a cycle on itself (chrissimpkins/naked#19) from a Linux machine using Python 3.10 but for any Mac using Python 3.7 through 3.11:

$ uname -a
Linux Gill-Windows 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ python -V
Python 3.10.6
$ pex3 lock create \
  --pip-version 23.2 \
  --style universal \
  --target-system mac \
  --interpreter-constraint "CPython>=3.7,<3.12" \
  "Naked==0.1.31" \
  -o lock.json \
  --indent 2

lock.json

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9",
              "url": "https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082",
              "url": "https://files.pythonhosted.org/packages/98/98/c2ff18671db109c9f10ed27f5ef610ae05b73bd876664139cf95bd1429aa/certifi-2023.7.22.tar.gz"
            }
          ],
          "project_name": "certifi",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "2023.7.22"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "8e098148dd37b4ce3baca71fb394c81dc5d9c7728c95df695d2dca218edf40e6",
              "url": "https://files.pythonhosted.org/packages/bf/a0/188f223c7d8b924fb9b554b9d27e0e7506fd5bf9cfb6dbacb2dfd5832b53/charset_normalizer-3.2.0-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "855eafa5d5a2034b4621c74925d89c5efef61418570e5ef9b37717d9c796419c",
              "url": "https://files.pythonhosted.org/packages/09/79/1b7af063e7c57a51aab7f2aaccd79bb8a694dfae668e8aa79b0b045b17bc/charset_normalizer-3.2.0-cp39-cp39-macosx_10_9_universal2.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "46fb8c61d794b78ec7134a715a3e564aafc8f6b5e338417cb19fe9f57a5a9bf2",
              "url": "https://files.pythonhosted.org/packages/0f/16/8d50877a7215d31f024245a0acbda9e484dd70a21794f3109a6d8eaeba99/charset_normalizer-3.2.0-cp311-cp311-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace",
              "url": "https://files.pythonhosted.org/packages/2a/53/cf0a48de1bdcf6ff6e1c9a023f5f523dfe303e4024f216feac64b6eb7f67/charset-normalizer-3.2.0.tar.gz"
            },
            {
              "algorithm": "sha256",
              "hash": "1a100c6d595a7f316f1b6f01d20815d916e75ff98c27a01ae817439ea7726329",
              "url": "https://files.pythonhosted.org/packages/79/55/9aef5046a1765acacf28f80994f5a964ab4f43ab75208b1265191a11004b/charset_normalizer-3.2.0-cp38-cp38-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "203f0c8871d5a7987be20c72442488a0b8cfd0f43b7973771640fc593f56321f",
              "url": "https://files.pythonhosted.org/packages/7b/c6/7f75892d87d7afcf8ed909f3e74de1bc61abd9d77cd9aab1f449430856c5/charset_normalizer-3.2.0-cp39-cp39-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "7c70087bfee18a42b4040bb9ec1ca15a08242cf5867c58726530bdf3945672ed",
              "url": "https://files.pythonhosted.org/packages/81/a0/96317ce912b512b7998434eae5e24b28bcc5f1680ad85348e31e1ca56332/charset_normalizer-3.2.0-cp310-cp310-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "4957669ef390f0e6719db3613ab3a7631e68424604a7b448f079bee145da6e09",
              "url": "https://files.pythonhosted.org/packages/8e/a2/77cf1f042a4697822070fd5f3f5f58fd0e3ee798d040e3863eac43e3a2e5/charset_normalizer-3.2.0-cp311-cp311-macosx_10_9_universal2.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "f779d3ad205f108d14e99bb3859aa7dd8e9c68874617c72354d7ecaec2a054ac",
              "url": "https://files.pythonhosted.org/packages/91/e6/8fa919fc84a106e9b04109de62bdf8526899e2754a64da66e1cd50ac1faa/charset_normalizer-3.2.0-cp311-cp311-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "c57921cda3a80d0f2b8aec7e25c8aa14479ea92b5b51b6876d975d925a2ea346",
              "url": "https://files.pythonhosted.org/packages/95/ee/8bb03c3518a228dc5956d1b4f46d8258639ff118881fba456b72b06561cf/charset_normalizer-3.2.0-cp37-cp37m-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "95eb302ff792e12aba9a8b8f8474ab229a83c103d74a750ec0bd1c1eea32e669",
              "url": "https://files.pythonhosted.org/packages/97/f6/0bae7bdfb07ca42bf5e3e37dbd0cce02d87dd6e87ea85dff43106dfc1f48/charset_normalizer-3.2.0-cp38-cp38-macosx_10_9_universal2.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "6339d047dab2780cc6220f46306628e04d9750f02f983ddb37439ca47ced7149",
              "url": "https://files.pythonhosted.org/packages/9c/71/bf12b8e0d6e1d84ed29c3e16ea1efc47ae96487bde823130d12139c434a0/charset_normalizer-3.2.0-cp38-cp38-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "0b87549028f680ca955556e3bd57013ab47474c3124dc069faa0b6545b6c9710",
              "url": "https://files.pythonhosted.org/packages/af/6f/b9b1613a5b672004f08ef3c02242b07406ff36164725ff15207737601de5/charset_normalizer-3.2.0-cp310-cp310-macosx_10_9_universal2.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "e857a2232ba53ae940d3456f7533ce6ca98b81917d47adc3c7fd55dad8fab858",
              "url": "https://files.pythonhosted.org/packages/d3/d8/50a33f82bdf25e71222a55cef146310e3e9fe7d5790be5281d715c012eae/charset_normalizer-3.2.0-cp39-cp39-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "a103b3a7069b62f5d4890ae1b8f0597618f628b286b03d4bc9195230b154bfa9",
              "url": "https://files.pythonhosted.org/packages/ec/a7/96835706283d63fefbbbb4f119d52f195af00fc747e67cc54397c56312c8/charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "charset-normalizer",
          "requires_dists": [],
          "requires_python": ">=3.7.0",
          "version": "3.2.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2",
              "url": "https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4",
              "url": "https://files.pythonhosted.org/packages/8b/e1/43beb3d38dba6cb420cefa297822eac205a277ab43e5ba5d5c46faf96438/idna-3.4.tar.gz"
            }
          ],
          "project_name": "idna",
          "requires_dists": [],
          "requires_python": ">=3.5",
          "version": "3.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "19de9961f4edb29e75cf837e8e031d6b52fbba4f0033515893d26f69c74b3b1f",
              "url": "https://files.pythonhosted.org/packages/02/36/b8107b51adca73402ec1860d88f41d958e275e60eea6eeaa9c39ddb89a40/Naked-0.1.31-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "12b76b8a14595d07039422f1d2219ca8fbef8b237f9cdf5d8e947c03e148677e",
              "url": "https://files.pythonhosted.org/packages/62/0e/7b009568d9bbd8c0e659f1ea286e68d80f4cc31daa18447299c39445541f/Naked-0.1.31.tar.gz"
            }
          ],
          "project_name": "naked",
          "requires_dists": [
            "Naked",
            "pyyaml",
            "requests"
          ],
          "requires_python": null,
          "version": "0.1.31"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c8098ddcc2a85b61647b2590f825f3db38891662cfc2fc776415143f599bb859",
              "url": "https://files.pythonhosted.org/packages/0e/88/21b2f16cb2123c1e9375f2c93486e35fdc86e63f02e274f0e99c589ef153/PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "f003ed9ad21d6a4713f0a9b5a7a0a79e08dd0f221aff4525a2be4c346ee60aab",
              "url": "https://files.pythonhosted.org/packages/28/09/55f715ddbf95a054b764b547f617e22f1d5e45d83905660e9a088078fe67/PyYAML-6.0.1-cp311-cp311-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8",
              "url": "https://files.pythonhosted.org/packages/57/c5/5d09b66b41d549914802f482a2118d925d876dc2a35b2d127694c1345c34/PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f",
              "url": "https://files.pythonhosted.org/packages/5b/07/10033a403b23405a8fc48975444463d3d10a5c2736b7eb2550b07b367429/PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "1d4c7e777c441b20e32f52bd377e0c409713e8bb1386e1099c2415f26e479595",
              "url": "https://files.pythonhosted.org/packages/7f/5d/2779ea035ba1e533c32ed4a249b4e0448f583ba10830b21a3cddafe11a4e/PyYAML-6.0.1-cp38-cp38-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a",
              "url": "https://files.pythonhosted.org/packages/96/06/4beb652c0fe16834032e54f0956443d4cc797fe645527acee59e7deaa0a2/PyYAML-6.0.1-cp310-cp310-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "b1275ad35a5d18c62a7220633c913e1b42d44b46ee12554e5fd39c70a243d6a3",
              "url": "https://files.pythonhosted.org/packages/c7/d1/02baa09d39b1bb1ebaf0d850d106d1bdcb47c91958557f471153c49dc03b/PyYAML-6.0.1-cp37-cp37m-macosx_10_9_x86_64.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43",
              "url": "https://files.pythonhosted.org/packages/cd/e5/af35f7ea75cf72f2cd079c95ee16797de7cd71f29ea7c68ae5ce7be1eda0/PyYAML-6.0.1.tar.gz"
            },
            {
              "algorithm": "sha256",
              "hash": "6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007",
              "url": "https://files.pythonhosted.org/packages/ec/0d/26fb23e8863e0aeaac0c64e03fd27367ad2ae3f3cccf3798ee98ce160368/PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl"
            }
          ],
          "project_name": "pyyaml",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "6.0.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f",
              "url": "https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1",
              "url": "https://files.pythonhosted.org/packages/9d/be/10918a2eac4ae9f02f6cfe6414b7a155ccd8f7f9d4380d62fd5b955065c3/requests-2.31.0.tar.gz"
            }
          ],
          "project_name": "requests",
          "requires_dists": [
            "PySocks!=1.5.7,>=1.5.6; extra == \"socks\"",
            "certifi>=2017.4.17",
            "chardet<6,>=3.0.2; extra == \"use-chardet-on-py3\"",
            "charset-normalizer<4,>=2",
            "idna<4,>=2.5",
            "urllib3<3,>=1.21.1"
          ],
          "requires_python": ">=3.7",
          "version": "2.31.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "de7df1803967d2c2a98e4b11bb7d6bd9210474c46e8a0401514e3a42a75ebde4",
              "url": "https://files.pythonhosted.org/packages/9b/81/62fd61001fa4b9d0df6e31d47ff49cfa9de4af03adecf339c7bc30656b37/urllib3-2.0.4-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "8d22f86aae8ef5e410d4f539fde9ce6b2113a001bb4d189e0aed70642d602b11",
              "url": "https://files.pythonhosted.org/packages/31/ab/46bec149bbd71a4467a3063ac22f4486ecd2ceb70ae8c70d5d8e4c2a7946/urllib3-2.0.4.tar.gz"
            }
          ],
          "project_name": "urllib3",
          "requires_dists": [
            "brotli>=1.0.9; platform_python_implementation == \"CPython\" and extra == \"brotli\"",
            "brotlicffi>=0.8.0; platform_python_implementation != \"CPython\" and extra == \"brotli\"",
            "certifi; extra == \"secure\"",
            "cryptography>=1.9; extra == \"secure\"",
            "idna>=2.0.0; extra == \"secure\"",
            "pyopenssl>=17.1.0; extra == \"secure\"",
            "pysocks!=1.5.7,<2.0,>=1.5.6; extra == \"socks\"",
            "urllib3-secure-extra; extra == \"secure\"",
            "zstandard>=0.18.0; extra == \"zstd\""
          ],
          "requires_python": ">=3.7",
          "version": "2.0.4"
        }
      ],
      "platform_tag": null
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.140",
  "pip_version": "23.2",
  "prefer_older_binary": false,
  "requirements": [
    "Naked==0.1.31"
  ],
  "requires_python": [
    "<3.12,>=3.7"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "target_systems": [
    "mac"
  ],
  "transitive": true,
  "use_pep517": null
}

And there is interoperability with Pip depending on where you want to stop using the Pex tool set:

$ pex3 lock export lock.json
naked==0.1.31 \
  --hash=sha256:19de9961f4edb29e75cf837e8e031d6b52fbba4f0033515893d26f69c74b3b1f \
  --hash=sha256:12b76b8a14595d07039422f1d2219ca8fbef8b237f9cdf5d8e947c03e148677e
pyyaml==6.0.1 \
  --hash=sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43
requests==2.31.0 \
  --hash=sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f \
  --hash=sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1
certifi==2023.7.22 \
  --hash=sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9 \
  --hash=sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082
charset-normalizer==3.2.0 \
  --hash=sha256:8e098148dd37b4ce3baca71fb394c81dc5d9c7728c95df695d2dca218edf40e6 \
  --hash=sha256:3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace
idna==3.4 \
  --hash=sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2 \
  --hash=sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4
urllib3==2.0.4 \
  --hash=sha256:de7df1803967d2c2a98e4b11bb7d6bd9210474c46e8a0401514e3a42a75ebde4 \
  --hash=sha256:8d22f86aae8ef5e410d4f539fde9ce6b2113a001bb4d189e0aed70642d602b11

If this is germane and you need more details feel free to ping me.

[phaer]

Hey @jsirois, thank you so much for much this!

I stumbled upon pex before, in the context of environment specifications in pip and quickly skimmed its docs back than, but didn't notice the lock file features back then.

Looks very promising at first glance, I'll take a deeper look and probably ping you soon :)