`batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.` #466

Ma27 · 2024-08-14T10:23:50Z

Given

Python 3.11.9 installed from nixpkgs (with setuptools 69.5.1.post0)
appenv at 2aee66bb3bd71eb2327afd654b7b09d862c39db3
batou 2.5.0b3

I get the following error when running ./appenv python:

Existing envdir not consistent, deleting
Creating venv ...
Ensuring pip ...
Installing ...
['/Users/ma27/Projects/deployment-batou/./.appenv/781ad8b9/bin/python', '-m', 'pip', 'check'] returned with exit code 1
batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

Traceback (most recent call last):
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 50, in cmd
    return subprocess.check_output(c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/pha8krh62hzq1z1j1hhmhsxn0kw9v439-python3-3.11.9/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/pha8krh62hzq1z1j1hhmhsxn0kw9v439-python3-3.11.9/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/ma27/Projects/deployment-batou/./.appenv/781ad8b9/bin/python', '-m', 'pip', 'check']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 609, in <module>
    main()
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 603, in main
    appenv.meta()
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 365, in meta
    args.func(args, remaining)
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 506, in python
    self.run('python', remaining)
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 368, in run
    self.prepare()
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 453, in prepare
    pip(env_dir, ["check"])
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 62, in pip
    return python(path, ["-m", "pip"] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 58, in python
    return cmd([os.path.join(path, "bin/python")] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 54, in cmd
    raise ValueError(e.output.decode("utf-8", "replace"))
ValueError: batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

Workaround for me was nix shell nixpkgs#python312.

I assume this is because appenv bootstraps itself from https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tgz which has a setuptools 65.5.0 bundled.

If I understand the issue correctly, the bump of setuptools to 66.1 breaks most of the Python versions supported by batou because of the way appenv works.

cc @zagy @elikoga

The text was updated successfully, but these errors were encountered:

Ma27 · 2024-08-14T10:31:01Z

Hmm, with Python 3.12 and batou 2.5b3 I get

      Exception: No module named 'botocore.vendored.six.moves'
Traceback (simplified, most recent call last):
  File "/Users/ma27/Projects/deployment-batou/components/redacted/component.py", line 11, in <module>
    import batou_ext.s3
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/batou_ext/s3.py", line 25, in <module>
    import boto3
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/botocore/session.py", line 29, in <module>
    import botocore.configloader
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/botocore/compat.py", line 34, in <module>
    from botocore.vendored.six.moves import http_client

The requirements.txt looked like this:

batou_ext[version-select] @ https://github.com/flyingcircusio/batou_ext/archive/1d185016c0049d64c509263b5028f75ca3c2d2ac.zip#sha256=ff87879d35c0bb77fb24022f2fd8eed0c605ada48befbc04ee0648faa817cabf
batou>=2.4
boto3
docutils==0.17.1
python-dateutil==2.8.1
s3transfer==0.4.2
jmespath==0.10.0

Guess I'll do batou>=2.4,<2.5.0b3 for now.

elikoga · 2024-08-14T11:44:19Z

What are the preferred python versions?

The bootstrapping mechanism should only trigger when import ensurepip errors. Nixpkgs Python 3.11 includes a working ensurepip though, so the old setuptools may come from somewhere else.

Ma27 · 2024-08-14T11:53:25Z

No preferences specified. Confirmed that prepending (# appenv-python-preference: 3.11,3.12) solves the issue and I cannot reproduce the boto3 issue in my comment above yours.

elikoga · 2024-08-14T11:56:21Z

Yes, we probably should create and merge https://github.com/flyingcircusio/appenv/compare/pip-freeze-all-package-regardless-of-version?expand=1

I'd love to merge https://github.com/flyingcircusio/appenv/compare/venv-create-update-deps?expand=1 but upgrade_deps was added in python 3.9 :/

elikoga · 2024-08-14T11:58:11Z

It looks like https://github.com/python/cpython/blob/3.11/Lib/ensurepip/__init__.py#L13 got us into this mess. When creating a venv, ensurepip is used to install pip (and setuptools as a pip dependency)

elikoga · 2024-09-03T09:03:12Z

Should be closed by #55

Ma27 · 2024-09-04T14:02:13Z

With batou 2.5.0 I still get

Traceback (most recent call last):
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 601, in <module>
    main()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 597, in main
    appenv.run(application_name, sys.argv[1:])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 368, in run
    self.prepare()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 453, in prepare
    pip(env_dir, ["check"])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 62, in pip
    return python(path, ["-m", "pip"] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 58, in python
    return cmd([os.path.join(path, "bin/python")] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 54, in cmd
    raise ValueError(e.output.decode("utf-8", "replace"))
ValueError: batou 2.5.0 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

while deploying.

Python is Python 3.11.9.
Reproducible with latest appenv (just downloaded the latest version of appenv.py which should be flyingcircusio/appenv@4692dff).

PhilTaken · 2024-09-04T14:10:02Z

Manually adding e.g. setuptools==66.1.0 to requirements.{txt,lock} then updating the lockfile via appenv fixes this issue

Ma27 · 2024-09-04T14:14:39Z

Sounds like underspecified dependencies somewhere then. I don't think we should maintain a setuptools dependency in our deployments.

PhilTaken · 2024-09-04T14:15:16Z

I suggest we move further discussion to flyingcircusio/appenv instead since it appears the lockfile does not include setuptools by default

elikoga · 2024-09-10T08:20:15Z

With batou 2.5.0 I still get

Traceback (most recent call last):
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 601, in <module>
    main()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 597, in main
    appenv.run(application_name, sys.argv[1:])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 368, in run
    self.prepare()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 453, in prepare
    pip(env_dir, ["check"])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 62, in pip
    return python(path, ["-m", "pip"] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 58, in python
    return cmd([os.path.join(path, "bin/python")] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 54, in cmd
    raise ValueError(e.output.decode("utf-8", "replace"))
ValueError: batou 2.5.0 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

while deploying.

Python is Python 3.11.9. Reproducible with latest appenv (just downloaded the latest version of appenv.py which should be flyingcircusio/appenv@4692dff).

After the update to the new appenv version, did the lockfile get rebuilt?

elikoga · 2024-09-10T09:07:15Z

Just checked, this error resolves once the lockfile is rebuilt using ./appenv update-lockfile.

We could

Connect the lockfile hash to the appenv version, such that new appenv versions see old lockfiles as invalid/outdated and prompt an update
Introduce an update command like in add self-updater appenv#39 or Allow updating the appenv script with a meta command appenv#1 that includes a lockfile update.

@Ma27 @PhilTaken what do you think of each of the solutions?

Ma27 · 2024-09-10T09:55:49Z

Yes indeed.

So what I think happened was

Updated something in a requirements.txt (with batou>=2.5.0b2)
./appenv update-lockfile works, but subsequent deploy fails
I remember to update appenv, but forgot to rerun the update (which seems kinda... obvious looking back now, sorry 😅 )

Personally, I'd prefer the latter option.
To me it still seems kinda awkward to just curl a new version of the new script when I need to update it.

Though, connecting the lockfile to appenv seems kinda orthogonal to the second approach: maybe we also want that since it generally seems a little more robust (however not as important to implement with (2) being done).

PhilTaken · 2024-09-10T12:40:40Z

I would strongly argue in favor of the first option.

It does not impact lockfile updates unless you update appenv, in which case you might as well update the requirements anyways - and should, as we can see here.

Running an appenv-specific update script is error-prone (pebcak) and tedious with no way to even tell if it has been run unless you also tack the appenv version into the lockfile at which point you've just implemented a worse version of the first option.

Ma27 · 2024-09-20T10:47:38Z

at which point you've just implemented a worse version of the first option.

It's a good point that you'll probably need option 1 to correctly implement option 2. If that's actually the case, I fully agree.

My point is mostly motivated from the fact that I'd expect an auto-updater to be easier to use as end-user here.

Anyways, there are issues for both things, so this can be closed for good :D

elikoga mentioned this issue Aug 14, 2024

Freeze all packages regardless of version flyingcircusio/appenv#55

Merged

elikoga closed this as completed Sep 3, 2024

Ma27 reopened this Sep 4, 2024

Ma27 closed this as completed Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.` #466

`batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.` #466

Ma27 commented Aug 14, 2024

Ma27 commented Aug 14, 2024

elikoga commented Aug 14, 2024

Ma27 commented Aug 14, 2024

elikoga commented Aug 14, 2024

elikoga commented Aug 14, 2024 •

edited

Loading

elikoga commented Sep 3, 2024

Ma27 commented Sep 4, 2024

PhilTaken commented Sep 4, 2024

Ma27 commented Sep 4, 2024

PhilTaken commented Sep 4, 2024 •

edited

Loading

elikoga commented Sep 10, 2024

elikoga commented Sep 10, 2024

Ma27 commented Sep 10, 2024

PhilTaken commented Sep 10, 2024

Ma27 commented Sep 20, 2024

batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0. #466

batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0. #466

Comments

Ma27 commented Aug 14, 2024

Ma27 commented Aug 14, 2024

elikoga commented Aug 14, 2024

Ma27 commented Aug 14, 2024

elikoga commented Aug 14, 2024

elikoga commented Aug 14, 2024 • edited Loading

elikoga commented Sep 3, 2024

Ma27 commented Sep 4, 2024

PhilTaken commented Sep 4, 2024

Ma27 commented Sep 4, 2024

PhilTaken commented Sep 4, 2024 • edited Loading

elikoga commented Sep 10, 2024

elikoga commented Sep 10, 2024

Ma27 commented Sep 10, 2024

PhilTaken commented Sep 10, 2024

Ma27 commented Sep 20, 2024

`batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.` #466

`batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.` #466

elikoga commented Aug 14, 2024 •

edited

Loading

PhilTaken commented Sep 4, 2024 •

edited

Loading