Fix conda install instructions after `setuptools` update #36411

mkoeppe · 2023-10-05T15:58:20Z

Updates install instructions for use with modern setuptools versions.
Restore bootstrap step in instructions, lost in Simplify experimental all-conda installation instructions via pkgs/sage-conf_conda #36367
Unpin setuptools.

📝 Checklist

The title is concise, informative, and self-explanatory.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation accordingly.

⌛ Dependencies

tobiasdiez · 2023-10-06T01:01:04Z

There are now build errors.

mkoeppe · 2023-10-06T21:12:58Z

possibly from Cython 3.0.3, which was installed in the conda environment. I'll test that version separately in #36416

mkoeppe · 2023-10-06T23:10:43Z

Yes, that did the trick.

Remaining error in pip check: "sagemath-standard 10.2b5 has requirement ipython<8.9.0,>=7.13.0, but you have ipython 8.16.1."

tobiasdiez · 2023-10-07T07:54:23Z

build/pkgs/cython/distros/conda.txt

@@ -1 +1 @@
-cython
+cython==3.0.2


Wouldn't it be better to exclude the broken 3.0.3 instead of fixing the version (here and in install-requires)? From what I read, the bug should be fixed in 3.0.4.

No, we have to expect further breakage in the 3.0.x series

Any basis for this expectation? Looking at #36110, there was no problem upgrading from 3.0.0 to 0.1 and 0.2.

I think we can always exclude newer versions once they are known to break something...

we can always exclude newer versions once they are known to break something...

I think I have explained this to you several times in the past already. This type of breakage affects releases that we have already made retroactively. As our project does not make bug fix releases, we cannot fix such breakage after it happens.

So when our experience indicates that we have to expect breakage from updates, it is prudent to put in suitable version constraints.

The breakage with 3.0.3 affected a major project (scipy), so this is an indication that the Cython project is currently struggling with release management. This is understandable because after the rollout of Cython 3 in distributions, there is likely very high demand for developer attention as lots of projects are forced to add support for Cython 3.

So when our experience indicates that we have to expect breakage from updates, it is prudent to put in suitable version constraints.

Yes, that's why you put in the major version constraint for cython. Bugs in other projects can always happen, but that's not a valid reason to fix the version. (A better solution in the form of a lock file is waiting for review: #35986)

For example, if we release sage with this version constraint, then it is no longer possible to use it in a conda environment with a newer version of cython. But someone might need 3.0.4 because earlier versions break of them.

So please loosen the constraint again. Thanks

that's not a valid reason to fix the version.

Sorry, Tobias, you don't get to declare that.

By that logic you would need to introduce a lot more fixed version constraints for a lot more packages. A lot of our dependencies have weaker ci than cython and thus may introduce new bugs.

For me these version constraints say "we expect that sage is compatible with these versions". Since that range usual contains future releases, it implicitly has the addition "of course, only given that the dependency doesn't introduce breaking changes/bugs".

From what I can see the cython issue that affects us is likely fixed in 3.0.4 (it already has patches discussed etc). So until you have evidence that there might be more bugs that affect us, I have the expectation that 3.0.4 works.

It's not helpful to repeat the same points every time that a version constraint has to be set. I'm fully aware of any and all of these points, and they are already factored in.

I'm fully aware of any and all of these points, and they are already factored in.

Perfect! In that case you might want to point out where our evaluation differs, since apparently we come to a different end result. For example, you might want to start with discussing why we didn't needed a fixed version before but now we do. For all I can see the cython 3 gets more stable and stable, so there are less and less bugs expected. By this logic, we would need a fixed version less and less.

Would the following compromise work for you: we simply exclude 3.0.3 for now, and if 3.0.4 breaks things again, we introduce an upper version constraint <= 3.0.2?

mkoeppe · 2023-10-09T03:05:35Z

Tobias, you misunderstand. I've marked the thread as resolved because there were no useful points in it, nor do I expect to learn anything new from it if were to continue. It's all empty calories.

tobiasdiez · 2023-10-09T06:52:47Z

An alternative is suggested in #35986 (ready for review), where the cython lock is present in the environment files that users/devs will us to generate the conda env, but not in the conda.txt. I think this is a nice compromise.

mkoeppe · 2023-10-10T05:17:33Z

I don't see the conda lock as a solution for this problem. It can be nice for creating stability for deployments on the application level, such as our gitpod or devcontainers.
But it's hardly a replacement for the thoughtful use of version ranges.

dimpase · 2023-10-11T10:43:59Z

I'm trying to run "Using conda to provide all dependencies for the Sage library (experimental)" on this branch, and get

$ pip install --no-build-isolation --config-settings editable_mode=compat -v -v --editable ./src
...
  adding 'sage_conf-9.8.dist-info/top_level.txt'
  adding 'sage_conf-9.8.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
  Building wheel for sage-conf (pyproject.toml) ... done
  Created wheel for sage-conf: filename=sage_conf-9.8-py3-none-any.whl size=5929 sha256=2452cdf180ceebae3ac425dc4be670ed835ba9e14dabadcca092846ec7d3003b
  Stored in directory: /home/scratch/dimpase/.cache/pip/wheels/42/8d/24/ee2834b08081c68f0473dced1cfa0c801641dffdf6648cb811
Successfully built sage-conf
Failed to build sagemath-standard
ERROR: Could not build wheels for sagemath-standard, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "/home/scratch/scratch2/dimpase/mambaforge/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/home/scratch/scratch2/dimpase/mambaforge/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch/scratch2/dimpase/mambaforge/envs/sage-dev/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 429, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for sagemath-standard, which is required to install pyproject.toml-based projects
Remote version of pip: 23.2.1
Local version of pip:  23.2.1
Was pip installed by pip? False

Any idea what it could be?

I don't know whether it's a regression or not, though.

dimpase · 2023-10-11T10:59:12Z

it appears that one needs to run ./bootstrap, not just ./bootstrap-conda, to set everything up properly (aftre which the error above went away)

So this part of the doc should be fixed.

tobiasdiez · 2023-10-11T13:25:17Z

Did you run pip install --no-build-isolation -v -v --editable ./pkgs/sage-conf_conda ./pkgs/sage-setup before this? This should call bootstrap internally (#36367)

dimpase · 2023-10-11T15:11:56Z

Did you run pip install --no-build-isolation -v -v --editable ./pkgs/sage-conf_conda ./pkgs/sage-setup before this? This should call bootstrap internally (#36367)

I did run

$ ./bootstrap-conda
$ mamba activate foo_env
(foo_env) $ pip install --no-build-isolation -v -v --editable ./pkgs/sage-conf_conda ./pkgs/sage-setup
(foo_env) $ pip install --no-build-isolation --config-settings editable_mode=compat -v -v --editable ./src

I don't see where ./bootstrap is called in sage/pkgs/sage-conf_conda/setup.py

I see it calls ./configure.

But honestly I think it's insane that pip would call ./bootstrap.

mkoeppe · 2023-10-11T16:17:52Z

I don't see where ./bootstrap is called in sage/pkgs/sage-conf_conda/setup.py

You are right, that was a mistake in #36367. I'll fix it here.

dimpase · 2023-10-11T19:21:34Z

There seems to be one more bug/feature in setup.py: configure is not run if config.status is present.

And if you have a stale config.status lying around, then there is a good chance you get a misconfiguration. Not sure how to resolve this in general, but here removing it first of all is a workaround..

mkoeppe · 2023-10-11T19:31:14Z

You are right, it should probably at least run ./config.status --recheck && ./config.status in this case, and maybe check that prefix is configured to be the same as the conda prefix

mkoeppe · 2023-10-12T00:57:39Z

This PR is introducing an additional test failure in interpreter.py.

Link to the failure please. There is no such failure in the Build & Test of this very PR.

tobiasdiez · 2023-10-12T01:19:25Z

e.g. https://github.com/sagemath/sage/actions/runs/6451909843/job/17624459499?pr=36431. I've opened #36453 for the cython version fix, so we can see if it's really related to the other changes of this PR here, or if its some updated dependency causing it.

mkoeppe · 2023-10-12T01:26:09Z

You mean this?

sage -t --random-seed=219809728447624716937891191463986234770 src/sage/repl/interpreter.py
**********************************************************************
File "src/sage/repl/interpreter.py", line 77, in sage.repl.interpreter
Failed example:
    print("dummy line"); shell.run_cell('1/0') # see #25320 for the reason of the `...` and the dummy line in this test
Expected:
    dummy line
    ...
    ZeroDivisionError...Traceback (most recent call last)
    ...
    ----> 1 Integer(1)/Integer(0)
    .../sage/rings/integer.pyx... in sage.rings.integer.Integer...div...
    ...
    -> ...                  raise ZeroDivisionError("rational division by zero")
       ...            x = <Rational> Rational.__new__(Rational)
       ...            mpq_div_zz(x.value, ....value, (<Integer>right).value)
    <BLANKLINE>
    ZeroDivisionError: rational division by zero
Got:
    dummy line
    ---------------------------------------------------------------------------
    ZeroDivisionError                         Traceback (most recent call last)
    Cell In[1], line 1
    ----> 1 Integer(1)/Integer(0)
    <BLANKLINE>
    File sage/rings/integer.pyx:2016, in sage.rings.integer.Integer.__truediv__()
    <BLANKLINE>
    ZeroDivisionError: rational division by zero
*********************************************************************

mkoeppe · 2023-10-12T01:28:31Z

The ipython version bound set here on this PR branch is the only thing that could be relevant for this doctest

tobiasdiez · 2023-10-12T02:58:27Z

Yes, that's the one. #36453 is working.

dimpase · 2023-10-12T07:56:42Z

It's a warning of sorts ("haha, just overwrote your config!")

please write warnings as warnings, not in this kind of haha-speak

dimpase · 2023-10-12T07:58:57Z

"re-using" might mean different things, e.g. not ignoring rather than ignoring the existing settings

dimpase · 2023-10-12T10:37:54Z

how about using the approach of #36453, i.e. not pinning a concrete version of Cython, but restricting the range and excluding 3.0.2, here?

…instructions

…s, lost in sagemath#36367

…message

mkoeppe · 2023-10-12T16:18:58Z

I've narrowed the scope of the PR, removing the change to the ipython (#36411 (comment)) and cython version constraints, and have removed the message.

mkoeppe · 2023-10-12T16:32:57Z

how about using the approach of #36453

See previous discussion here on this PR on this topic. To be continued on #36453.

mkoeppe · 2023-10-12T22:53:35Z

Thanks

…pdate   - Updates install instructions for use with modern `setuptools` versions. - Restore `bootstrap` step in instructions, lost in sagemath#36367 - Unpin `setuptools`.    ### 📝 Checklist     - [x] The title is concise, informative, and self-explanatory. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies   URL: sagemath#36411 Reported by: Matthias Köppe Reviewer(s): Dima Pasechnik, Matthias Köppe, Tobias Diez

mkoeppe added c: documentation c: distribution labels Oct 5, 2023

mkoeppe self-assigned this Oct 5, 2023

mkoeppe added s: needs review p: critical / 2 labels Oct 5, 2023

tobiasdiez added the s: run conda ci Run the conda workflow on this PR. label Oct 5, 2023

mkoeppe changed the title ~~Fix conda install instructions after setuptools update~~ Fix conda install instructions after setuptools update, pin cython to 3.0.2 Oct 6, 2023

tobiasdiez reviewed Oct 7, 2023

View reviewed changes

mkoeppe added p: blocker / 1 and removed p: critical / 2 labels Oct 7, 2023

tobiasdiez added s: needs work and removed s: needs review labels Oct 8, 2023

mkoeppe added s: needs review and removed s: needs work labels Oct 8, 2023

mkoeppe requested a review from dimpase October 8, 2023 16:01

This comment was marked as off-topic.

Sign in to view

mkoeppe requested a review from isuruf October 10, 2023 05:18

tobiasdiez removed the p: blocker / 1 label Oct 12, 2023

Matthias Koeppe added 5 commits October 12, 2023 09:11

conda: Do not pin setuptools, instead add legacy editable to install …

e3dce3f

…instructions

build/pkgs/setuptools_wheel: Make distros a symlink too

c60de9a

src/doc/en/installation/conda.rst: Restore 'bootstrap' in instruction…

756e5c4

…s, lost in sagemath#36367

pkgs/sage-conf_conda/setup.py: Always run configure

ddbbdc5

pkgs/sage-conf_conda/setup.py: Remove 'reusing configured SAGE_ROOT' …

d0cae80

…message

mkoeppe force-pushed the conda_legacy_editable branch from 6b33bd7 to d0cae80 Compare October 12, 2023 16:13

mkoeppe changed the title ~~Fix conda install instructions after setuptools update, pin cython to 3.0.2~~ Fix conda install instructions after setuptools update Oct 12, 2023

mkoeppe added the p: critical / 2 label Oct 12, 2023

dimpase approved these changes Oct 12, 2023

View reviewed changes

dimpase added s: positive review and removed s: needs review labels Oct 12, 2023

tobiasdiez mentioned this pull request Oct 13, 2023

Fix/ignore all remaining test failures when run under conda #36372

Closed

5 tasks

vbraun merged commit 1071779 into sagemath:develop Oct 21, 2023
21 of 31 checks passed

github-actions bot removed the s: positive review label Oct 21, 2023

mkoeppe added this to the sage-10.2 milestone Oct 21, 2023

mkoeppe deleted the conda_legacy_editable branch November 26, 2023 00:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix conda install instructions after `setuptools` update #36411

Fix conda install instructions after `setuptools` update #36411

mkoeppe commented Oct 5, 2023 •

edited

Loading

tobiasdiez commented Oct 6, 2023

mkoeppe commented Oct 6, 2023 •

edited

Loading

mkoeppe commented Oct 6, 2023

tobiasdiez Oct 7, 2023

mkoeppe Oct 7, 2023

tobiasdiez Oct 7, 2023

mkoeppe Oct 7, 2023

tobiasdiez Oct 8, 2023

mkoeppe Oct 8, 2023

tobiasdiez Oct 8, 2023

mkoeppe Oct 8, 2023

tobiasdiez Oct 8, 2023 •

edited

Loading

This comment was marked as off-topic.

mkoeppe commented Oct 9, 2023

tobiasdiez commented Oct 9, 2023

mkoeppe commented Oct 10, 2023

dimpase commented Oct 11, 2023

dimpase commented Oct 11, 2023 •

edited

Loading

tobiasdiez commented Oct 11, 2023

dimpase commented Oct 11, 2023

mkoeppe commented Oct 11, 2023

dimpase commented Oct 11, 2023

mkoeppe commented Oct 11, 2023

mkoeppe commented Oct 12, 2023

tobiasdiez commented Oct 12, 2023 •

edited

Loading

mkoeppe commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

tobiasdiez commented Oct 12, 2023

dimpase commented Oct 12, 2023

dimpase commented Oct 12, 2023

dimpase commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

		@@ -1 +1 @@
		cython
		cython==3.0.2

Fix conda install instructions after setuptools update #36411

Fix conda install instructions after setuptools update #36411

Conversation

mkoeppe commented Oct 5, 2023 • edited Loading

📝 Checklist

⌛ Dependencies

tobiasdiez commented Oct 6, 2023

mkoeppe commented Oct 6, 2023 • edited Loading

mkoeppe commented Oct 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tobiasdiez Oct 8, 2023 • edited Loading

Choose a reason for hiding this comment

This comment was marked as off-topic.

mkoeppe commented Oct 9, 2023

tobiasdiez commented Oct 9, 2023

mkoeppe commented Oct 10, 2023

dimpase commented Oct 11, 2023

dimpase commented Oct 11, 2023 • edited Loading

tobiasdiez commented Oct 11, 2023

dimpase commented Oct 11, 2023

mkoeppe commented Oct 11, 2023

dimpase commented Oct 11, 2023

mkoeppe commented Oct 11, 2023

mkoeppe commented Oct 12, 2023

tobiasdiez commented Oct 12, 2023 • edited Loading

mkoeppe commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

tobiasdiez commented Oct 12, 2023

dimpase commented Oct 12, 2023

dimpase commented Oct 12, 2023

dimpase commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

mkoeppe commented Oct 12, 2023

Fix conda install instructions after `setuptools` update #36411

Fix conda install instructions after `setuptools` update #36411

mkoeppe commented Oct 5, 2023 •

edited

Loading

mkoeppe commented Oct 6, 2023 •

edited

Loading

tobiasdiez Oct 8, 2023 •

edited

Loading

dimpase commented Oct 11, 2023 •

edited

Loading

tobiasdiez commented Oct 12, 2023 •

edited

Loading