Upgrade to Python 3.12 #2072

max-muoto · 2024-01-04T16:53:14Z

Describe your changes

Upgrade base image to 3.12

max-muoto · 2024-01-04T16:59:17Z

Haven't tested this yet, but it would be nice if we could start looking at moving the base image to being 3.12.

.readthedocs.yaml

mathbunnyru · 2024-01-04T19:56:09Z

It seems we depend on this PR: conda-forge/pycurl-feedstock#27

mathbunnyru · 2024-01-07T09:40:50Z

Waiting for the upstream packages to be fixed.
@max-muoto if you want to move our docker stacks to Python 3.12 faster, please, consider helping conda-forge contributors by sending PRs to update packages.

I marked this as a draft and subscribed to the relevant PR, will re-run, when the issue resolves.

consideRatio · 2024-03-10T11:25:47Z

Issue for pycurl resolved upstream!

mathbunnyru · 2024-03-10T11:41:53Z

@consideRatio thanks!

@max-muoto could you please resolve conflicts? Unfortunately, GitHub is not able to do it in the web browser.

mathbunnyru · 2024-03-10T16:43:55Z

Something is wrong, as the change should have stayed small, just a few lines of code.

max-muoto · 2024-03-10T16:44:39Z

Something is wrong, as the change should have stayed small, just a few lines of code.

Currently rebasing, should be fixed soon.

max-muoto · 2024-03-10T17:07:00Z

Something is wrong, as the change should have stayed small, just a few lines of code.

Should be good to take a look at now.

max-muoto · 2024-03-10T17:51:18Z

Seems we also need to upgrade Pandas, just went ahead and did that.

mathbunnyru · 2024-03-10T17:52:56Z

images/pyspark-notebook/Dockerfile

@@ -63,7 +63,7 @@ USER ${NB_UID}
 RUN mamba install --yes \
    'grpcio-status' \
    'grpcio' \
-    'pandas=2.0.3' \
+    'pandas=2.2.1' \


Have you checked the comment above to make sure you’re using the proper version?

They have pandas<=2.2.1 so we should be good here.

https://github.com/apache/spark/blob/master/dev/infra/Dockerfile

You need to check the latest stable tag, not the current main branch.

Ah sorry there. Looks like we'll need for their next release, as this commit isn't included in the latest stable tag: ericm-db/spark@98ca3ea

I think we'll need to wait on this as well, since we need at least Pandas 2.1.1 to ensure compatibility with 3.12.

@bjornjorgensen could you please tell us when the Spark release will include this commit? (at least approximately)

hmm.. well we are waiting for hadoop 3.4.0 and a new hive release. We haven't started any RC release yet. I build and test my own jupyterlab https://github.com/bjornjorgensen/jupyter-spark-master-docker and I did try python 3.12 but it breake so match so I'm using python 3.11 as debian testing are using.
And Spark 3.5.1 don't support python 3.12 have a look at apache/spark#43922

Could we just revert this particular change rather than waiting for the next release of Spark? This seems like an incredibly self-imposed blocker.

They have pinned version 2.2.2 apache/spark@7c639a1

bjornjorgensen · 2024-03-10T21:00:34Z

python 3.10 was released Oct 04, 2021 we updated it #1711 May 28, 2022
python 3.11 was released Oct 24, 2022 we updated it #1844 May 31, 2023
I'm using Manjaro unstable and even here it's Python 3.11.8

bjornjorgensen · 2024-04-06T16:43:55Z

[VOTE] Release Plan for Apache Spark 4.0.0 (June 2024)

Apache Spark 4.0.0 Release Plan

After creating branch-3.5, set "4.0.0-SNAPSHOT" in master branch.
Creating branch-4.0 on April 1st, 2024.
Apache Spark 4.0.0 RC1 on May 1st, 2024.
Apache Spark 4.0.0 Release in June, 2024.

bjornjorgensen · 2024-05-09T11:13:14Z

there will be a test soon Re: [DISCUSS] Spark 4.0.0 release

shreve · 2024-06-11T01:21:43Z

images/pyspark-notebook/Dockerfile

@@ -63,7 +63,7 @@ USER ${NB_UID}
 RUN mamba install --yes \
    'grpcio-status' \
    'grpcio' \
-    'pandas=2.0.3' \
+    'pandas=2.2.1' \


Could we just revert this particular change rather than waiting for the next release of Spark? This seems like an incredibly self-imposed blocker.

benz0li · 2024-06-11T03:56:44Z

@shreve You may be interested in b-data's/my JupyterLab Python docker stack.
ℹ️ There are both Python 3.12 and Python 3.11 images available.

mj0nez · 2024-08-30T08:18:32Z

Hey, any updates on this? :)

mathbunnyru · 2024-08-30T09:20:39Z

Hey, any updates on this? :)

Unfortunately, this is still blocked - we're waiting for a new spark release.
@bjornjorgensen any updates here? :)

bjornjorgensen · 2024-09-06T21:04:43Z

https://lists.apache.org/thread/ygtg445xg1pk8yw5brrv206y7lm8dxwj

mathbunnyru · 2024-09-06T21:15:12Z

https://lists.apache.org/thread/ygtg445xg1pk8yw5brrv206y7lm8dxwj

Cool, thanks!
Hope it won't be long till spark 4.0 is finally released.

flaviomartins · 2024-09-18T10:31:40Z

https://lists.apache.org/thread/os2233zqxfslcp0x3poo47sdgj2g475p

benz0li · 2024-10-15T10:43:12Z

@shreve You may be interested in b-data's/my JupyterLab Python docker stack.
ℹ️ There are both Python 3.12 and Python 3.11 images available.

And now also Python 3.13.0 images – without numba, though.

bjornjorgensen · 2024-10-15T11:40:23Z

[DISCUSS] Creating branch-4.0 and Feature Freeze for Apache Spark 4.0

Spark 4.0 will be out in 2025

mathbunnyru · 2024-10-15T14:38:56Z

I think we have waited enough for a new Spark release and we should discuss switching to Python 3.12 earlier than "everything is 100% ready".

Reasons:

Python 3.13 has just been released, and it's over a year since the 3.12 release
There has been an increasing demand for images with newer Python
We are losing users who would like to use Python 3.12. People, who want to use Spark at the same time are free to use slightly old images with Python 3.11, so it's not totally breaking for them

@mathbunnyru @consideRatio @yuvipanda @manics let's vote on this one.
Please, vote by October 22nd.
If some option has 3 votes before the deadline, then I will accept it, if not, we'll go with the one that has the most votes (it should have at least 2 votes though).

I think there are several solutions, and you can choose several of them (please, select at least one).
Vote with 1️⃣, 2️⃣ and 3️⃣ in the comments.

1️⃣ Wait for Spark 4.0 then switch to Python 3.12
2️⃣ Switch now to Python 3.12, but keep Spark 3.5 (with outdated pandas, probably). I have no idea how much it is breaking for Spark users)
3️⃣ Switch now to Python 3.12 and Spark 4 previews (with quite modern pandas). This is breaking for users who don't use tagged images (and it's fine to break these users from time to time)

mathbunnyru · 2024-10-15T14:44:51Z

I'm voting for 3️⃣

I think with such a long release process it's ok to switch to preview versions.
I don't want to lose users because we're not using Python 3.12, so it's either 2️⃣ or 3️⃣.
I prefer 3️⃣ because new Spark builds probably work better with Python 3.12 (but I can't be 100% sure, since I haven't used Spark at all).
People who want to have Spark 3.5 are still gonna have previously built images.

manics · 2024-10-15T14:49:54Z

I agree with 3️⃣. I don't think it's a policy we should generally adopt, rather this is an exception due to how long it's taking to get a working combination. We can highlight the compromises made in the CHANGELOG

bjornjorgensen · 2024-10-15T18:19:34Z

Spark 4.0 preview2 does support python3.12 and now there are support for python3.13 in master branch.
I vote for 3. But make sure that users understand that it is a preview, not the real 4.0

I hope and guess that there will be more preview's before the 4.0 final.

Spark 4.0 preview2 have support for pandas 2.2.2 apache/spark#46009
and for the next preview there will be support for pandas 2.2.3 apache/spark#48269

mathbunnyru · 2024-10-19T18:09:42Z

I pushed a commit which makes Spark scripts more robust, this change is non-breaking and keeps everything the same, but the scripts will work better with newer spark and preview versions as well.
c4cb04e

mathbunnyru · 2024-10-20T13:04:12Z

So, the release schedule:

Monday, October 21st: automatic build, Python 3.11, Spark v3
Tuesday, October 22nd: Start using spark4-preview versions #2159 will be merged, Python 3.11, Spark v4 preview
Wednesday, October 23rd: Upgrade to Python 3.12 #2072 will be merged, Python 3.12, Spark v4 preview. When I merge the previous PR to the main branch, I will merge latest changes here and update the changelog - this PR will become really simple one.

It's better to merge 2 PRs separately (and on different days) - this might be helpful if something goes wrong.
And October 22nd is the perfect day to merge because it will be exactly a week since I started the poll.

mathbunnyru · 2024-10-22T12:03:34Z

All looks great so far - I merged the Spark v4 preview to main, merged changes from main to here, and updated the changelog and the list of old images. This PR now looks really simple (as it should be in most cases). I will merge this PR tomorrow.

mathbunnyru reviewed Jan 4, 2024

View reviewed changes

.readthedocs.yaml Outdated Show resolved Hide resolved

mathbunnyru marked this pull request as draft January 7, 2024 09:39

max-muoto force-pushed the python-3.12 branch 3 times, most recently from 2455425 to 9035f65 Compare March 10, 2024 17:06

max-muoto marked this pull request as ready for review March 10, 2024 17:06

Upgrade to 3.12

8f3dce1

max-muoto force-pushed the python-3.12 branch from 23a8e13 to 8f3dce1 Compare March 10, 2024 17:08

Upgrade Pandas

e4b6dad

mathbunnyru reviewed Mar 10, 2024

View reviewed changes

Merge branch 'main' into python-3.12

51a4943

Merge branch 'main' into python-3.12

4a9db95

shreve suggested changes Jun 11, 2024

View reviewed changes

mathbunnyru added 2 commits August 6, 2024 08:10

Merge branch 'main' into python-3.12

18ee4ad

Merge branch 'main' into python-3.12

6c4f7cb

mathbunnyru added 2 commits September 7, 2024 13:08

Merge branch 'main' into python-3.12

70769d9

Merge branch 'main' into python-3.12

122eccc

mathbunnyru added 3 commits September 29, 2024 21:57

Update Dockerfile

70b2ba6

Merge branch 'main' into python-3.12

72c166e

Update unit_pandas_version.py

7eb512a

mathbunnyru mentioned this pull request Oct 5, 2024

variable python version #2152

Closed

Merge branch 'main' into python-3.12

d49922f

Merge branch 'main' into python-3.12

5a449cd

Merge branch 'main' into python-3.12

d86c6d6

mathbunnyru mentioned this pull request Oct 20, 2024

Start using spark4-preview versions #2159

Merged

4 tasks

mathbunnyru added 8 commits October 21, 2024 09:32

Merge branch 'main' into python-3.12

392dbe8

Merge branch 'main' into python-3.12

779aa11

Update CHANGELOG.md

9abaf2a

Update CHANGELOG.md

0f5e3ce

Merge branch 'main' into python-3.12

f3bfbe0

Update CHANGELOG.md

9d6073a

Merge branch 'main' into python-3.12

a18bad4

Update README.md

56ff72e

Upgrade to Python 3.12 #2072

Are you sure you want to change the base?

Upgrade to Python 3.12 #2072

Conversation

max-muoto commented Jan 4, 2024

Describe your changes

max-muoto commented Jan 4, 2024

mathbunnyru commented Jan 4, 2024

mathbunnyru commented Jan 7, 2024

consideRatio commented Mar 10, 2024

mathbunnyru commented Mar 10, 2024

mathbunnyru commented Mar 10, 2024

max-muoto commented Mar 10, 2024

max-muoto commented Mar 10, 2024

max-muoto commented Mar 10, 2024

mathbunnyru Mar 10, 2024

Choose a reason for hiding this comment

max-muoto Mar 10, 2024 • edited Loading

Choose a reason for hiding this comment

max-muoto Mar 10, 2024

Choose a reason for hiding this comment

mathbunnyru Mar 10, 2024

Choose a reason for hiding this comment

max-muoto Mar 10, 2024

Choose a reason for hiding this comment

max-muoto Mar 10, 2024

Choose a reason for hiding this comment

mathbunnyru Mar 10, 2024

Choose a reason for hiding this comment

bjornjorgensen Mar 10, 2024

Choose a reason for hiding this comment

shreve Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

flaviomartins Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

bjornjorgensen commented Mar 10, 2024

bjornjorgensen commented Apr 6, 2024

Apache Spark 4.0.0 Release Plan

bjornjorgensen commented May 9, 2024

shreve Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

benz0li commented Jun 11, 2024 • edited Loading

mj0nez commented Aug 30, 2024

mathbunnyru commented Aug 30, 2024

bjornjorgensen commented Sep 6, 2024

mathbunnyru commented Sep 6, 2024

flaviomartins commented Sep 18, 2024

benz0li commented Oct 15, 2024

bjornjorgensen commented Oct 15, 2024

mathbunnyru commented Oct 15, 2024 • edited Loading

mathbunnyru commented Oct 15, 2024

manics commented Oct 15, 2024

bjornjorgensen commented Oct 15, 2024 • edited Loading

mathbunnyru commented Oct 19, 2024

mathbunnyru commented Oct 20, 2024 • edited Loading

mathbunnyru commented Oct 22, 2024

max-muoto Mar 10, 2024 •

edited

Loading

shreve Jun 11, 2024 •

edited

Loading

flaviomartins Sep 24, 2024 •

edited

Loading

shreve Jun 11, 2024 •

edited

Loading

benz0li commented Jun 11, 2024 •

edited

Loading

mathbunnyru commented Oct 15, 2024 •

edited

Loading

bjornjorgensen commented Oct 15, 2024 •

edited

Loading

mathbunnyru commented Oct 20, 2024 •

edited

Loading