python311: enable windows cross-compilation, make win-dll-hook link to distant dependancies #227818

marius851000 · 2023-04-23T14:49:02Z

Description of changes

A minimal set of change to enable cross-compilation of the python 3.11 interpreter to windows (pkgsCross.mingwW64.python311), thanks to patch used by fedora.

It also change win-dll-hook to not only link necessary dlls from all the outputs in the binary folder, but also those in distant package, based on a new environment variable for Windows cross-build, LINK_DLL_FOLDERS, that list folders containing DLLs that may be copied to the binary folder.

(for exemple, it will move libgcc_s_seh-1.dll, libwinpthread-1.dll, libpython3.11.dll and mcfgthread-12.dll in the bin folder of python311, the python fixup phase will do the same for compiled python dependancies, that are stored in a different folder)

It also cause a world rebuild due to the change in the gcc build script and the generic CC wrapper.

Note that this is only sufficient for compiling the interpreter itself, but not dependancies. I’ll make another PR improving the makeBinaryWrapper, that should also replace shebang with an executable for Windows before trying to take a deeper look at compiling dependancies.

ps: Only tested with Wine on my machine. I haven’t tested on Windows, and I’m a bit worried in regard to the Nix store. Ideally, everything use a relative path, so I can just copy a folder containing all run-time derivation and move it around.

Things done

I tried to make particularly sure that change that are not behind an isWindows or similar doesn’t have side-effect for other packages, althought I didn’t tried to re-compile everything.

superseed #84815

marius851000 · 2023-04-24T06:52:45Z

Seems that both failed builds are due to timeout.

marius851000 · 2023-04-24T08:01:22Z

Additionnally, this should fix #38451 and partially replace #38632 . specific mention to @Ericson2314 , which seems interested by that.

marius851000 · 2023-04-24T08:05:21Z

Seems that I messed the part that try to not copy dll already loaded by python into the dynload folder. I’ll fix that later today.

Artturin · 2023-08-27T17:58:12Z

pkgs/development/libraries/gettext/default.nix

@@ -52,7 +52,7 @@ stdenv.mkDerivation rec {
    xz
    xz.bin
  ];
-  buildInputs = [ bash ]
+  buildInputs = lib.optionals (!stdenv.hostPlatform.isWindows) [ bash ]


can be dropped 7bc650e

pkgs/development/interpreters/python/cpython/default.nix

Artturin · 2023-08-27T18:49:45Z

Non python non hook changes split to #251828

Artturin · 2023-09-05T16:10:46Z

pkgs/development/interpreters/python/hooks/default.nix

+  pythonNamespacesHook = callPackage ({ makePythonHook, buildPackages }:
    makePythonHook {
      name = "python-namespaces-hook.sh";
      substitutions = {
-        inherit pythonSitePackages findutils;
+        inherit pythonSitePackages;
+        findutils = buildPackages.findutils;


should be unneeded since #245475

Artturin · 2023-09-05T16:47:55Z

Rebased and tweaked, now only one commit remains! 😄

Ericson2314 · 2023-09-05T18:47:10Z

@Artturin Do you want to hack this up to make it a non-mass-rebuild, and then fix up the code (to be like it is now) on staging afterwards?

Artturin · 2023-09-06T21:15:03Z

@Artturin Do you want to hack this up to make it a non-mass-rebuild, and then fix up the code (to be like it is now) on staging afterwards?

Should be like that now

Artturin · 2023-09-06T22:14:41Z

pkgs/development/interpreters/python/cpython/default.nix

+# these dont build for windows
+, withGdbm ? !stdenv.hostPlatform.isWindows
+, withReadline ? !stdenv.hostPlatform.isWindows


Fedora has windows patches for these too

https://src.fedoraproject.org/rpms/mingw-gdbm/tree/rawhide
https://src.fedoraproject.org/rpms/mingw-readline/tree/rawhide

for readline

diff --git a/pkgs/development/libraries/readline/8.2.nix b/pkgs/development/libraries/readline/8.2.nix index 1c53da3cdfa4..eca19897e73e 100644 --- a/pkgs/development/libraries/readline/8.2.nix +++ b/pkgs/development/libraries/readline/8.2.nix @@ -1,4 +1,4 @@ -{ fetchurl, stdenv, lib, ncurses +{ fetchurl, stdenv, lib, ncurses, fetchgit }: stdenv.mkDerivation rec { @@ -33,6 +33,23 @@ stdenv.mkDerivation rec { ] ++ upstreamPatches; + postPatch = if (stdenv.hostPlatform.isMinGW) then ( + let + # https://src.fedoraproject.org/rpms/mingw-gdbm + readline-patch = fetchgit { + name = "mingw-readline-patches"; + url = "https://src.fedoraproject.org/rpms/mingw-readline.git"; + rev = "c15e77591106244a9f1c318226167d5bcea7bf04"; # patches for readline 8.2 + sha256 = "sha256-QcMJ/nq2T4QlLkw0tAutMohAB0zWa1mRUYuN7D/Kvh0="; + }; + in + '' + for p in ${readline-patch}/*.patch; do + patch -p1 -d . -i $p + done + '' + ) else null; + meta = with lib; { description = "Library for interactive line editing"; @@ -57,7 +74,7 @@ stdenv.mkDerivation rec { maintainers = with maintainers; [ dtzWill ]; - platforms = platforms.unix; + platforms = platforms.unix ++ platforms.windows; branch = "8.2"; }; }

readline-x86_64-w64-mingw32> /nix/store/8gfd58vf0arh22p7bc8da0qidrydfi4b-x86_64-w64-mingw32-binutils-2.40/bin/x86_64-w64-mingw32-ld: cannot find -lncurses: No such file or directory readline-x86_64-w64-mingw32> /nix/store/8gfd58vf0arh22p7bc8da0qidrydfi4b-x86_64-w64-mingw32-binutils-2.40/bin/x86_64-w64-mingw32-ld: cannot find -ltermcap: No such file or directory

fedora has mingw-termcap https://src.fedoraproject.org/rpms/mingw-termcap/blob/rawhide/f/mingw-termcap.spec

which has a comment

# Note: Termcap was deprecated and removed from Fedora after F-8. It # has been replaced by ncurses. However ncurses cannot be compiled on # Windows so we have to supply termcap.

but pkgsCross.mingwW64.ncurses builds for us and is already in propagatedBuildInputs but does not work.

the gdbm patch is for 1.11 and doesn't apply to our 1.23

Artturin · 2023-09-06T22:27:02Z

pkgs/development/interpreters/python/cpython/default.nix

+    # Both fail when building for windows, normally configure checks this by itself but on other platforms this is set to yes always.
+    "ac_cv_file__dev_ptmx=${if stdenv.hostPlatform.isWindows then "no" else "yes"}"
+    "ac_cv_file__dev_ptc=${if stdenv.hostPlatform.isWindows then "no" else "yes"}"


Removed the in-tree patch and did this instead.

Artturin · 2023-09-06T23:14:14Z

pkgs/development/interpreters/python/cpython/default.nix

+    mingw-patch = fetchgit {
+      name = "mingw-python-patches";
+      url = "https://src.fedoraproject.org/rpms/mingw-python3.git";
+      rev = "45c45833ab9e5480ad0ae00778a05ebf35812ed4"; # for python 3.11.5 at the time of writing.


I updated the rev to python 3.11.5 even though the patch files haven't changed.

Artturin

Merge after eval is done and 0 rebuilds

marius851000 · 2023-09-08T05:57:20Z

Thanks, and sorry for not having fixed that myselr. Kept pushing that back to tomorrow since I'm back from vacation. For now, my mind is rather away to Nixpkgs and into other projects (maybe NixCon's stream will change that, maybe not).

Have a nice day.

marius851000 requested review from matthewbauer, FRidh, jonringer and Ericson2314 as code owners April 23, 2023 14:49

github-actions bot added the 6.topic: python label Apr 23, 2023

ofborg bot added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label Apr 23, 2023

marius851000 force-pushed the python_win branch 2 times, most recently from 43c4fe4 to 96b698d Compare April 23, 2023 15:20

ofborg bot added 10.rebuild-linux-stdenv This PR causes stdenv to rebuild 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Apr 23, 2023

marius851000 force-pushed the python_win branch from 96b698d to 14f3b85 Compare April 24, 2023 19:43

wegank added the 6.topic: exotic Exotic hardware or software platform label May 6, 2023

Artturin added the 6.topic: windows Running, or buiding, packages on Windows label Aug 27, 2023

Artturin reviewed Aug 27, 2023

View reviewed changes

pkgs/development/interpreters/python/cpython/default.nix Outdated Show resolved Hide resolved

Artturin mentioned this pull request Aug 27, 2023

gcc: move dll to output #251828

Merged

12 tasks

brano543 mentioned this pull request Aug 30, 2023

win-dll-links: also copy dll from dependencies #252459

Merged

12 tasks

Artturin reviewed Sep 5, 2023

View reviewed changes

Artturin force-pushed the python_win branch from 14f3b85 to 84a0cac Compare September 5, 2023 16:48

ofborg bot added the 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild label Sep 5, 2023

Artturin force-pushed the python_win branch from 84a0cac to e5c1a03 Compare September 6, 2023 21:03

Artturin changed the base branch from staging to master September 6, 2023 21:06

Artturin force-pushed the python_win branch 3 times, most recently from 6a4ef75 to 6740ec4 Compare September 6, 2023 21:14

Artturin reviewed Sep 6, 2023

View reviewed changes

Artturin force-pushed the python_win branch from 6740ec4 to f3e6940 Compare September 6, 2023 22:26

Artturin reviewed Sep 6, 2023

View reviewed changes

Artturin force-pushed the python_win branch 2 times, most recently from acac81b to ff131bc Compare September 6, 2023 23:13

Artturin reviewed Sep 6, 2023

View reviewed changes

python311: enable cross-compilation to windows

a02761d

Artturin force-pushed the python_win branch from ff131bc to a02761d Compare September 6, 2023 23:16

Artturin approved these changes Sep 6, 2023

View reviewed changes

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Sep 7, 2023

Artturin merged commit 57095e2 into NixOS:master Sep 8, 2023
10 checks passed

Artturin mentioned this pull request Oct 31, 2024

python3: proper syntax for Windows patches #351010

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python311: enable windows cross-compilation, make win-dll-hook link to distant dependancies #227818

python311: enable windows cross-compilation, make win-dll-hook link to distant dependancies #227818

marius851000 commented Apr 23, 2023 •

edited

Loading

marius851000 commented Apr 24, 2023

marius851000 commented Apr 24, 2023

marius851000 commented Apr 24, 2023 •

edited

Loading

Artturin Aug 27, 2023

Artturin commented Aug 27, 2023 •

edited

Loading

Artturin Sep 5, 2023

Artturin commented Sep 5, 2023 •

edited

Loading

Ericson2314 commented Sep 5, 2023

Artturin commented Sep 6, 2023

Artturin Sep 6, 2023 •

edited

Loading

Artturin Sep 6, 2023 •

edited

Loading

Artturin Sep 6, 2023

Artturin Sep 6, 2023

Artturin Sep 6, 2023

Artturin left a comment

marius851000 commented Sep 8, 2023

python311: enable windows cross-compilation, make win-dll-hook link to distant dependancies #227818

python311: enable windows cross-compilation, make win-dll-hook link to distant dependancies #227818

Conversation

marius851000 commented Apr 23, 2023 • edited Loading

Description of changes

Things done

marius851000 commented Apr 24, 2023

marius851000 commented Apr 24, 2023

marius851000 commented Apr 24, 2023 • edited Loading

Artturin Aug 27, 2023

Choose a reason for hiding this comment

Artturin commented Aug 27, 2023 • edited Loading

Artturin Sep 5, 2023

Choose a reason for hiding this comment

Artturin commented Sep 5, 2023 • edited Loading

Ericson2314 commented Sep 5, 2023

Artturin commented Sep 6, 2023

Artturin Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

Artturin Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

Artturin Sep 6, 2023

Choose a reason for hiding this comment

Artturin Sep 6, 2023

Choose a reason for hiding this comment

Artturin Sep 6, 2023

Choose a reason for hiding this comment

Artturin left a comment

Choose a reason for hiding this comment

marius851000 commented Sep 8, 2023

marius851000 commented Apr 23, 2023 •

edited

Loading

marius851000 commented Apr 24, 2023 •

edited

Loading

Artturin commented Aug 27, 2023 •

edited

Loading

Artturin commented Sep 5, 2023 •

edited

Loading

Artturin Sep 6, 2023 •

edited

Loading

Artturin Sep 6, 2023 •

edited

Loading