python36Packages.scipy: segmentation fault when running tests #92458

Enteee · 2020-07-06T14:02:01Z

Describe the bug
On current nixos-unstable Scipy fails to build with a segmentation fault during testing:

../cluster/tests/test_vq.py::TestKMean::test_kmeans2_init <- ../../nix/store/d00fds078va595ifakggj6srmky01jq2-python3.6-scipy-1.4.1/lib/python3.6/site-packages/scipy/cluster/tests/test_vq.py PASSED [  0%]
../cluster/tests/test_vq.py::TestKMean::test_krandinit <- ../../nix/store/d00fds078va595ifakggj6srmky01jq2-python3.6-scipy-1.4.1/lib/python3.6/site-packages/scipy/cluster/tests/test_vq.py Fatal Python error: Segmentation fault

Current thread 0x00007ffff7898f40 (most recent call first):
  File "/nix/store/d00fds078va595ifakggj6srmk/nix/store/ds7r5c4z1vzrm0jw7ay712zs59p64hf7-stdenv-linux/setup: line 1302:  9640 Segmentation fault      (core dumped) /nix/store/acwrn1di4lgiz80qxam9vk28b2mkr2py-python3-3.6.10/bin/python3.6 -c 'import scipy; scipy.test("fast", verbose=10)'
builder for '/nix/store/2sa0mm8g3rb2l4x4j4y2i7qihb2lbdpy-python3.6-scipy-1.4.1.drv' failed with exit code 139
cannot build derivation '/nix/store/0mxrinnvr4f43h17wpcfpfisc3kbqywk-python3-3.6.10-env.drv': 1 dependencies couldn't be built
error: build of '/nix/store/0mxrinnvr4f43h17wpcfpfisc3kbqywk-python3-3.6.10-env.drv' failed

To Reproduce
Steps to reproduce the behavior:

save the following as shell.nix

{
  nixpkgs ?  import (builtins.fetchGit {
    url = "https://github.com/nixos/nixpkgs-channels/";

    #
    # Commit hash for nixos-unstable
    # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-unstable`
    name = "nixos-unstable-2020-07-06";
    ref = "refs/heads/nixos-unstable";
    rev = "dc80d7bc4a244120b3d766746c41c0d9c5f81dfa";

    #
    # Commit hash for nixos-20.03
    # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-20.03`
    #name = "nixos-20.03-2020-07-06";
    #ref = "refs/heads/nixos-20.03";
    #rev = "afa9ca61924f05aacfe495a7ad0fd84709d236cc";
  }) {}
}:

with nixpkgs;

let
  python = python36;
  python-with-base-packages = python.withPackages(ps: with ps; [
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #

    scipy
  ]);

in mkShell {

  buildInputs = [
    python-with-base-packages
    python3Packages.venvShellHook
  ];

  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
    venvShellHook
  '';
}

run nix-shell

Expected behavior

Should not segfault during test.

Additional context
SciPY seems to build just fine on current 20.03 release.

Notify maintainers
cc @FRidh , @matthewbauer

Metadata

system: "x86_64-linux"
host os: Linux 5.4.46, NixOS, 20.09pre232692.7db146538e4 (Nightingale)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.6
channels(root): "home-manager, nixos-20.09pre233323.dc80d7bc4a2, nixos-hardware, nixpkgs-20.09pre233075.6e139365424"
channels(ente): ""
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

The text was updated successfully, but these errors were encountered:

jcumming · 2020-09-10T14:35:23Z

Also fails with python38Packages.scipy

chekoopa · 2020-10-09T04:04:04Z

Particular stacktrace is:

../cluster/tests/test_vq.py::TestKMean::test_kmeans2_simple <- ../../nix/store/x1vz10qv9ryk4ixiizdyq4mk6awqq168-python3.8-scipy-1.5.2/lib/python3.8/site-packages/scipy/cluster/tests/test_vq.py PASSED [  0%]
../cluster/tests/test_vq.py::TestKMean::test_kmeans2_rank1 <- ../../nix/store/x1vz10qv9ryk4ixiizdyq4mk6awqq168-python3.8-scipy-1.5.2/lib/python3.8/site-packages/scipy/cluster/tests/test_vq.py PASSED [  0%]
../cluster/tests/test_vq.py::TestKMean::test_kmeans2_rank1_2 <- ../../nix/store/x1vz10qv9ryk4ixiizdyq4mk6awqq168-python3.8-scipy-1.5.2/lib/python3.8/site-packages/scipy/cluster/tests/test_vq.py PASSED [  0%]
../cluster/tests/test_vq.py::TestKMean::test_kmeans2_high_dim <- ../../nix/store/x1vz10qv9ryk4ixiizdyq4mk6awqq168-python3.8-scipy-1.5.2/lib/python3.8/site-packages/scipy/cluster/tests/test_vq.py PASSED [  0%]
../cluster/tests/test_vq.py::TestKMean::test_kmeans2_init <- ../../nix/store/x1vz10qv9ryk4ixiizdyq4mk6awqq168-python3.8-scipy-1.5.2/lib/python3.8/site-packages/scipy/cluster/tests/test_vq.py PASSED [  0%]
../cluster/tests/test_vq.py::TestKMean::test_krandinit <- ../../nix/store/x1vz10qv9ryk4ixiizdyq4mk6awqq168-python3.8-scipy-1.5.2/lib/python3.8/site-packages/scipy/cluster/tests/test_vq.py Fatal Python error: Fatal Python error: Segmentation fault

Thread 0x00007ffff7843f40 (most recent call first):
  File "/nix/store/x1vz10qv9ryk4ixiizdyq4mkSegmentation fault

/nix/store/333six1faw9bhccsx9qw5718k6b1wiq2-stdenv-linux/setup: line 1303: 10293 Segmentation fault      (core dumped) /nix/store/wm30fgxfnn26z7z24y1iw64phb3djrq0-python3-3.8.5/bin/python3.8 -c 'import scipy; scipy.test("fast", verbose=10)'
builder for '/nix/store/2xmrj9b9zf4c2wxgknrbyhaw7h58gsry-python3.8-scipy-1.5.2.drv' failed with exit code 139
cannot build derivation '/nix/store/mfqj9l730b1wjhi8yhixj03mhb71va6k-python3-3.8.5-env.drv': 1 dependencies couldn't be built

adrianparvino · 2020-10-19T20:42:47Z

I believe the fix is contained in:
OpenMathLib/OpenBLAS#2729
and upstream tracking issue is:
OpenMathLib/OpenBLAS#2728
and Debian issue:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966463

glittershark · 2020-10-26T17:30:57Z

I believe the fix is contained in:
xianyi/OpenBLAS#2729
and upstream tracking issue is:
xianyi/OpenBLAS#2728
and Debian issue:
bugs.debian.org/cgi-bin/bugreport.cgi?bug=966463

Opened #101780 to bump openblas to a version with that fix

glittershark · 2020-10-27T14:01:35Z

#101780 fixes this for me - @Enteee @jcumming @chekoopa can you confirm?

Enteee · 2020-11-01T13:26:32Z

{
  nixpkgs ?  import (builtins.fetchGit {
    url = "https://github.com/nixos/nixpkgs/";

    #
    # Commit hash for nixos-unstable
    # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-unstable`
    ref = "refs/heads/staging";
    #rev = "3b4cd4f4dacd7db4011e88d819a1ae5acc3f3aff";
    rev = "5219a3ade1939470f2965ad4d80c87f7fb33db3e";

    #
    # Commit hash for nixos-20.03
    # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-20.03`
    #name = "nixos-20.03-2020-07-06";
    #ref = "refs/heads/nixos-20.03";
    #rev = "afa9ca61924f05aacfe495a7ad0fd84709d236cc";
  }) {}
}:

with nixpkgs;

let
  python = python36;
  python-with-base-packages = python.withPackages(ps: with ps; [
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #

    scipy
  ]);

in mkShell {

  buildInputs = [
    python-with-base-packages
    python3Packages.venvShellHook
  ];

  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
    venvShellHook
  '';
}

builds for me

jcumming · 2020-11-11T00:30:28Z

Works, thanks!

nomeata · 2021-01-07T18:21:24Z

This still affects release-20.09. Is there a simple fix we can backports (either in nixpkgs, or here locally)?

TredwellGit · 2021-01-07T18:29:01Z

Backport openblas updates: https://github.com/NixOS/nixpkgs/commits/staging/pkgs/development/libraries/science/math/openblas/default.nix

nomeata · 2021-01-07T18:37:43Z

Thanks, I’ll try that. Would such a backport be eligible to go to release-20.09?

TredwellGit · 2021-01-07T18:47:52Z

I don't really do backports, but since it fixes something known to be broken I assume it would be fine.

TredwellGit · 2021-01-07T18:50:24Z

Pull request should target staging-20.09 not release-20.09.

fixing NixOS#92458 fetching the file from nixpkgs master at cc8db6e

Enteee added the 0.kind: bug Something is broken label Jul 6, 2020

veprbl added the 6.topic: python label Jul 6, 2020

FRidh changed the title ~~Builiding of SciPy fails with segmentation fault when running tests~~ python36Packages.scipy: segmentation fault when running tests Jul 8, 2020

glittershark mentioned this issue Oct 26, 2020

openblas: 0.3.10 -> 0.3.12 #101780

Merged

10 tasks

TredwellGit mentioned this issue Oct 30, 2020

openblas: 0.3.10 -> 0.3.12 #102126

Merged

3 tasks

Enteee closed this as completed Nov 1, 2020

nomeata added a commit to dfinity-lab/nixpkgs that referenced this issue Jan 7, 2021

openblas: 0.3.10 -> 0.3.13

f7fc01b

fixing NixOS#92458 fetching the file from nixpkgs master at cc8db6e

nomeata mentioned this issue Jan 7, 2021

[staging-20.09] openblas: 0.3.10 -> 0.3.12 #108709

Merged

pjones mentioned this issue Apr 5, 2021

Nix database directory '/nix/var/nix/db' is not writable: No such file or directory nix-community/nix-on-droid#109

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python36Packages.scipy: segmentation fault when running tests #92458

python36Packages.scipy: segmentation fault when running tests #92458

Enteee commented Jul 6, 2020 •

edited

Loading

jcumming commented Sep 10, 2020

chekoopa commented Oct 9, 2020

adrianparvino commented Oct 19, 2020 •

edited

Loading

glittershark commented Oct 26, 2020

glittershark commented Oct 27, 2020

Enteee commented Nov 1, 2020

jcumming commented Nov 11, 2020

nomeata commented Jan 7, 2021

TredwellGit commented Jan 7, 2021

nomeata commented Jan 7, 2021

TredwellGit commented Jan 7, 2021

TredwellGit commented Jan 7, 2021

python36Packages.scipy: segmentation fault when running tests #92458

python36Packages.scipy: segmentation fault when running tests #92458

Comments

Enteee commented Jul 6, 2020 • edited Loading

jcumming commented Sep 10, 2020

chekoopa commented Oct 9, 2020

adrianparvino commented Oct 19, 2020 • edited Loading

glittershark commented Oct 26, 2020

glittershark commented Oct 27, 2020

Enteee commented Nov 1, 2020

jcumming commented Nov 11, 2020

nomeata commented Jan 7, 2021

TredwellGit commented Jan 7, 2021

nomeata commented Jan 7, 2021

TredwellGit commented Jan 7, 2021

TredwellGit commented Jan 7, 2021

Enteee commented Jul 6, 2020 •

edited

Loading

adrianparvino commented Oct 19, 2020 •

edited

Loading