Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression: CondaMemoryError when building #5267

Closed
Tracked by #5319
mxschmitt opened this issue Apr 2, 2024 · 7 comments · Fixed by #5348
Closed
Tracked by #5319

Regression: CondaMemoryError when building #5267

mxschmitt opened this issue Apr 2, 2024 · 7 comments · Fixed by #5348
Labels
in-progress issue is actively being worked on locked [bot] locked due to inactivity severity::3 major; broken functionality with a workaround source::community catch-all for issues filed by community members type::bug describes erroneous operation, use severity::* to classify the type
Milestone
24.5.x

Comments

@mxschmitt
Copy link

mxschmitt commented Apr 2, 2024 •
edited by kenodegard
Loading

What happened?

Looks like this broke in #5228.

Reproduction steps:

git clone https://github.com/microsoft/playwright-python
conda config --add channels conda-forge
conda install anaconda-client conda-build=24.3.0 conda-verify py-lief=0.14.1 # These are the latest ones as of today.
conda install cross-python_linux-aarch64
conda build . -m conda_build_config.yaml -m conda_build_config_linux_aarch64.yaml

Expected: Works
Actual: CondaMemoryError: The conda process ran out of memory. Increase system memory and/or try again.

This machine has 64GB of free memory and works with:

conda install anaconda-client conda-build=24.1.2 conda-verify py-lief=0.12.3

so I'm fairly confident, that this is a regression.

cc @mbargul

Conda Details

conda info 
active environment : None
       user config file : /home/codespace/.condarc
 populated config files : /home/codespace/.condarc
          conda version : 24.3.0
    conda-build version : 24.3.0
         python version : 3.12.2.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen2
                          __conda=24.3.0=0
                          __glibc=2.31=0
                          __linux=6.2.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/codespace/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/codespace/.conda/envs
               platform : linux-64
             user-agent : conda/24.3.0 requests/2.31.0 CPython/3.12.2 Linux/6.2.0-1019-azure ubuntu/20.04.6 glibc/2.31 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.3 aau/0.4.4 c/. s/. e/.
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
conda config 
==> /home/codespace/.condarc <==
channels:
  - conda-forge
  - defaults
conda list 
+# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-aarch64_curr_repodata_hack 4                   h57d6b7b_14    conda-forge
anaconda-anon-usage       0.4.4           py312hfc0e8ea_100    defaults
anaconda-client           1.12.3             pyhd8ed1ab_0    conda-forge
anaconda-project          0.11.1             pyhd8ed1ab_0    conda-forge
archspec                  0.2.3              pyhd3eb1b0_0    defaults
attrs                     23.1.0          py312h06a4308_0    defaults
backports                 1.1                pyhd3eb1b0_0    defaults
backports.functools_lru_cache 1.6.4              pyhd3eb1b0_0    defaults
backports.tempfile        1.0                pyhd3eb1b0_1    defaults
backports.weakref         1.0.post1                  py_1    defaults
beautifulsoup4            4.12.2          py312h06a4308_0    defaults
binutils_impl_linux-64    2.38                 h2a08ee3_1    defaults
binutils_impl_linux-aarch64 2.36.1               he24a65f_2    conda-forge
binutils_linux-aarch64    2.36                h38357a0_10    conda-forge
boltons                   23.0.0          py312h06a4308_0    defaults
brotli-python             1.0.9           py312h6a678d5_7    defaults
bzip2                     1.0.8                h5eee18b_5    defaults
c-ares                    1.19.1               h5eee18b_0    defaults
ca-certificates           2024.3.11            h06a4308_0    defaults
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py312h5eee18b_0    defaults
chardet                   4.0.0           py312h06a4308_1003    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.1.7           py312h06a4308_0    defaults
conda                     24.3.0          py312h7900ff3_0    conda-forge
conda-build               24.3.0          py312h7900ff3_1    conda-forge
conda-index               0.4.0              pyhd3eb1b0_0    defaults
conda-libmamba-solver     23.12.0            pyhd3eb1b0_1    defaults
conda-pack                0.7.1              pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0           py312h06a4308_0    defaults
conda-package-streaming   0.9.0           py312h06a4308_0    defaults
conda-verify              3.4.2                      py_1    defaults
cross-python_linux-aarch64 3.12                 41_cpython    conda-forge
crossenv                  1.4.0              pyhd8ed1ab_0    conda-forge
cryptography              41.0.4                   pypi_0    pypi
defusedxml                0.7.1              pyhd3eb1b0_0    defaults
distro                    1.8.0           py312h06a4308_0    defaults
expat                     2.5.0                h6a678d5_0    defaults
filelock                  3.13.1          py312h06a4308_0    defaults
fmt                       9.1.0                hdb19cb5_0    defaults
freetype                  2.12.1               h267a509_2    conda-forge
future                    0.18.3          py312h06a4308_0    defaults
gcc_impl_linux-64         12.1.0              hea43390_17    conda-forge
gcc_impl_linux-aarch64    12.1.0              h053c1db_17    conda-forge
gcc_linux-aarch64         12.1.0              h93c90ae_10    conda-forge
icu                       73.1                 h6a678d5_0    defaults
idna                      3.4             py312h06a4308_0    defaults
jinja2                    3.1.3           py312h06a4308_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
jsonschema                4.19.2          py312h06a4308_0    defaults
jsonschema-specifications 2023.7.1        py312h06a4308_0    defaults
jupyter_core              5.5.0           py312h06a4308_0    defaults
kernel-headers_linux-64   2.6.32              he073ed8_17    conda-forge
kernel-headers_linux-aarch64 4.18.0              h5b4a56d_14    conda-forge
krb5                      1.20.1               h143b758_1    defaults
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.38                 h1181459_1    defaults
ld_impl_linux-aarch64     2.36.1               h4e2ea54_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libarchive                3.6.2                h6ac8c49_2    defaults
libcurl                   8.5.0                h251f7ec_0    defaults
libdeflate                1.20                 hd590300_0    conda-forge
libedit                   3.1.20230828         h5eee18b_0    defaults
libev                     4.33                 h7f8727e_1    defaults
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-devel_linux-64     12.1.0              h1ec3361_17    conda-forge
libgcc-devel_linux-aarch64 12.1.0              hb67fff7_17    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblief                   0.14.1               hac33072_1    conda-forge
libmamba                  1.5.3                haf1ee3a_0    defaults
libmambapy                1.5.3           py312h2dafd23_0    defaults
libnghttp2                1.57.0               h2d74bed_0    defaults
libnsl                    2.0.1                hd590300_0    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libsanitizer              12.1.0              ha89aaad_17    conda-forge
libsolv                   0.7.24               he621ea3_0    defaults
libsqlite                 3.45.2               h2797004_0    conda-forge
libssh2                   1.10.0               hdbd6064_2    defaults
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libtiff                   4.6.0                h1dd3fc0_3    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.10.4               hf1b16e4_1    defaults
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                h6a678d5_0    defaults
markupsafe                2.1.3           py312h5eee18b_0    defaults
menuinst                  2.0.2           py312h06a4308_0    defaults
more-itertools            10.1.0          py312h06a4308_0    defaults
nbformat                  5.9.2           py312h06a4308_0    defaults
ncurses                   6.4                  h6a678d5_0    defaults
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openssl                   3.2.1                hd590300_1    conda-forge
packaging                 23.1            py312h06a4308_0    defaults
patch                     2.7.6             h7b6447c_1001    defaults
patchelf                  0.17.2               h6a678d5_0    defaults
pcre2                     10.42                hebb0a14_0    defaults
pillow                    10.2.0          py312hf3581a9_0    conda-forge
pip                       24.0                     pypi_0    pypi
pkginfo                   1.9.6           py312h06a4308_0    defaults
platformdirs              3.10.0          py312h06a4308_0    defaults
pluggy                    1.0.0           py312h06a4308_1    defaults
popt                      1.16              h0b475e3_2002    conda-forge
psutil                    5.9.0           py312h5eee18b_0    defaults
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
py-lief                   0.14.1          py312h7070661_1    conda-forge
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.6           py312h5eee18b_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 24.0.0          py312h06a4308_0    defaults
pysocks                   1.7.1           py312h06a4308_0    defaults
python                    3.12.2          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
python-fastjsonschema     2.16.2          py312h06a4308_0    defaults
python-libarchive-c       2.9                pyhd3eb1b0_1    defaults
python_abi                3.12                    4_cp312    conda-forge
pytz                      2023.3.post1    py312h06a4308_0    defaults
pyyaml                    6.0.1           py312h5eee18b_0    defaults
readline                  8.2                  h5eee18b_0    defaults
referencing               0.30.2          py312h06a4308_0    defaults
reproc                    14.2.4               h295c915_1    defaults
reproc-cpp                14.2.4               h295c915_1    defaults
requests                  2.31.0          py312h06a4308_1    defaults
requests-toolbelt         1.0.0           py312h06a4308_0    defaults
ripgrep                   14.1.0               he8a937b_0    conda-forge
rpds-py                   0.10.6          py312hb02cf49_0    defaults
rsync                     3.2.7                h70740c4_0    conda-forge
ruamel.yaml               0.17.21         py312h5eee18b_0    defaults
ruamel_yaml               0.15.80         py312h98912ed_1009    conda-forge
sed                       4.8                  he412f7d_0    conda-forge
setuptools                68.2.2          py312h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
soupsieve                 2.5             py312h06a4308_0    defaults
sqlite                    3.41.2               h5eee18b_0    defaults
sysroot_linux-64          2.12                he073ed8_17    conda-forge
sysroot_linux-aarch64     2.17                h5b4a56d_14    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tornado                   6.4             py312h98912ed_0    conda-forge
tqdm                      4.65.0          py312he106c6f_0    defaults
traitlets                 5.7.1           py312h06a4308_0    defaults
truststore                0.8.0           py312h06a4308_0    defaults
tzdata                    2024a                h04d1e81_0    defaults
urllib3                   1.26.18         py312h06a4308_0    defaults
wheel                     0.41.2          py312h06a4308_0    defaults
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xxhash                    0.8.0                h7f98852_3    conda-forge
xz                        5.4.6                h5eee18b_0    defaults
yaml                      0.2.5                h7b6447c_0    defaults
yaml-cpp                  0.8.0                h6a678d5_0    defaults
zlib                      1.2.13               hd590300_5    conda-forge
zstandard                 0.19.0          py312h5eee18b_0    defaults
zstd                      1.5.5                hc292b87_0    defaults
@mxschmitt mxschmitt added the type::bug describes erroneous operation, use severity::* to classify the type label Apr 2, 2024
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Apr 2, 2024
@kenodegard
Copy link
Contributor

@mxschmitt what about with conda-build=24.3.0 and py-lief<0.14?

@kenodegard
Copy link
Contributor

cc @mbargull

@kenodegard kenodegard added the source::community catch-all for issues filed by community members label Apr 2, 2024
@mxschmitt
Copy link
Author

mxschmitt commented Apr 3, 2024 •
edited
Loading

I can confirm that:

good: py-lief=0.12.3
bad: py-lief=0.14.1

Note: There is no 0.13 on conda.

Quite a large range: lief-project/LIEF@0.12.1...0.14.1

@mxschmitt
Copy link
Author

mxschmitt commented Apr 3, 2024 •
edited
Loading

I got a minimal repro:

curl -O https://nodejs.org/dist/v20.12.0/node-v20.12.0-linux-arm64.tar.xz
tar -xf node-v20.12.0-linux-arm64.tar.xz

python -c 'import lief; binary = lief.parse("node-v20.12.0-linux-arm64/bin/node"); print(str(binary))'

Expected: Some stringified value
Actual: MemoryError: std::bad_alloc

good: py-lief=0.12.3
bad: py-lief=0.14.1

Maybe we should re-think the str() in get_uniqueness_key which was added in #5228 - with the good py-lief version it ended up in a 33 MB long string which does not sound ideal.

Should I file it upstream that it now ends up in MemoryError: std::bad_alloc?

@mxschmitt mxschmitt mentioned this issue Apr 4, 2024
Closed
@mxschmitt
Copy link
Author

Update: lief-project/LIEF#1039 (comment)

@arkaprava08 arkaprava08 mentioned this issue May 1, 2024
Closed
5 tasks
@mbargull
Copy link
Member

mbargull commented May 8, 2024

Oh, my bad! The change from gh-5228 to use str(file) is simply wrong; we should just return the filename instead (this is what the previous binary.name did).
Thanks for the bug report!
I'll put a fix together in a bit.

@mbargull mbargull mentioned this issue May 21, 2024
Merged
3 tasks
@kenodegard kenodegard added severity::3 major; broken functionality with a workaround in-progress issue is actively being worked on labels May 21, 2024
@kenodegard kenodegard moved this from 🆕 New to 🏗️ In Progress in 🧭 Planning May 21, 2024
@kenodegard kenodegard added this to the 24.5.x milestone May 21, 2024
@kenodegard kenodegard mentioned this issue May 21, 2024
Closed
31 tasks
@kenodegard
Copy link
Contributor

Fixed in #5348, to be included in the upcoming patch release conda-build 24.5.1

@github-project-automation github-project-automation bot moved this from 🏗️ In Progress to 🏁 Done in 🧭 Planning May 22, 2024
@beeankha beeankha mentioned this issue May 24, 2024
Merged
@mxschmitt mxschmitt mentioned this issue Nov 7, 2024
Merged
@github-actions github-actions bot added the locked [bot] locked due to inactivity label Nov 19, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
in-progress issue is actively being worked on locked [bot] locked due to inactivity severity::3 major; broken functionality with a workaround source::community catch-all for issues filed by community members type::bug describes erroneous operation, use severity::* to classify the type
Projects
🧭 Planning
Archived in project
Milestone
24.5.x
Development

Successfully merging a pull request may close this issue.

Fix excessive memory use in inspect_linkages_lief mbargull/conda-build
3 participants
@kenodegard @mbargull @mxschmitt