Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python-package] fit() segfaults #6025

Closed
Realsumen opened this issue Aug 7, 2023 · 8 comments
Closed

[python-package] fit() segfaults #6025

Realsumen opened this issue Aug 7, 2023 · 8 comments
Labels

Comments

@Realsumen
Copy link

Realsumen commented Aug 7, 2023

Hello team,

I ran the code below and ran into a strange behavior, the fit method kills the Jupyter kernel.

import numpy as np
import pandas as pd
import lightgbm as lgb
from sklearn.preprocessing import OrdinalEncoder


train = pd.read_csv([forestfires.csv](https://github.com/microsoft/LightGBM/files/12285726/forestfires.csv))
features = train.columns.to_list()
label = 'area'
features.remove('area')
lgb.LGBMRegressor().fit(train[features], train[label])

LightGBM version:
lightgbm (4.0.0)

Command(s) used to install LightGBM

pip install lightgbm

Additional Comments

When I ran the the code in a .py file, this is the error message:

collecting ... collected 1 item

test.py::test Fatal Python error: Segmentation fault

Thread 0x00000001f964a500 (most recent call first):
File "/Users/sumen/anaconda3/lib/python3.10/site-packages/lightgbm/basic.py", line 1990 in __init_from_np2d
File "/Users/sumen/anaconda3/lib/python3.10/site-packages/lightgbm/basic.py", line 1856 in _lazy_init
File "/Users/sumen/anaconda3/lib/python3.10/site-packages/lightgbm/basic.py", line 2210 in construct
File "/Users/sumen/anaconda3/lib/python3.10/site-packages/lightgbm/basic.py", line 3096 in init
File

Could you please advise how would it be possible to tackle this point?

Thank you!

@jameslamb
Copy link
Collaborator

Thanks for using LightGBM and for the write-up with a reproducible example.

What operating system are you on? If on macOS, what version of OpenMP do you have installed? I ask in case you might be facing an issue like #4229.

Could you also check... do you have multiple copies of lightgbm installed? What does this command return?

find "${HOME}" -name 'lib_lightgbm.so'

@jameslamb jameslamb changed the title fit method kills the Jupyter kernel [python-package] fit() segfaults Aug 8, 2023
@jameslamb jameslamb added the bug label Aug 8, 2023
@Realsumen
Copy link
Author

Realsumen commented Aug 8, 2023

Hi,
I am using macOS.
brew install libomp Warning: libomp 16.0.6 is already installed and up-to-date.

find "${HOME}" -name 'lib_lightgbm.so'

/Users/sumen/anaconda3/pkgs/lightgbm-3.3.5-py310h313beb8_0/lib/python3.10/site-packages/lightgbm/lib_lightgbm.so
/Users/sumen/anaconda3/lib/python3.10/site-packages/lightgbm/lib/lib_lightgbm.so

@jameslamb
Copy link
Collaborator

jameslamb commented Aug 8, 2023

Aha! Having those two copies of lib_lightgbm definitely seems like a problem. Especially since you said in your initial report that you are using v4.0.0, that lightgbm-3.3.5-.../lib_lightgbm.so looks like it's the problem.

Also, since both of these are in paths like ${HOME}/anaconda3/lib, I suspect you're operating only in the base conda environment, not a dedicated one.

Try this:

  1. Uninstall lightgbm using BOTH conda and pip
conda uninstall --yes lightgbm
pip uninstall --yes lightgbm

2a. Run the find command again.

# this should now return 0 results
find "${HOME}/anaconda3" -name 'lib_lightgbm.so'

2b. If that find command is still matching lightgbm-3.3.5-py310h313beb8_0/lib/python3.10/site-packages/lightgbm/lib_lightgbm.so, manually delete that file with rm

/Users/sumen/anaconda3/pkgs/lightgbm-3.3.5-py310h313beb8_0/lib/python3.10/site-packages/lightgbm/lib_lightgbm.so
  1. Re-install lightgbm, using conda
conda install -c conda-forge --yes lightgbm
  1. restart your jupyter kernel, re-run the code

Please let me know how that goes.

@Realsumen
Copy link
Author

Realsumen commented Aug 8, 2023

Hi, I tried reinstalled and got a dependency error.

conda install -c conda-forge --yes lightgbm
Collecting package metadata (current_repodata.json): done
Solving environment: \ 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - defaults/noarch::conda-pack==0.6.0=pyhd3eb1b0_0
  - defaults/noarch::tifffile==2021.7.2=pyhd3eb1b0_2
  - defaults/osx-arm64::zope.interface==5.4.0=py310h1a28f6b_0
  - defaults/osx-arm64::astropy==5.1=py310h96f19d2_0
  - defaults/osx-arm64::patsy==0.5.3=py310hca03da5_0
  - defaults/osx-arm64::panel==0.14.3=py310hca03da5_0
  - defaults/osx-arm64::bottleneck==1.3.5=py310h96f19d2_0
  - defaults/osx-arm64::scipy==1.10.0=py310h20cbe94_1
  - defaults/osx-arm64::anaconda-project==0.11.1=py310hca03da5_0
  - defaults/osx-arm64::conda-repo-cli==1.0.41=py310hca03da5_0
  - defaults/osx-arm64::qtconsole==5.4.0=py310hca03da5_0
  - defaults/osx-arm64::python-lsp-black==1.2.1=py310hca03da5_0
  - defaults/osx-arm64::nbclassic==0.5.2=py310hca03da5_0
  - defaults/osx-arm64::datashader==0.14.4=py310hca03da5_0
  - defaults/osx-arm64::scikit-learn==1.2.1=py310h313beb8_0
  - defaults/osx-arm64::pywavelets==1.4.1=py310h80987f9_0
  - defaults/osx-arm64::xarray==2022.11.0=py310hca03da5_0
  - defaults/noarch::backports.functools_lru_cache==1.6.4=pyhd3eb1b0_0
  - defaults/osx-arm64::python-lsp-server==1.7.1=py310hca03da5_0
  - defaults/osx-arm64::pytables==3.7.0=py310ha5d4e50_1
  - defaults/osx-arm64::hvplot==0.8.2=py310hca03da5_0
  - defaults/osx-arm64::gensim==4.3.0=py310h46d7db6_0
  - defaults/noarch::pyls-spyder==0.4.0=pyhd3eb1b0_0
  - defaults/osx-arm64::statsmodels==0.13.5=py310hbda83bc_1
  - defaults/osx-arm64::bokeh==2.4.3=py310hca03da5_0
  - defaults/osx-arm64::jupyterlab==3.5.3=py310hca03da5_0
  - defaults/osx-arm64::contourpy==1.0.5=py310h525c30c_0
  - defaults/osx-arm64::anaconda-client==1.11.2=py310hca03da5_0
  - defaults/osx-arm64::imageio==2.26.0=py310hca03da5_0
  - defaults/osx-arm64::numpy==1.23.5=py310hb93e574_0
  - defaults/osx-arm64::pyerfa==2.0.0=py310h1a28f6b_0
  - defaults/osx-arm64::matplotlib==3.7.0=py310hca03da5_0
  - defaults/osx-arm64::h5py==3.7.0=py310h181c318_0
  - defaults/osx-arm64::dask==2022.7.0=py310hca03da5_0
  - defaults/osx-arm64::datashape==0.5.4=py310hca03da5_1
  - defaults/osx-arm64::ipykernel==6.19.2=py310h33ce5c2_0
  - defaults/osx-arm64::anaconda-navigator==2.4.2=py310hca03da5_0
  - defaults/osx-arm64::scrapy==2.8.0=py310hca03da5_0
  - defaults/noarch::conda-verify==3.4.2=py_1
  - defaults/osx-arm64::twisted==22.2.0=py310h1a28f6b_1
  - defaults/osx-arm64::imbalanced-learn==0.10.1=py310hca03da5_0
  - defaults/osx-arm64::numba==0.56.4=py310h46d7db6_0
  - defaults/osx-arm64::conda-build==3.24.0=py310hca03da5_0
  - defaults/osx-arm64::imagecodecs==2021.8.26=py310h48bc37f_2
  - defaults/osx-arm64::scikit-image==0.19.3=py310h313beb8_1
  - defaults/osx-arm64::bcrypt==3.2.0=py310h1a28f6b_1
  - defaults/osx-arm64::holoviews==1.15.4=py310hca03da5_0
  - defaults/osx-arm64::conda==23.3.1=py310hca03da5_0
  - defaults/osx-arm64::transformers==4.24.0=py310hca03da5_0
  - defaults/osx-arm64::spyder==5.4.1=py310hca03da5_0
  - defaults/osx-arm64::navigator-updater==0.3.0=py310hca03da5_0
  - defaults/osx-arm64::notebook==6.5.2=py310hca03da5_0
  - defaults/osx-arm64::intake==0.6.7=py310hca03da5_0
  - defaults/osx-arm64::numexpr==2.8.4=py310hecc3335_0
  - defaults/osx-arm64::distributed==2022.7.0=py310hca03da5_0
  - defaults/osx-arm64::seaborn==0.12.2=py310hca03da5_0
  - defaults/noarch::conda-token==0.4.0=pyhd3eb1b0_0
  - defaults/osx-arm64::pandas==1.5.3=py310h46d7db6_0
  - defaults/osx-arm64::clyent==1.2.2=py310hca03da5_1
  - defaults/osx-arm64::spyder-kernels==2.4.1=py310hca03da5_0
  - defaults/osx-arm64::matplotlib-base==3.7.0=py310h46d7db6_0
  - defaults/osx-arm64::pytorch==1.12.1=cpu_py310h8370978_1
failed with initial frozen solve. Retrying with flexible solve.
Solving environment: done
Collecting package metadata (repodata.json): done
Solving environment: \ 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - defaults/noarch::conda-pack==0.6.0=pyhd3eb1b0_0
  - defaults/noarch::tifffile==2021.7.2=pyhd3eb1b0_2
  - defaults/osx-arm64::zope.interface==5.4.0=py310h1a28f6b_0
  - defaults/osx-arm64::astropy==5.1=py310h96f19d2_0
  - defaults/osx-arm64::patsy==0.5.3=py310hca03da5_0
  - defaults/osx-arm64::panel==0.14.3=py310hca03da5_0
  - defaults/osx-arm64::bottleneck==1.3.5=py310h96f19d2_0
  - defaults/osx-arm64::scipy==1.10.0=py310h20cbe94_1
  - defaults/osx-arm64::anaconda-project==0.11.1=py310hca03da5_0
  - defaults/osx-arm64::conda-repo-cli==1.0.41=py310hca03da5_0
  - defaults/osx-arm64::qtconsole==5.4.0=py310hca03da5_0
  - defaults/osx-arm64::python-lsp-black==1.2.1=py310hca03da5_0
  - defaults/osx-arm64::nbclassic==0.5.2=py310hca03da5_0
  - defaults/osx-arm64::datashader==0.14.4=py310hca03da5_0
  - defaults/osx-arm64::scikit-learn==1.2.1=py310h313beb8_0
  - defaults/osx-arm64::pywavelets==1.4.1=py310h80987f9_0
  - defaults/osx-arm64::xarray==2022.11.0=py310hca03da5_0
  - defaults/noarch::backports.functools_lru_cache==1.6.4=pyhd3eb1b0_0
  - defaults/osx-arm64::python-lsp-server==1.7.1=py310hca03da5_0
  - defaults/osx-arm64::pytables==3.7.0=py310ha5d4e50_1
  - defaults/osx-arm64::hvplot==0.8.2=py310hca03da5_0
  - defaults/osx-arm64::gensim==4.3.0=py310h46d7db6_0
  - defaults/noarch::pyls-spyder==0.4.0=pyhd3eb1b0_0
  - defaults/osx-arm64::statsmodels==0.13.5=py310hbda83bc_1
  - defaults/osx-arm64::bokeh==2.4.3=py310hca03da5_0
  - defaults/osx-arm64::jupyterlab==3.5.3=py310hca03da5_0
  - defaults/osx-arm64::contourpy==1.0.5=py310h525c30c_0
  - defaults/osx-arm64::anaconda-client==1.11.2=py310hca03da5_0
  - defaults/osx-arm64::imageio==2.26.0=py310hca03da5_0
  - defaults/osx-arm64::numpy==1.23.5=py310hb93e574_0
  - defaults/osx-arm64::pyerfa==2.0.0=py310h1a28f6b_0
  - defaults/osx-arm64::matplotlib==3.7.0=py310hca03da5_0
  - defaults/osx-arm64::h5py==3.7.0=py310h181c318_0
  - defaults/osx-arm64::dask==2022.7.0=py310hca03da5_0
  - defaults/osx-arm64::datashape==0.5.4=py310hca03da5_1
  - defaults/osx-arm64::ipykernel==6.19.2=py310h33ce5c2_0
  - defaults/osx-arm64::anaconda-navigator==2.4.2=py310hca03da5_0
  - defaults/osx-arm64::scrapy==2.8.0=py310hca03da5_0
  - defaults/noarch::conda-verify==3.4.2=py_1
  - defaults/osx-arm64::twisted==22.2.0=py310h1a28f6b_1
  - defaults/osx-arm64::imbalanced-learn==0.10.1=py310hca03da5_0
  - defaults/osx-arm64::numba==0.56.4=py310h46d7db6_0
  - defaults/osx-arm64::conda-build==3.24.0=py310hca03da5_0
  - defaults/osx-arm64::imagecodecs==2021.8.26=py310h48bc37f_2
  - defaults/osx-arm64::scikit-image==0.19.3=py310h313beb8_1
  - defaults/osx-arm64::bcrypt==3.2.0=py310h1a28f6b_1
  - defaults/osx-arm64::holoviews==1.15.4=py310hca03da5_0
  - defaults/osx-arm64::conda==23.3.1=py310hca03da5_0
  - defaults/osx-arm64::transformers==4.24.0=py310hca03da5_0
  - defaults/osx-arm64::spyder==5.4.1=py310hca03da5_0
  - defaults/osx-arm64::navigator-updater==0.3.0=py310hca03da5_0
  - defaults/osx-arm64::notebook==6.5.2=py310hca03da5_0
  - defaults/osx-arm64::intake==0.6.7=py310hca03da5_0
  - defaults/osx-arm64::numexpr==2.8.4=py310hecc3335_0
  - defaults/osx-arm64::distributed==2022.7.0=py310hca03da5_0
  - defaults/osx-arm64::seaborn==0.12.2=py310hca03da5_0
  - defaults/noarch::conda-token==0.4.0=pyhd3eb1b0_0
  - defaults/osx-arm64::pandas==1.5.3=py310h46d7db6_0
  - defaults/osx-arm64::clyent==1.2.2=py310hca03da5_1
  - defaults/osx-arm64::spyder-kernels==2.4.1=py310hca03da5_0
  - defaults/osx-arm64::matplotlib-base==3.7.0=py310h46d7db6_0
  - defaults/osx-arm64::pytorch==1.12.1=cpu_py310h8370978_1
done


==> WARNING: A newer version of conda exists. <==
  current version: 23.3.1
  latest version: 23.7.2

Please update conda by running

conda update -n base -c defaults conda

Or to minimize the number of packages updated during conda update use

conda install conda=23.7.2



Package Plan ##

  environment location: /Users/sumen/anaconda3

  added / updated specs:
    - lightgbm


The following NEW packages will be INSTALLED:

  comm               conda-forge/noarch::comm-0.1.4-pyhd8ed1ab_0 
  lightgbm           pkgs/main/osx-arm64::lightgbm-3.3.5-py310h313beb8_0 
  numpy-base         pkgs/main/osx-arm64::numpy-base-1.23.5-py310haf87e8b_0 
  pip                conda-forge/noarch::pip-23.2.1-pyhd8ed1ab_0 
  setuptools         conda-forge/noarch::setuptools-68.0.0-pyhd8ed1ab_0 
  wheel              conda-forge/noarch::wheel-0.41.1-pyhd8ed1ab_0 

The following packages will be UPDATED:

  ca-certificates    pkgs/main::ca-certificates-2023.05.30~ --> conda-forge::ca-certificates-2023.7.22-hf0a4a13_0 
  certifi            pkgs/main/osx-arm64::certifi-2023.5.7~ --> conda-forge/noarch::certifi-2023.7.22-pyhd8ed1ab_0 
  openssl              pkgs/main::openssl-1.1.1u-h1a28f6b_0 --> conda-forge::openssl-1.1.1v-h53f4e23_0 



Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: failed

RemoveError: 'setuptools' is a dependency of conda and cannot be removed from
conda's operating environment.

@jameslamb
Copy link
Collaborator

I've reformatted your post, to make the diference between your own words, commands you ran, and logs from those commands clearer. If you're not familiar with how to do that, please read https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax.


Based on your most recent message, it seems to me that your conda environment is broken. I suspect that that came from using a mix of conda install and lightgbm, or maybe from mixing packages from different conda channels.

That is not a lightgbm-specific problem... I suspect if you ran something like conda install -c conda-forge --yes xgboost, you'd see the same errors.

The risk of this happening is why I personally try to avoid using the base conda environment, and always create dedicated environments for different projects. Try the following.

  1. Completely uninstall conda and all packages you've installed with it
rm -r /Users/sumen/anaconda3
  1. Re-install conda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/macos.html
  2. create a new environment for use with lightgbm + jupyterlab
conda create \
    --name ml-dev \
   -c conda-forge \
   --yes \
       ipykernel \
       jupyterlab \
       'lightgbm>=4.0.0' \
       numpy \
       'python=3.11' \
       scikit-learn \
       scipy
  1. Active that conda environment and start JupyterLab
source activate ml-dev
jupyter lab
  1. In JupyterLab, run lightgbm
import lightgbm as lgb
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=1_000)
dtrain = lgb.Dataset(X, label=y)
bst = lgb.train(
    train_set=dtrain,
    params={
        "objective": "regression",
        "num_iterations": 10
    }
)
image

@Realsumen
Copy link
Author

Hello,
I followed your advice to set up a new environment and it works. Your guidance has been very helpful, and I appreciate for your assistance in this issue.

@jameslamb
Copy link
Collaborator

Great, glad it helped! Thanks for using LightGBM, come back any time 👋🏻

Copy link

github-actions bot commented Nov 8, 2023

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants