Please document dependency version requirements.

[molpopgen]

This package is currently rather difficult to get working due to tf/keras issues.

Installing current versions leads to import errors when training:

python ../../submodules/diploSHIC/diploSHIC.py train train test model
2021-06-08 14:43:18.605122: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-06-08 14:43:18.605144: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "../../submodules/diploSHIC/diploSHIC.py", line 126, in <module>
    from keras.utils.layer_utils import convert_all_kernels_in_model
ImportError: cannot import name 'convert_all_kernels_in_model' from 'keras.utils.layer_utils' (/home/molpopgen/src/polygenic_classification/prototype/single_core_training/training/lib/python3.8/site-packages/keras/utils/layer_utils.py)

This is for the following versions in a clean venv:

pip show keras
Name: Keras
Version: 2.4.3
Summary: Deep Learning for humans
Home-page: https://github.com/keras-team/keras
Author: Francois Chollet
Author-email: francois.chollet@gmail.com
License: MIT
Location: /home/molpopgen/src/polygenic_classification/prototype/single_core_training/training/lib/python3.8/site-packages
Requires: numpy, pyyaml, h5py, scipy
Required-by: 
(training)

pip show tensorflow
Name: tensorflow
Version: 2.5.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /home/molpopgen/src/polygenic_classification/prototype/single_core_training/training/lib/python3.8/site-packages
Requires: h5py, tensorflow-estimator, typing-extensions, flatbuffers, astunparse, protobuf, keras-nightly, google-pasta, six, tensorboard, wheel, keras-preprocessing, wrapt, gast, absl-py, grpcio, termcolor, numpy, opt-einsum
Required-by: 

Google searches suggest that the import error is fixed by installing earlier versions of tf. However, pip seems to reject that now:

pip --version
pip 20.1.1 from /home/molpopgen/src/polygenic_classification/prototype/single_core_training/training/lib/python3.8/site-packages/pip (python 3.8)

pip install tensorflow==1.15.2
ERROR: Could not find a version that satisfies the requirement tensorflow==1.15.2 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.2.2, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.3.2, 2.4.0rc0, 2.4.0rc1, 2.4.0rc2, 2.4.0rc3, 2.4.0rc4, 2.4.0, 2.4.1, 2.5.0rc0, 2.5.0rc1, 2.5.0rc2, 2.5.0rc3, 2.5.0)
ERROR: No matching distribution found for tensorflow==1.15.2

pip install tensorflow-cpu==1.15.2
ERROR: Could not find a version that satisfies the requirement tensorflow-cpu==1.15.2 (from versions: 2.2.0rc1, 2.2.0rc2, 2.2.0rc3, 2.2.0rc4, 2.2.0, 2.2.1, 2.2.2, 2.3.0rc0, 2.3.0rc1, 2.3.0rc2, 2.3.0, 2.3.1, 2.3.2, 2.4.0rc0, 2.4.0rc1, 2.4.0rc2, 2.4.0rc3, 2.4.0rc4, 2.4.0, 2.4.1, 2.5.0rc0, 2.5.0rc1, 2.5.0rc2, 2.5.0rc3, 2.5.0)
ERROR: No matching distribution found for tensorflow-cpu==1.15.2

The solution to this is to provide a requirements.txt that you have verified to work.

[jdaron]

I have been ruining into the same problem, and I highly recommend to create a virtual environment to run some of the sub program of diploSHIC.

Here is the workflow I've done:

1. create a new ven with python <= 3.7 since the version of keras and tensorflow compatible with diploSHIC do not work with higher python version
   conda create -n ven python=3.6 anaconda
   conda activate ven

2. install the appropriate version of tensorflow and keras
   pip install tensorflow==1.15.2
   pip install keras==2.4.3

3. you're all set to run the diploSHIC.py train and diploSHIC.py predict tools. Note that I only activate the ven to run those two tools and I run the rest of diploSHIC using my normal environment:
   Python 3.8.8
   numpy 1.21.4
   scipy 1.6.2
   pandas 1.2.4
   scikit-allel 1.3.2
   scikit-learn 0.24.1

[molpopgen]

It is possible for it all to work in a single venv, but it was non-trivial to work out. The following works on Python 3.8 and I believe also 3.9:

scikit-allel
# scipy
scipy==1.4.1
matplotlib
keras==2.4.3
tensorflow-cpu<=2.3
sklearn

I've used this on several machines with Pop OS 20.04 and 21.04, which are basically the same as the Ubuntu's with the same version numbers.

[andrewkern]

hey @molpopgen and @jdaron -- i'm currently working on #39 which cleans up these issues and will provide a clean pip install. i'll close this issue for now and move conversation to that PR if it suits you all ok

[andrewkern]

reopening this issue as @molpopgen points out #39 doesn't provide a sufficient fix to the issues here

[andrewkern]

i'd encourage you guys to check out the new pip package and confirm that it works on your systems. it should be able to be installed on a clean virtual environment simply with pip install diploshic.

so far i've been able to confirm that this install works on a mac systems and linux

[jdaron]

Dear Andrew,
I've been testing the new version of diploSHIC, which I installed using pip install diploshic. However I am still getting an error while launching diploSHIC fvecSim diploid (see bellow). I generated this error with numpy==1.19.5 while with numpy==1.21.4 (pip install -U numpy) I don't get this error and the program is running smoothly.
Do you think it could be a problem with my conda environement ?

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
File "/home/daron/anaconda3/bin/makeFeatureVecsForSingleMsDiploid.py", line 5, in
from diploshic.msTools import *
File "/home/daron/anaconda3/lib/python3.8/site-packages/diploshic/init.py", line 1, in
from diploshic.fvTools import *
File "/home/daron/anaconda3/lib/python3.8/site-packages/diploshic/fvTools.py", line 7, in
import diploshic.shicstats as dps
ImportError: numpy.core.multiarray failed to import

[molpopgen]

). I generated this error with numpy==1.19.5 while with numpy==1.21.4 (pip install -U numpy) I don't get this error and the program is running smoothly.

Aha-- this is important. The error is due to something being compiled against the numpy C API. This API is fickle. @andrewkern -- you probably have to pin the package to the exact numpy versions used, else the wheels will be very fragile.

[andrewkern]

Dear Andrew, I've been testing the new version of diploSHIC, which I installed using pip install diploshic. However I am still getting an error while launching diploSHIC fvecSim diploid (see bellow). I generated this error with numpy==1.19.5 while with numpy==1.21.4 (pip install -U numpy) I don't get this error and the program is running smoothly. Do you think it could be a problem with my conda environement ?

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd Traceback (most recent call last): File "/home/daron/anaconda3/bin/makeFeatureVecsForSingleMsDiploid.py", line 5, in from diploshic.msTools import * File "/home/daron/anaconda3/lib/python3.8/site-packages/diploshic/init.py", line 1, in from diploshic.fvTools import * File "/home/daron/anaconda3/lib/python3.8/site-packages/diploshic/fvTools.py", line 7, in import diploshic.shicstats as dps ImportError: numpy.core.multiarray failed to import

hi @jdaron -- yes I think this is a conda issue. in likelihood you have conflicting package versions in that old environment. can you try this again in a clean (brand new) conda install? i.e.,

conda create -n test_diploshic python=3.9 --yes
conda activate test_diploshic
pip install diploshic

[jdaron]

I've been running the whole diploshic workflow in a virtual env and everything run smoothly, thanks a lot for making those updates.

[andrewkern]

awesome-- great to hear.

@molpopgen is this working on your systems too now?

[molpopgen]

awesome-- great to hear.

@molpopgen is this working on your systems too now?

Haven't had time to test. Will try to in a few days.

[andrewkern]

hey @molpopgen -- should we close this issue?

[molpopgen]

No -- setup.py doesn't address this issues. There's no version numbers pinned/ranges pinned for tensorflow/keras.

Also, the version numbers for this package make it difficult/impossible to realistically used. Adding a 3 to 0..3333 when bumping is still the same semver, so a user has a hard time pinning to anything other than exact versions.