numpy.dtype size changed, may indicate binary incompatibility

[astrofrog]

I'm having issues with the latest scipy, pandas, and numpy versions from defaults:

/home/travis/miniconda/envs/test/lib/python2.7/site-packages/pandas/_libs/__init__.py:3: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88

See:

https://travis-ci.org/glue-viz/glue/jobs/288485426#L1789

The output of conda env export can be found here:

https://travis-ci.org/glue-viz/glue/jobs/288485426#L1034

Relevant lines:

- numpy=1.13.3=py27hbcc08e0_0
- pandas=0.20.3=py27h820b67f_2
- scipy=0.19.1=py27h1edc525_3

[lucabaldini]

I am seeing something that, surely enough, seems related to this using Anaconda3-5.0.0.1-Linux-x86_64 out of the box, where the seemingly simple script

import numpy
import unittest
from astropy.io import fits

class TestFiltering(unittest.TestCase):
        
    def test_hdu(self):
        """Test some basic conditions.
        """
        col = fits.Column('COLA', 'D', 'AU', array=numpy.ones(100))
        data = fits.FITS_rec.from_columns([col])
        hdu = fits.BinTableHDU(data)


if __name__ == "__main__":
    unittest.main()

outputs

/data/install/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/data/install/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/data/install/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/data/install/anaconda3/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)

The weird thing is that outside a test case, e.g.,

import numpy
import unittest
from astropy.io import fits

col = fits.Column('COLA', 'D', 'AU', array=numpy.ones(100))
data = fits.FITS_rec.from_columns([col])
hdu = fits.BinTableHDU(data)

everything seems to work fine.

[jjhelmus]

The numpy.dtype size change warnings are coming from Cython and can be safely ignored, see numpy/numpy#432. Stack overflow has a good question of the topic.

[jjhelmus]

@lucabaldini I believe warnings have different visibility by default inside a test case than outside of one. Explicitly setting the warning visibility with warning.filterwarnings should give you more uniform result. For example

import warnings
warnings.filterwarnings("always")

will show all warnings. Do note that warnings are not the same as errors, many packages use them to reports unusual states that are benign but may be of interest to certain developers.

[msarahan]

Thanks for figuring this out @jjhelmus - closing.

[astrofrog]

@jjhelmus - just to understand, why has this suddenly started being an issue though? (in the past week or so I've seen these warnings pop up everywhere). That Numpy PR is ancient.

[mingwandroid]

Instead of building against a complete matrix of numpy versions we build against a single one now. This is to reduce the number of packages we need to build and carry.

[msarahan]

Just as a side note, this is consistent with practices initiated by @jjhelmus at conda-forge. If you build against an older numpy version, it is forwards-compatible with newer numpy versions. The inverse is not true.

[lucabaldini]

@jjhelmus - thanks a lot for taking the time to explain the matter to me.

For future reference I'll also add that the filterwarnings things needs to be done in the test body, rather than at the beginning of the module as I naively expected. It took me a few minutes of googling to figure that out:
http://www.neuraldump.com/2017/06/how-to-suppress-python-unittest-warnings/

[hua-henan]

I encountered this problem in getdist, finally found this due to numpy version incompatibility, retrieve numpy 1.15.0 to 1.13.3， problems solved

[zhiqwang]

The pandas were build agains different version of numpy. we need to rebuild pandas agains the local numpy.
pip install --no-binary pandas -I pandas

Reference

[msarahan]

Please stop. Please read @jjhelmus comment above at #6678 (comment). This warning (not an error) is safe to ignore. Locking this issue to prevent any future fumbling.

[msarahan]

My apologies. There is an actual new error that is not the same as the original error here. We are investigating and will report progress soon.

[msarahan]

As long as the message is a warning, this is safe to ignore. If this message is an error, you need to check if the numpy version you're using newer than the one that your package was built with. If pandas is built against numpy 1.11, you can use pandas with any newer version of numpy. Much of Anaconda is built against numpy 1.9, but some newer stuff is built against 1.11. 1.11 is a safe bet.

If you use pip to install anything, all bets are off. By doing so, we no longer have any idea what numpy version your pip-installed package has. Heaven help you.

[turiya]

I am trying:
pip install --no-binary pandas -I pandas
Looking in indexes: http://mirrors.aliyun.com/pypi/simple/
Collecting pandas
Downloading http://mirrors.aliyun.com/pypi/packages/e9/ad/5e92ba493eff96055a23b0a1323a9a803af71ec859ae3243ced86fcbd0a4/pandas-0.23.4.tar.gz (10.5MB)
100% |████████████████████████████████| 10.5MB 48.1MB/s
Collecting python-dateutil>=2.5.0 (from pandas)
Downloading http://mirrors.aliyun.com/pypi/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB)
100% |████████████████████████████████| 215kB 49.9MB/s
Collecting pytz>=2011k (from pandas)
Downloading http://mirrors.aliyun.com/pypi/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl (510kB)
100% |████████████████████████████████| 512kB 46.0MB/s
Collecting numpy>=1.9.0 (from pandas)
Downloading http://mirrors.aliyun.com/pypi/packages/88/29/f4c845648ed23264e986cdc5fbab5f8eace1be5e62144ef69ccc7189461d/numpy-1.15.0-cp36-cp36m-manylinux1_x86_64.whl (13.9MB)
100% |████████████████████████████████| 13.9MB 46.9MB/s
Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas)
Downloading http://mirrors.aliyun.com/pypi/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Skipping bdist_wheel for pandas, due to binaries being disabled for it.
piston-cli 0.5.1 has requirement colorama==0.3.6, but you'll have colorama 0.3.9 which is incompatible.
btsbots 0.1.6 has requirement graphenelib==0.4.8, but you'll have graphenelib 0.6.3 which is incompatible.
btsbots 0.1.6 has requirement scrypt==0.7.1, but you'll have scrypt 0.8.6 which is incompatible.
bts 0.6.17 has requirement graphenelib==0.4.8, but you'll have graphenelib 0.6.3 which is incompatible.
bts 0.6.17 has requirement requests==2.10.0, but you'll have requests 2.18.4 which is incompatible.
bts 0.6.17 has requirement scrypt==0.7.1, but you'll have scrypt 0.8.6 which is incompatible.
Installing collected packages: six, python-dateutil, pytz, numpy, pandas
Running setup.py install for pandas ... -

[mingwandroid]

You should issue conda install pandas

…

On Wed, Aug 22, 2018, 11:14 AM turiya ***@***.***> wrote: I am trying: pip install --no-binary pandas -I pandas Looking in indexes: http://mirrors.aliyun.com/pypi/simple/ Collecting pandas Downloading http://mirrors.aliyun.com/pypi/packages/e9/ad/5e92ba493eff96055a23b0a1323a9a803af71ec859ae3243ced86fcbd0a4/pandas-0.23.4.tar.gz (10.5MB) 100% |████████████████████████████████| 10.5MB 48.1MB/s Collecting python-dateutil>=2.5.0 (from pandas) Downloading http://mirrors.aliyun.com/pypi/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB) 100% |████████████████████████████████| 215kB 49.9MB/s Collecting pytz>=2011k (from pandas) Downloading http://mirrors.aliyun.com/pypi/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl (510kB) 100% |████████████████████████████████| 512kB 46.0MB/s Collecting numpy>=1.9.0 (from pandas) Downloading http://mirrors.aliyun.com/pypi/packages/88/29/f4c845648ed23264e986cdc5fbab5f8eace1be5e62144ef69ccc7189461d/numpy-1.15.0-cp36-cp36m-manylinux1_x86_64.whl (13.9MB) 100% |████████████████████████████████| 13.9MB 46.9MB/s Collecting six>=1.5 (from python-dateutil>=2.5.0->pandas) Downloading http://mirrors.aliyun.com/pypi/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl Skipping bdist_wheel for pandas, due to binaries being disabled for it. piston-cli 0.5.1 has requirement colorama==0.3.6, but you'll have colorama 0.3.9 which is incompatible. btsbots 0.1.6 has requirement graphenelib==0.4.8, but you'll have graphenelib 0.6.3 which is incompatible. btsbots 0.1.6 has requirement scrypt==0.7.1, but you'll have scrypt 0.8.6 which is incompatible. bts 0.6.17 has requirement graphenelib==0.4.8, but you'll have graphenelib 0.6.3 which is incompatible. bts 0.6.17 has requirement requests==2.10.0, but you'll have requests 2.18.4 which is incompatible. bts 0.6.17 has requirement scrypt==0.7.1, but you'll have scrypt 0.8.6 which is incompatible. Installing collected packages: six, python-dateutil, pytz, numpy, pandas Running setup.py install for pandas ... - — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#6678 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_pdGjSJqUkECD884T6QMC1bpVT7POiks5uTS8fgaJpZM4P6asM> .

[msarahan]

If you use pip to install anything, all bets are off. By doing so, we no longer have any idea what numpy version your pip-installed package has. Heaven help you.

[alexraju91]

I updated pandas and numpy and its gone.