TST: the pandas test suite fails on 32bit systems

[topper-123]

I've been trying tonight to get the test suite to pass and keep getting some failures:

Results (25.73s):
     830 passed
       6 failed
         - pandas\tests\indexes/test_base.py:1370 TestIndex.test_get_indexer_with_NA_values[None-nan]
         - pandas\tests\indexes/test_base.py:1370 TestIndex.test_get_indexer_with_NA_values[None-unique_nulls_fixture22]
         - pandas\tests\indexes/test_base.py:1370 TestIndex.test_get_indexer_with_NA_values[nan-None]
         - pandas\tests\indexes/test_base.py:1370 TestIndex.test_get_indexer_with_NA_values[nan-unique_nulls_fixture22]
         - pandas\tests\indexes/test_base.py:1370 TestIndex.test_get_indexer_with_NA_values[unique_nulls_fixture2-None]
         - pandas\tests\indexes/test_base.py:1370 TestIndex.test_get_indexer_with_NA_values[unique_nulls_fixture2-nan]
       2 xfailed
      63 skipped

The issue is related to integer dtypes:

E       AssertionError: numpy array are different
E
E       Attribute "dtype" are different
E       [left]:  int32
E       [right]: int64

Can anyone confirm if this is a failure on pandas on windows?

[gfyoung]

@topper-123 :

• I cannot replicate the test failures (Windows 10, 64-bit)
• We test Windows on the Azure pipelines, so looks like it could be some local issues

[jbrockmendel]

Any chance you’re running 32 bit python?

[gfyoung]

64-bit actually. Is this 32-bit related?

[jschendel]

I think @topper-123 uses 32-bit (see the discussion in #21905).

It looks like the same tests are also failing on some of the nightlies, e.g. https://travis-ci.org/MacPython/pandas-wheels/jobs/432034742#L4858

[gfyoung]

Makes sense. Looks like we should patch if we can ASAP.

[topper-123]

Yes, I run 32bit python, 64bit Windows.

I use miniconda as my python package mansager, so didn't choose 32bit myself. So likely this issue would affect all conda users on Windows?

[gfyoung]

So likely this issue would affect all conda users on Windows?

IMO That's a little bit of an exaggeration. I have 64-bit Python 😉

[topper-123]

I've looked more into this and have found the culprit to be #22296.

pandas/pandas/tests/indexes/test_base.py

Line 1382 in f4fae35

expected = np.array([0, 1, -1], dtype=np.int64)

I.e. in line 1382 in test_base.py, the dtype should be np.intp rather than np.int64.

Do people agree?

Returning platform dtype is the current behaviour for Index.get_indexer, e.g. (on pandas 0.23.1 on 32bit python on Windows 10):

In [1]: index = pd.Index(['a', 'b'], dtype=np.object)
In [2]: index.get_indexer(['a', 'b', 'c'])
Out[2]: array([ 0,  1, -1], dtype=int32)

[h-vetinari]

Yes, I run 32bit python, 64bit Windows.
I use miniconda as my python package manager, so didn't choose 32bit myself. So likely this issue would affect all conda users on Windows?

Same here - it seems to be the miniconda default (or at least it must have used to be), and as such will surely affect many (64bit-)Windows users.

After running the whole test suite, I have found two further persistent failures on current master (5ce06b5), one possibly related to #21333/#22549 (@jbrockmendel),

(pandas-dev) C:\Users\[...]\eclipse-workspace\pddev>pytest pandas/tests/indexes/datetimes/test_astype.py
============================= test session starts =============================
platform win32 -- Python 3.6.6, pytest-3.8.0, py-1.5.3, pluggy-0.7.1
rootdir: C:\Users\[...]\eclipse-workspace\pddev, inifile: setup.cfg
plugins: xdist-1.22.2, forked-0.2, cov-2.5.1, hypothesis-3.59.1
collected 27 items

pandas\tests\indexes\datetimes\test_astype.py .......................F.. [ 96%]
.                                                                        [100%]

================================== FAILURES ===================================
_____________________ TestToPeriod.test_to_period_tz[tz2] _____________________

self = <pandas.tests.indexes.datetimes.test_astype.TestToPeriod object at 0x0000027686590BA8>
tz = tzlocal()

    @pytest.mark.parametrize('tz', [
        'US/Eastern', pytz.utc, tzlocal(), 'dateutil/US/Eastern',
        dateutil.tz.tzutc()])
    def test_to_period_tz(self, tz):
        ts = date_range('1/1/2000', '2/1/2000', tz=tz)

        with tm.assert_produces_warning(UserWarning):
            # GH#21333 warning that timezone info will be lost
            result = ts.to_period()[0]
            expected = ts[0].to_period()

>       assert result == expected
E       AssertionError: assert Period('1999-12-31', 'D') == Period('2000-01-01', 'D')

pandas\tests\indexes\datetimes\test_astype.py:277: AssertionError

and one XPASS(strict):

(pandas-dev) C:\Users\[...]\eclipse-workspace\pddev>pytest pandas/tests/io/test_parquet.py
============================= test session starts =============================
platform win32 -- Python 3.6.6, pytest-3.8.0, py-1.5.3, pluggy-0.7.1
rootdir: C:\Users\[...]\eclipse-workspace\pddev, inifile: setup.cfg
plugins: xdist-1.22.2, forked-0.2, cov-2.5.1, hypothesis-3.59.1
collected 46 items

pandas\tests\io\test_parquet.py .....F.........ss..ss...........X...s..s [ 86%]
...s..                                                                   [100%]

================================== FAILURES ===================================
___________________________ test_cross_engine_pa_fp ___________________________
[XPASS(strict)] reading pa metadata failing on Windows/mac

[mroeschke]

The period error you found @h-vetinari was noted here: #22905