TypeError using DataFrame.mask

[sanzoghenzo]

Hi there, as always thanks for your hard work!

I'm runing into problem while trying to use the DataFrame.mask method on a DataFrame with PintArray.

Here's my code:

import pandas as pd
import pint_pandas

pint_pandas.show_versions()

df = pd.DataFrame({
    "distance": pd.Series([1, 2, 3, 4], dtype="pint[m]"),
    "numbers1": pd.Series([1, 2, 2, 3], dtype="int64"),
    "numbers2": pd.Series([1, 1, 2, 3], dtype="int64"),

})

mask = df["numbers2"] > 1

df.mask(mask)

It returns:

{'numpy': '1.19.4', 'pandas': '1.2.0', 'pint': '0.17', 'pint_pandas': '0.2'}


Traceback (most recent call last):
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pint\quantity.py", line 1851, in __getitem__
    return type(self)(self._magnitude[key], self._units)
TypeError: 'float' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/a.ghensi/AppData/Roaming/JetBrains/PyCharm2021.1/scratches/scratch_1.py", line 29, in <module>
    main()
  File "C:/Users/a.ghensi/AppData/Roaming/JetBrains/PyCharm2021.1/scratches/scratch_1.py", line 23, in main
    df2 = df.mask(mask)
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pandas\core\generic.py", line 9317, in mask
    errors=errors,
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pandas\core\generic.py", line 9280, in where
    cond, other, inplace, axis, level, errors=errors, try_cast=try_cast
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pandas\core\generic.py", line 9135, in _where
    axis=block_axis,
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pandas\core\internals\managers.py", line 564, in where
    axis=axis,
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pandas\core\internals\managers.py", line 427, in apply
    applied = getattr(b, f)(**kwargs)
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pandas\core\internals\blocks.py", line 2001, in where
    set_other = other[icond]
  File "C:\tools\miniconda3\envs\dprock\lib\site-packages\pint\quantity.py", line 1857, in __getitem__
    "supports indexing".format(self._magnitude)
TypeError: Neither Quantity object nor its magnitude (nan)supports indexing

The error stems from pandas ExtensionBlock.where method:

• first, if the replacement value (nan in this case) is a scalar and isna, then it uses the dtype.na_value (Quantity(np.nan, "meters") in this case)
• then, it re-checks if the value is a scalar, and if not it treats it as an indexable and uses the mask as index. This throws the error because of pd.api.types.is_scalar(Quantity(np.nan, "meters") == False.

Is it something that can be done here or it's a pandas issue?

[sanzoghenzo]

I should have searched the pandas issues first!

[MichaelTiemannOSC]

I'm getting hit by this, too! df.where does not behave well when it passes rows with NaNs. I see the above link to pandas...we'll see what happens.

[andrewgsavage]

this works now

	distance	numbers1	numbers2
0	1.0	1.0	1.0
1	2.0	2.0	1.0
2	nan	NaN	NaN
3	nan	NaN	NaN