API: ExtensionArray.isin treatment of missing values

[mzeitlin11]

The current behavior singles out pd.NA:

In [3]: ser = pd.Series([pd.NA], dtype="Int64")

In [4]: ser.isin([pd.NA])
Out[4]:
0    True
dtype: boolean

In [5]: ser.isin([np.nan])
Out[5]:
0    False
dtype: boolean

StringArray.isin also follows this behavior, so it is not MaskedArray specific.

There was discussion about this being problematic since _from_sequence will treat other missing values just the same as pd.NA (#42473 (comment)). In that case, the output in the second case should also be True.

As a final option, both outputs could be pd.NA. In #38379 there was discussion of propagating pd.NA instead of True/False depending on the presence of missing values in the values argument. A nice description of that debate is here #38379 (comment).

[realead]

This behavior of isin was agreed upon, before pd.NA existed: #22296.

The idea back then was: in order to be consistent in all algorithms (isin, unique, mode, count and so on), the most robust way is to use the wanted equivalence relation already in khash-tables/maps. For two elements a and b being in the same equivalency class/being equivalent, the following should hold:

• a == b
• hash(a) == hash(b)

The decision back then was:

• all float-nans are equivalent ( thus Python's default == and hash need to be overwritten)
• pd.NaT is its own equivalency class
• None is its own equivalency class

It was a safer choice: easier to implement, less performance hits, there is still possibility through preproccessing to ensure that e.g. None and np.nan are mangled and there are probably scenarios, where None and np.nan shouldn't be the same.

This behavior was extrapolated to pd.NA, i.e. pd.NA became its own equivalency class.

The question is, what should be the behavior of pd.NA? Should it be in the same equivalency class as np.nan or should it be in the same equivalency asnp.nan, pd.NaT and None, i.e. all these classes should become the same equivalency class (which is quite a change of behavior and can have subtle changes of behavior as result).

Having pd.NA and np.nan in the same equivalency class is probably quite straight forward, putting all into the same equivalency class is probably more work (and might have more impact on behavior and performance).

[mzeitlin11]

Closing as a duplicate in favor of #31990