pd.concat of Series with int64 column and Series with int64-ExtensionArray yields int64

[xhochy]

Code Sample

import fletcher as fr
import pandas as pd

df_ext = pd.DataFrame({'a': fr.FletcherArray([1, 2])})
df_ext.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 2 entries, 0 to 1
# Data columns (total 1 columns):
# a    2 non-null fletcher[int64]
# dtypes: fletcher[int64](1)
# memory usage: 100.0 bytes

df_normal = pd.DataFrame({'a': [3, 4]})
df_normal.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 2 entries, 0 to 1
# Data columns (total 1 columns):
# a    2 non-null int64
# dtypes: int64(1)
# memory usage: 96.0 bytes

# Works
pd.concat([df_ext, df_normal]).info()
# <class 'pandas.core.frame.DataFrame'>
# Int64Index: 4 entries, 0 to 1
# Data columns (total 1 columns):
# a    4 non-null object
# dtypes: object(1)
# memory usage: 64.0+ bytes

# yield int64 instead of object
pd.concat([df_ext['a'], df_normal['a']]).dtype
# dtype('int64')

Problem description

This currently leads BaseReshapingTests.test_concat_mixed_dtypes to fail on ExtensionArrays that can be converted to any numeric data NumPy datatype.

[jreback]

where do you have a Int64 EA array defined?

[xhochy]

@jreback see my "prototype" package https://github.com/xhochy/fletcher that wraps Arrow arrays of any type into ExtensionArrays. The fr.FletcherArray([1, 2]) gives me a nullable int64 (Arrow-based) ExtensionArray.

[jreback]

as an FYI: #21160

this is currently not dependent on pyarrow, though its pushing the EA protocols.

[jreback]

also xref #21789

[xhochy]

Fixing this might also be a breaking change, e.g. the following behaviour is probably wanted:

In [4]: pd.concat([pd.Series([1, 2]), pd.Series([1., 2.])])
Out[4]:
0    1.0
1    2.0
0    1.0
1    2.0
dtype: float64

[xhochy]

Narrowed this down to

pandas/pandas/core/dtypes/concat.py

Line 181 in 63eff04

if any(extensions) and axis == 1:

Here we only do the special handling in the case of axis == 1 but in the series case, we have axis == 0.

Removing the check for axis == 1 sadly then produces a Series backed by a 2D-ndarray:

In [9]: s.values
Out[9]:
array([[1, 2],
       [1, 2]], dtype=object)

[jorisvandenbossche]

@xhochy this is generally fixed, in the meantime. The example now returns an object-dtype columns, but Fletcher can now override this by implementing the _get_common_dtype on the extension dtype to control casting behaviour in concat (#33607)