ENH: allow storing ExtensionArrays in the Index

[jorisvandenbossche]

I started experimenting with allowing EAs to be used in the Index (so we can put nullable int/bool, string dtype etc in the Index).

My approach is to use the base Index class for this (and not a custom subclass):

• We use a similar approach for Series, which can also hold either ndarray or EA as underlying values
• I don't think we want to start adding subclasses for all new EAs (like Int64Index for the new Int64Dtype is not quite possible ..)
• This also works for external EAs

For now, it just integrates it, so an EA can be put in an index and tested operations work. This PR does not (yet) handle ensuring that operations relying on the index' Engine are efficient (currently it simply uses np.asarray(EA) as the engine values, which eg will be object dtype for Int64Dtype)

cc @jbrockmendel @TomAugspurger @jreback

[jbrockmendel]

This is the idea behind ExtensionIndex

[jreback]

I agree with @jbrockmendel here, this is the exactly the point of ExtensionIndex. trying to modify Index doesn't make much sense.

[jorisvandenbossche]

Sorry, I should have been more explicit regarding ExtensionIndex in my reasoning above:

• Yes, this is (somewhat/fully) duplicating ExtensionIndex. But I think some things in ExtensionIndex can just be moved to the base class index. Again, for Series we have exactly the same approach of having both ndarray or EA as underlying values (we don't have an "ExtensionSeries").
• Currently, the ExtensionIndex is only used as base class that concrete Index classes subclass from. Here, it is meant to be the actual class that users use/see. I personally don't think this should be "ExtensionIndex"

[jorisvandenbossche]

This is the idea behind ExtensionIndex

@jbrockmendel to make it concrete: are there (technical) reasons for you that are specific to Index and that make it more important to use a custom ExtensionIndex subclass instead of handling this in the base Index class?

For example, both have a __getitem__, which in both cases dispatch to self._data[..], but with some different checks (eg the base Index also checks for object boolean masks). But is there a fundamental reason those can't be merged?

Other example: the dropna in both is exactly the same, except for the use of self._data vs self._values, but self._values is defined to be self._data.
For repeat the same.
For take as well, except that the base class method just has a branch for _can_hold_na being False as well (which is never the case for the ExtensionIndex)

(a pre-cursor PR to this one could be to remove the duplicated methods between Index and ExtensionIndex)

[jbrockmendel]

to make it concrete: are there (technical) reasons for you that are specific to Index and that make it more important to use a custom ExtensionIndex subclass instead of handling this in the base Index class?

Nothing that I have a particularly strong opinion about. If the implementation in this PR turns out to be elegant (unless asked, ill wait to review in detail until green), i wouldn't have a problem with it.

That said, ATM whenever we have an Index object we know it is object-dtype. I expect there are places in the code where we are relying on that. (might be worth making an ObjectIndex? orthogonal discussion)

[jreback]

@jorisvandenbossche yeah this is almost certainly not going to 100% overlap with an object dtyped Index base, because that serves 2 master right now, the object dtyped and a base index. If we would create a ObjectIndex then maybe what you are doing would work, but I find it easier / more compatible to simply have an ExtensionArrayIndex that can hold any type of EA.

[jbrockmendel]

arr = self._values could go outside the if/elif

[jbrockmendel]

doesnt np.asarray(self._values) work unconditionally?

[jbrockmendel]

+1

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[jbrockmendel]

@jorisvandenbossche im working on getting this feature out the door; i think we can close this to clear the queue

[jreback]

closing as duplicate of #37869 can move discussion there.

[jbrockmendel]

@jorisvandenbossche I suggested closing this not based on the Index[EA] vs ExtensionIndex difference, but bc you don't seem to be working on it. If we get the other one working well, we can revisit the possibility of getting Index[EA] working.

(FWIW as I see more and more of the DTI/TDI/PI code getting shared, I've found myself thinking about the possibility those may not need to be subclasses at all [in this daydream, the subclass-specific stuff would be accessed by a .dt accessor xref #17134], which bears a resemblance to the Index[EA] design)

[jreback]

@jorisvandenbossche pls update or close

[jorisvandenbossche]

We first need to agree on how to do this (see discussion above, and also the overview issue I created at #39133)

[mroeschke]

Appears this PR draft has been dormant for a while so closing. Feel free to reopen when you have time to continue

[jorisvandenbossche]

Note for future reference: this work is now actually done in #43930