PERF: Improve performance for df.duplicated with one column subset

[phofl]

• closes PERF: DataFrame.duplicated with subset= for 1 column is slower than Series.duplicated #45236
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

%timeit df["a"].duplicated()
3.16 ms ± 96 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit df.duplicated(subset=['a'])
3.1 ms ± 75.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The Index creation and the get_group_index were both at fault for the slowdown. Removing the Index call should improve the performance for longer subsets too.

[jreback]

thanks @phofl

[simonjayhawkins]

@phofl several cases where the changes maybe viewed as breaking changes by users. maybe need to expand the release note or revert this PR.

[phofl]

I'd rather expand the release note than reverting, because the new behavior seems to be correct?

[simonjayhawkins]

It is now only "correct" where we have a single column DataFrame and previously the DataFrame methods had bugs. I think that an inconsistency between a single column DataFrame (or single subset) is less desirable than the existing inconsistency between the Series and DataFrame methods.

Of course this is only applicable to the few cases reported and that all the bugs reported apply to object dtype columns only.

So probably not necessary to revert.