Allow to select index in drop_duplicates and duplicated

[flying-sheep]

there’s no way to drop rows with duplicated index using drop_duplicates.

we’d have to add a copy of the index as column, or do this:

df[np.logical_not(df.index.duplicated(take_last=True).values)]

[TomAugspurger]

Typically I'll use a df.groupby(level=0).last() (or more typically .first()). It works fine, but a groupby isn't necessarily the first thought for deduplication.

I'm +0 on whether we should have a dedicated method for this.

[jreback]

As @TomAugspurger indicates the following are equivalent.

I suppose the drop_duplicates section could have this an alterative example. If you would like to pull-request for a doc update would be ok.

In [6]: df = pd.DataFrame({'A' : range(4), 'B' : list('aabb')})            

In [7]: df                                                                 
Out[7]:                                                                    
   A  B                                                                    
0  0  a                                                                    
1  1  a                                                                    
2  2  b                                                                    
3  3  b                                                                    

In [9]: df2 = df.set_index('B')                                            

In [10]: df2                                                               
Out[10]:                                                                   
   A                                                                       
B                                                                          
a  0                                                                       
a  1                                                                       
b  2                                                                       
b  3   

In [13]: df2.groupby(level=0).first()                        
Out[13]:                                                     
   A                                                         
B                                                            
a  0                                                         
b  2                                                         

In [16]: df2.reset_index().drop_duplicates(subset='B',take_last=False).set_index('B')                                                      
Out[16]:                                                                                                                                   
   A                                                                                                                                       
B                                                                                                                                          
a  0                                                                                                                                       
b  2

[flying-sheep]

sorry, i don’t get it. you mean i should add the second code block as exemple to the docs?

[jreback]

I would add the groupby method as an alternative as its is another common way of performing this task

[flying-sheep]

to which file? indexing.rtf?

[jreback]

http://pandas.pydata.org/pandas-docs/stable/indexing.html#duplicate-data
(which is in indexing.rst)

[zydariv]

Where is the problem to just add this functionality?

df2.reset_index().drop_duplicates(subset='B',take_last=False).set_index('B')
looks not really clean to me.

df2.drop_duplicates(subset='index', take_last=False)
would look much cleaner and we could add the reset_index() and set_index() into drop_duplicates()

cheers

[jreback]

something like this was already added: #30405