rename multi-index

[waitingkuo]

Fix Issue #4160

While renamed a MultiIndex, the mapper walk through every tuple in the MultiIndex and every element in a tuple. So the mapper { (index1, index2): (new_index1, new_index2) } didn't work for the MultiIndex case.

This pull request is to modify it to walk through every tuple in a MultiIndex and use the mapper to map each tuple.
Also, the test case has been added.

[jreback]

I think this could use some more corner test cases (it wlll prob work, just like to have them in there)

e,.g rename 'foo'

In [7]: df = DataFrame([['foo','foo',1.0],['bar','foo',1.5]],columns=list('ABC')).set_index(['A','B'])

In [8]: df
Out[8]: 
           C
A   B       
foo foo  1.0
bar foo  1.5

[jreback]

also, pls put a mention in doc/source/release.rst about this issue

[hayd]

I think you should be able to rename things from one level (like you can with columns) i.e. rename all the foos from level='B' above. Otherwise I think we're replacing one bug with a new bug. :(

No sure what syntax ought to be, should accept level name and number somehow

df.rename({'B': {'foo': 'foo2'}})
df.rename({2: {'foo': 'foo2'}})

?

[waitingkuo]

How do you think if I modify the function to something like

DataFrame.rename(mapper, axis=0, level=None, copy=True, inplace=False)

Where mapper is a dict-like or function

[hayd]

I'm not sure what is good API choice here... not sure levels argument makes sense (as you may want to change foos in A to something and foos in B to something else at the same time, indeed changing MultiIndex cols at the same time..., using either rename or rename_axis) though obv depends what you intended...

To dump my thoughts: there's quite a few cases here:

df.rename(index={'foo': 'foo2'})  # rename all foos in MultiIndex to foo2 (regardless of level)
df.rename(index={'A': {'foo': 'foo2'}, {'B': {'foo': 'foo1'}})  # rename with level names
df.rename(index={0: {'foo': 'foo2'}, {'B': {'foo': 'foo1'}})  # with mixed level names/numbers (should this work?)
df.rename(index={('foo', foo'): ('foo1', 'foo2')})  # with tuples 

... # all of the above with functions which essentially do the same thing

possibly more. Not sure how to test for the latter...

[cpcloud]

You could convert a MultiIndex to a DataFrame via something like mi_df = DataFrame(np.row_stack(df.index), columns=df.index.names) and then use the replace method to do your bidding and then convert back....I think this might be prohibitively expensive though

[jtratner]

If @waitingkuo doesn't mind, can we hold off on this for a day or two to see if we can't get #4039 in first?

[waitingkuo]

It's ok, let's hold it off.

[jtratner]

@waitingkuo I totally think you could add this, just want this to be merged after we merge #4039 (so, totally makes sense to prep/test/finalize this now). I only ask because #4039 changes name handling a little.

[waitingkuo]

@jreback Ok, I'll design the api first and then post here to discuss with you.

[jreback]

@waitingkuo I think @hayd example above all make sense

[jreback]

@waitingkuo any thoughts on API?

FYI, pls rebase off of master; renaming is now pretty simplified (and consistent for all NDFrame objects)

so should be changed in 1 place now (core/internals/rename and friends)

[jtratner]

@waitingkuo can you rebase on to master -- there have been some changes, but as @jreback noted, should be somewhat easier to do.

[jreback]

@waitingkuo can you rebase on master?

[jreback]

@waitingkuo ping

[jreback]

@waitingkuo can you rebase this...and see where we are?

[jreback]

@waitingkuo whats the state of this?

[waitingkuo]

@jreback , sorry for the very late reply. Due to some personal reason, I left here for a while. Now I'm back.

@hayd's makes sense and I started to implement it

df.rename(index={'foo': 'foo2'})  # rename all foos in MultiIndex to foo2 (regardless of level)
df.rename(index={'A': {'foo': 'foo2'}, {'B': {'foo': 'foo1'}})  # rename with level names
df.rename(index={0: {'foo': 'foo2'}, {'B': {'foo': 'foo1'}})  # with mixed level names/numbers (should this work?)
df.rename(index={('foo', foo'): ('foo1', 'foo2')})  # with tuples 

But there's a problem I just met and would like to discuss. Can I deem all the numbers as the level numbers? The names of the MultiIndex could be integer, it might be confused.

[jreback]

index._get_level_number(n) handles this I believe

though you could also do this to remove ambiguity

• if any level names are integers, and the renamer level is number just raise a valueError

pls add a test to show the ambiguity (e.g. use reversed level names like 2,1,0)

[hayd]

You'll definitely want to remove slightly different type of ambiguity (e.g. having 0 and name of first level passed / trying to rename the same column):

len(set(map(index._get_level_number,keys))) == len(keys)

[jreback]

@waitingkuo this is a nice feature...can you rebase and make those changes?

[jreback]

@waitingkuo can you rebase and can see where this is?

[jreback]

@waitingkuo ?

[jreback]

@waitingkuo are you going to be able to address this soon?

[jreback]

status on this?

[jreback]

@waitingkuo this would be nice for 0.15.0

[jreback]

@waitingkuo ?

[jreback]

closing for now. @waitingkuo if you want to update, pls do.

[gordol]

is this possible yet?

[jreback]

see the issue - it's open

[gordol]

yeah but there are about 10-15 dupe issues of this, and some of them have merged resolutions... it's not clear what the status is.

[jorisvandenbossche]

@gordol there are also many ways to rename a MultiIndex. For example, to rename a label in a single level in the meantime works (#13766), but using tuples to rename a full entry (the open issue) not yet