ENH: let get_dummies take a DataFrame

[TomAugspurger]

Closes #8133

Thoughts? I encode every column with object dtype. You can control the ones that get encoded with categorical_variables. The encoded columns are concated to the columns that don't need encoding.

prefix and prefix_sep can be passed as a string, list, or dict mapping column names to strs.

In [1]: from pandas import *

In [2]: df = DataFrame({'A': ['a', 'b', 'a'], 'B': ['b', 'a', 'c'],
                        'C': [1, 2, 3]})

In [3]: df
Out[3]: 
   A  B  C
0  a  b  1
1  b  a  2
2  a  c  3

In [4]: get_dummies(df)
Out[4]: 
   C  A_a  A_b  B_a  B_b  B_c
0  1    1    0    0    1    0
1  2    0    1    1    0    0
2  3    1    0    0    0    1

In [5]: get_dummies(df, prefix=['from_A', 'from_B'])
Out[5]: 
   C  from_A_a  from_A_b  from_B_a  from_B_b  from_B_c
0  1         1         0         0         1         0
1  2         0         1         1         0         0
2  3         1         0         0         0         1

In [6]: get_dummies(df, prefix={"A": 'from_A', "B": 'from_B'})
Out[6]: 
   C  from_A_a  from_A_b  from_B_a  from_B_b  from_B_c
0  1         1         0         0         1         0
1  2         0         1         1         0         0
2  3         1         0         0         0         1

In [7]: get_dummies(df, prefix_sep='.')
Out[7]: 
   C  A.a  A.b  B.a  B.b  B.c
0  1    1    0    0    1    0
1  2    0    1    1    0    0
2  3    1    0    0    0    1

In [8]: get_dummies(df, prefix_sep='.', categorical_variables=['A'])
Out[8]: 
   B  C  A.a  A.b
0  b  1    1    0
1  a  2    0    1
2  c  3    1    0

[jreback]

you are inconsistent with cat_variables and categorical_variables
I would eliminate convert_dummies (or prfix with _)

can u show the output in the top of this pr (of the examples)

[TomAugspurger]

Yeah I chose to spell out cat_variables. I figured it would be ok since covert_dummies wasn't used. I can change to the shortened version to be consistent if need be.

I'll put a deprecation warning on covert_dummies saying to just use get dummies instead. We should be able to remove it entirely pretty soon.

[TomAugspurger]

I didn't mean to change covert_dummies at all. I'll change that keyword argument back to how it was.

[TomAugspurger]

Added convert_dummies to #6581 We'll remove in 0.16 I think.

I'm updating that issue to keep a log of when things need to be removed by release.

[jreback]

I don't this we should have prefix_sep
we don't do that anywhere else (you could create another issue for that however to add it to multiple functions and/or have an set_option for it which might be better)

[TomAugspurger]

Made an issue about the prefix_sep #8141

[jorisvandenbossche]

Some feedback:

• Now that we have a 'category' dtype, you should also check for that (so not only 'object' dtype)
• good to deprecate/remove convert_dummies
• I am not too fond on the categorical_variables keyword .. It is really long, what do other people think of the include/exclude as what you mentioned in the original issue (Easy function for making dummy variable matrices #955 (comment)), or simply columns?

[jorisvandenbossche]

variable -> variables.
Also wording has to be specified a bit, as we now have a distinct categorical type, which is not the same as object

[jreback]

I would use columns rather than category_variables

[TomAugspurger]

Thanks.

• Changed categorical_variables -> columns
• Encoding object and categorical dtypes by default
• Fixed typos
• Change Deprecation to FutureWarning and caught that in the tests.

[TomAugspurger]

See anything else that needs fixing?

[jorisvandenbossche]

Looks good to me.
@hayd I think you did some work previously on get_dummies?

[TomAugspurger]

Andy added the NaN handling. I didn't have this in an example, but it behaves pretty much the same except you can have multiple NaN columns in the output, one for each input with a NaN.

[hayd]

+1 on this and also on depreciating convert_dummies.

seems good extension! I like your NaN handling but maybe be worth adding test with dummy_na=False (I can't see one)?

Note: I previously added str.get_dummies but, relooking at that, that's slightly different.

[TomAugspurger]

Thanks. Added a test with NaNs, but dummy_na=False. I'll merge later today if there aren't any objects.

[jorisvandenbossche]

Now, with a Series, the nan column 'looks' like NaN in the repr:

In [54]: pd.get_dummies(ss, dummy_na=True)
Out[54]:
    a    b   NaN
0    1    0    0
1    0    1    0
2    1    0    0
3    0    0    1

So my first thought was: shouldn't this break? But it appeared the NaN was not a string but a real nan value.
But apart from that, to keep it 'visually' the same as the Series case, maybe use A_NaN and B_NaN ?

[TomAugspurger]

Good point. I'd prefer a string with 'name_NaN' than having multiple real NaNs in the output, agreed?

[jreback]

really try not to have actual nan in an index value - not friendly to indexing

[jorisvandenbossche]

yes, certainly agree (I would even wanted to argue that in a Series case it should also be a string, but that is another discussion, and also somewhat difficult if you dummify numerical values)

[TomAugspurger]

FYI this is the same as current behavior (master):
s is a Series.

In [10]: pd.get_dummies(s, dummy_na=True, prefix='A')
Out[10]: 
   A_a  A_b  A_c  A_nan
0    1    0    0      0
1    0    1    0      0
2    1    0    0      0
3    0    0    1      0
4    0    0    0      1

I can fix if you want, but it will be an API change. The problem is that str(np.nan) is called.

[jorisvandenbossche]

Ah yes, in that case, maybe leave it as is. Don't think it is worth an API change.

[TomAugspurger]

Going to merge now with the prefix_nan vs. prefix_NaN issue unchanged since that would (potentially) break code.