DEPR: deprecate .add_prefix and .add_suffix #18347

topper-123 · 2017-11-17T17:26:59Z

[x ] xref DEPR: let's deprecate #18262
[ x] tests added / passed
[ x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
[ x] whatsnew entry

Deprecate .add_prefix and .add_suffix as per #18262.

An issue is that .add_prefix and .add_suffix call the like-named methods in internals.py::BlockManager. Should BlockManager.add_prefix/add_suffix be changed somehow, e.g. have a warning as well?

EDIT: This is my first deprecation PR, so I'll be more than appreciative of criticism, so I understand these better in the future.

EDIT 2: With .add_prefix/add_postfix deprecated, I think it would be good to show in the doc strings for .rename how to do similar things, so I've changed its doc string a bit.

codecov · 2017-11-17T19:01:39Z

Codecov Report

Merging #18347 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #18347      +/-   ##
==========================================
- Coverage   91.59%   91.57%   -0.03%     
==========================================
  Files         150      150              
  Lines       48959    48961       +2     
==========================================
- Hits        44843    44835       -8     
- Misses       4116     4126      +10

Flag	Coverage Δ
#multiple	`89.93% <100%> (-0.03%)`	⬇️
#single	`41.13% <33.33%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/generic.py	`95.92% <100%> (ø)`	⬆️
pandas/plotting/_converter.py	`65.22% <0%> (-1.74%)`	⬇️
pandas/util/testing.py	`84.9% <0%> (+0.21%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cdebcf3...3a5785a. Read the comment docs.

gfyoung · 2017-11-18T00:01:50Z

pandas/tests/frame/test_api.py

-        with_pct_suffix = self.frame.add_suffix('%')
-        expected = pd.Index(['{}%'.format(c) for c in self.frame.columns])
-        tm.assert_index_equal(with_pct_suffix.columns, expected)
+        with pytest.warns(FutureWarning):


@jreback @jorisvandenbossche : Are we still going with tm.assert_produces_warning, or is using the pytest construct okay to use?

yes I prefer to use our tm.assert_produces_warning for now (if we change then we should simply globally change, but in a separate set of PRs)

jreback · 2017-11-19T16:08:48Z

pandas/tests/frame/test_api.py

-        with_pct_suffix = self.frame.add_suffix('%')
-        expected = pd.Index(['{}%'.format(c) for c in self.frame.columns])
-        tm.assert_index_equal(with_pct_suffix.columns, expected)
+        with pytest.warns(FutureWarning):


yes I prefer to use our tm.assert_produces_warning for now (if we change then we should simply globally change, but in a separate set of PRs)

jreback · 2017-11-19T16:09:15Z

pandas/core/generic.py

-        0  1  4
-        1  2  5
-        2  3  6
+        >>> df.rename(index="index_{}".format, columns={"A": "a", "B": "c"})


any use of .add_prefix/suffix in the main docs? if so can you update

No, it's not used in the docs. It's shown in 10min_to_pandas as a result of tab completion, but that should stay until removal.

no please change the docs now

Sorry, I miswrote: it's not in the docs, expect as a result of that tab comletion example.

ahh these need to be added to
_deprecations in series and dataframe

jreback · 2017-11-19T16:09:46Z

@jorisvandenbossche you had some concerns here?

jreback · 2017-11-19T20:06:36Z

pandas/core/frame.py

@@ -304,7 +304,8 @@ def _constructor(self):

    _constructor_sliced = Series
    _deprecations = NDFrame._deprecations | frozenset(
-        ['sortlevel', 'get_value', 'set_value', 'from_csv'])
+        ['sortlevel', 'get_value', 'set_value', 'from_csv', 'add_prefix',


instead add to NDFrame._deprecations (which are both used in Series/Frame)

jorisvandenbossche · 2017-11-19T20:12:57Z

@jorisvandenbossche you had some concerns here?

Yes, as I mentioned in the other issue, not sure if we should deprecate this one. Based on stackoverflow questions (and doc page views) this is actually used a bit (in contrast to eg asobject)

jreback · 2017-11-19T20:18:15Z

Yes, as I mentioned in the other issue, not sure if we should deprecate this one. Based on stackoverflow questions (and doc page views) this is actually used a bit (in contrast to eg asobject)

where do you see doc page views?

tdpetrou · 2017-11-22T23:38:10Z

They are used fairly often on stack overflow. I use them. You can see about 20 posts with them this month on stack overflow.

That said, personally, I'd like to shrink pandas as much as possible and wouldn't mind seeing them gone.

jorisvandenbossche

Just to give feedback on how you did the deprecation: added two small comments, but it's certainly good!
(so go ahead with some less controversial deprecations :-))

That said, I am still in doubt we should do this one.

jorisvandenbossche · 2017-11-23T09:08:05Z

pandas/core/generic.py

        """
+        warnings.warn("'add_prefix' is deprecated and will be removed in a "
+                      "future version.", FutureWarning, stacklevel=2)


Mention the alternative in the message

jorisvandenbossche · 2017-11-23T09:10:49Z

pandas/core/generic.py

@@ -2590,6 +2595,8 @@ def _update_inplace(self, result, verify_is_copy=True):

    def add_prefix(self, prefix):
        """
+        DEPRECATED: Use ``obj.rename('prefix_{}'.format)`` or similar instead.


I am a bit in doubt whether I prefer this alternative or rather obj.rename(lambda x: "prefix_" + x) or obj.rename(lambda x: 'prefix_{}'.format(x)) (it's just that I think such a bare format is not that easy to understand for beginners that it is a function, but of course lambda's are also a strange beast)

BTW, you need to add columns= to have the correct alternative I think

agree here, using the lambda is a bit simpler for beginners.

topper-123 · 2017-11-24T01:08:51Z

@jorisvandenbossche: Ok, I can see that pople use them, but they're just so useless. There was at one point a discussion on adding a .format method to pandas objects, which would solve the same problem as add_prefix/-suffix but be more general. So:

>>> df.columns = df.columns.format("prefix_{}_suffix")
>>> df.columns.format("prefix_{}_suffix", inplace=True)  # equivalent, but inplace

This would be much more generally useful than add_prefix/-suffix, and would operate on the relevant object, which IMO is often more readable than methods on the dataframe.

Just an idea, as I could often use such methods and add_prefix/-suffix are too limiting and apply feels 'dirty', for a lack of better words.

jreback · 2017-11-24T22:11:30Z

We actually have this almost already

In [13]: i = pd.Index(list('abc'))

In [15]: i.str.cat(['foo'] * 3, sep='_')
Out[15]: Index(['a_foo', 'b_foo', 'c_foo'], dtype='object')

wouldn't be a stretch to allow a single element here, e.g.
i.str.cat('foo'), or maybe i.str.suffix() and i.str.prefix()

these are much more appealing that a method on Series/DataFrame

topper-123 · 2017-11-24T22:43:58Z

these are much more appealing that a method on Series/DataFrame.

I agree, though two points:

.str accessor only works on indices of type object (prob. specifically str, haven't checked), while add_prefix/suffix work for all data types. So for e.g. a float Index, you'd still have to do i.apply('prefix_{:+.1f}_suffix'.format), which would mean different operations for str vs. non-str data types, which isn't good.
both str.cat and potential str.prefix/suffix are/would be quite specific. A .format method would be more general and would map very well over to people's general Python knowledge, i.e. their knowledge of the Python string formatting language would transfer well over to Pandas, which would be nice.

So, I'd be the most positive to adding a str.format method, but not str.add_prefix/-suffix. But I'd also like some sanctioned but still flexible way to cast non-string types to string dtype, as astype(str) is too crude (lacks string formatting options) and using .apply feels unsanctioned and dirty.

jreback · 2017-11-25T15:15:26Z

.str accessor only works on indices of type object (prob. specifically str, haven't checked), while add_prefix/suffix work for all data types. So for e.g. a float Index, you'd still have to do i.apply('prefix_{:+.1f}_suffix'.format), which would mean different operations for str vs. non-str data types, which isn't good.

This is by-definition and on-purpose. string methods should not work on non-strings, full-stop.

We also want composability. So and I think there is an issue about this somewhere, having a generic .format() method that is basically an .astype(str), or maybe .astype(str, format=....) is prob ok (IIRC we had an issue about this somewhere, @jorisvandenbossche ?)

topper-123 · 2017-11-26T01:29:40Z

(IIRC we had an issue about this somewhere, @jorisvandenbossche ?)

I posted an issue #17211 on this.

jorisvandenbossche · 2017-11-27T22:54:32Z

As I said in the issue some months ago, I am +1 on adding a .format method. However, I think that is more generally usefu and somewhat orthogonal to this one. Eg it does not allow chaining method calls (for that rename would still be the alternative)

pep8speaks · 2017-12-24T09:58:10Z

Hello @topper-123! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on December 24, 2017 at 10:45 Hours UTC

jreback · 2018-02-11T15:25:54Z

ok, so I think we need to add a .format() method before we can deprecate these, @topper-123 interested?

jreback · 2018-07-07T16:07:01Z

closing, we had a discussion about this, it is a bit premature to do this as these are useful. so alternative is to add a suffixes arg to pd.concat (#21791)

jbrockmendel · 2019-11-19T21:26:43Z

Should this be removed from the checklist in #6581?

topper-123 force-pushed the deprecate_prefix_postfix branch 3 times, most recently from dbc445e to b125c7b Compare November 17, 2017 20:12

topper-123 changed the title ~~WIP: deprecate add_prefix and add_postfix~~ DEPR: deprecate .add_prefix and .add_sufffix Nov 17, 2017

topper-123 force-pushed the deprecate_prefix_postfix branch 3 times, most recently from 59ff13b to eff7b11 Compare November 17, 2017 21:31

gfyoung added the Deprecate Functionality to remove in pandas label Nov 17, 2017

gfyoung reviewed Nov 18, 2017

View reviewed changes

topper-123 mentioned this pull request Nov 18, 2017

DEPR: let's deprecate #18262

Closed

34 tasks

jreback approved these changes Nov 19, 2017

View reviewed changes

jsexauer mentioned this pull request Nov 19, 2017

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

topper-123 force-pushed the deprecate_prefix_postfix branch 3 times, most recently from 79ab0aa to db08605 Compare November 19, 2017 20:03

jreback requested changes Nov 19, 2017

View reviewed changes

topper-123 force-pushed the deprecate_prefix_postfix branch from db08605 to 3997642 Compare November 19, 2017 20:33

jorisvandenbossche reviewed Nov 23, 2017

View reviewed changes

topper-123 force-pushed the deprecate_prefix_postfix branch from 3997642 to 1e83d89 Compare November 24, 2017 21:51

jreback changed the title ~~DEPR: deprecate .add_prefix and .add_sufffix~~ DEPR: deprecate .add_prefix and .add_suffix Nov 25, 2017

jreback mentioned this pull request Nov 26, 2017

DISCUSSION: Add format parameter to .astype when converting to str dtype #17211

Open

topper-123 force-pushed the deprecate_prefix_postfix branch from 1e83d89 to e797713 Compare December 24, 2017 09:58

Deprecate .add_prefix and .add_suffix

3a5785a

topper-123 force-pushed the deprecate_prefix_postfix branch from e797713 to 3a5785a Compare December 24, 2017 10:45

jreback mentioned this pull request Jul 7, 2018

ENH: add suffixes arg to concat #21791

Closed

jreback closed this Jul 7, 2018

topper-123 deleted the deprecate_prefix_postfix branch October 27, 2018 08:16

jorisvandenbossche mentioned this pull request Nov 8, 2018

Add to_flat_index method to MultiIndex #22866

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPR: deprecate .add_prefix and .add_suffix #18347

DEPR: deprecate .add_prefix and .add_suffix #18347

topper-123 commented Nov 17, 2017 •

edited by jreback

Loading

codecov bot commented Nov 17, 2017 •

edited

Loading

gfyoung Nov 18, 2017

jreback Nov 19, 2017

jreback Nov 19, 2017

jreback Nov 19, 2017

topper-123 Nov 19, 2017 •

edited

Loading

jreback Nov 19, 2017

topper-123 Nov 19, 2017

jreback Nov 19, 2017

topper-123 Nov 19, 2017

jreback commented Nov 19, 2017

jreback Nov 19, 2017

jorisvandenbossche commented Nov 19, 2017

jreback commented Nov 19, 2017

tdpetrou commented Nov 22, 2017 •

edited

Loading

jorisvandenbossche left a comment

jorisvandenbossche Nov 23, 2017

topper-123 Nov 24, 2017

jorisvandenbossche Nov 23, 2017

jreback Nov 23, 2017

topper-123 Nov 24, 2017

topper-123 commented Nov 24, 2017

jreback commented Nov 24, 2017

topper-123 commented Nov 24, 2017

jreback commented Nov 25, 2017

topper-123 commented Nov 26, 2017 •

edited

Loading

jorisvandenbossche commented Nov 27, 2017

pep8speaks commented Dec 24, 2017 •

edited

Loading

jreback commented Feb 11, 2018

jreback commented Jul 7, 2018

jbrockmendel commented Nov 19, 2019

DEPR: deprecate .add_prefix and .add_suffix #18347

DEPR: deprecate .add_prefix and .add_suffix #18347

Conversation

topper-123 commented Nov 17, 2017 • edited by jreback Loading

codecov bot commented Nov 17, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topper-123 Nov 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 19, 2017

Choose a reason for hiding this comment

jorisvandenbossche commented Nov 19, 2017

jreback commented Nov 19, 2017

tdpetrou commented Nov 22, 2017 • edited Loading

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topper-123 commented Nov 24, 2017

jreback commented Nov 24, 2017

topper-123 commented Nov 24, 2017

jreback commented Nov 25, 2017

topper-123 commented Nov 26, 2017 • edited Loading

jorisvandenbossche commented Nov 27, 2017

pep8speaks commented Dec 24, 2017 • edited Loading

Comment last updated on December 24, 2017 at 10:45 Hours UTC

jreback commented Feb 11, 2018

jreback commented Jul 7, 2018

jbrockmendel commented Nov 19, 2019

topper-123 commented Nov 17, 2017 •

edited by jreback

Loading

codecov bot commented Nov 17, 2017 •

edited

Loading

topper-123 Nov 19, 2017 •

edited

Loading

tdpetrou commented Nov 22, 2017 •

edited

Loading

topper-123 commented Nov 26, 2017 •

edited

Loading

pep8speaks commented Dec 24, 2017 •

edited

Loading