Series.str.replace() is not actually the same as str.replace()

[rosnfeld]

Code Sample

In [1]: import pandas as pd

In [2]: series = pd.Series(['a', '(b)'])

In [3]: series.str.replace('a', '[a]')
Out[3]: 
0    [a]
1    (b)
dtype: object

In [4]: series.str.replace('(b)', '[b]')  # unexpected behavior
Out[4]: 
0        a
1    ([b])
dtype: object

In [5]: series.str.replace('\(b\)', '[b]')  # need to escape
Out[5]: 
0      a
1    [b]
dtype: object

In [6]: '(b)'.replace('(b)', '[b]')   # Python str.replace is different, uses literal string
Out[6]: '[b]'

Problem description

The documentation for Series.str.replace says that it takes a "string or compiled regex" ... "String can be a character sequence or regular expression." ... "When repl is a string, every pat is replaced as with str.replace()"

However, that's not what is happening - it appears it's interpreting a string as a regex, so you need to escape characters like parentheses.

Expected Output

I would expect that for vanilla strings, it works like regular Python str.replace() - using literal strings instead of regexes.

Alternatively the documentation could be updated, but I think the Python str.replace() behavior is what most users would expect.

Output of pd.show_versions()

# Paste the output here pd.show_versions() here INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-83-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8

pandas: 0.20.2
pytest: 3.1.2
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.0
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

[jreback]

from the doc-string

Signature: Series.str.replace(self, pat, repl, n=-1, case=None, flags=0)
Docstring:
Replace occurrences of pattern/regex in the Series/Index with
some other string. Equivalent to :meth:str.replace or
:func:re.sub.

it is a bit ambiguous, though so is a pattern like (b), is it a regex or not? This is almost always a regex. So I suppose a case could be made for:

• adding a .sub function which is by-definition a regex and making .replace strict
• adding a regex=True arg to control whether to interpret this as a regex.

[rosnfeld]

I'm fine with either solution, not sure which is more intuitive or less disruptive to existing practice. I too had considered suggesting an additional "regex" param but had forgotten it when I went to write the issue up; if we go that way I'd argue for defaulting to regex=False.

For what it's worth, a user might not know they are passing a string that "looks like a regex". Maybe you got a result from some other operation and didn't inspect it, but still passed it in to series.str.replace(). I stumbled upon this issue doing something like the following: imagine an item description with a parenthetical remark - e.g. "Men's shirt (blue)" - and you try and replace all instances of that with some other text, maybe just "Men's clothing". When some of your replacements fail to take, it's pretty surprising.

In [1]: import pandas as pd

In [2]: series = pd.Series(['No parens', 'Some text (with parens)'])

In [3]: series.str.replace('No parens', 'Replaced')
Out[3]: 
0                   Replaced
1    Some text (with parens)
dtype: object

In [4]: series.str.replace('Some text (with parens)', 'Replaced') # fails to replace
Out[4]: 
0                  No parens
1    Some text (with parens)
dtype: object

In [5]: series.str.replace(series.max(), 'Replaced') # nothing gets replaced!
Out[5]: 
0                  No parens
1    Some text (with parens)
dtype: object

[Liam3851]

We just hit this issue. At this point the problem goes back at least to 0.16 and is still present in the current 0.23 dev. Problem is at

use_re = is_compiled_re or len(pat) > 1 or flags or callable(repl) 

Thus any pattern len>1 is interpreted as a regex, whether it actually can be parsed as a regex or not.

At this point, any change that made it non-default to regex would be breaking, so I think the options are:

1. Update the documentation only. This function does not support replacing "character sequences", it supports replacing a single character OR a regex if multiple characters.

2. Add regex=True and make the above line something like

use_re = is_compiled_re or (regex and (len(pat) > 1 or flags)) or callable(repl)

Any thoughts? Current behavior really is inconsistent with the doc.

[TomAugspurger]

+1 for adding regex=True. We have this elsewhere in, e.g. .str.contains.

Interested in making a PR @Liam3851?

[Liam3851]

I can do a PR. Question on API, though, before I write. Suppose someone gives us

ser.str.replace(re.compile('[abc]'), 'd', regex=False)

Should that raise or should we just silently use regex?

Also, while

use_re = is_compiled_re or (regex and (len(pat) > 1 or flags)) or callable(repl)

maintains back compat, it does mean that regex=True with a single character always basically implies regex=False. I suppose it could be confusing for someone to explicitly write

ser.str.replace('.', 'a', regex=True)

and have it only replace periods and not all characters. But maybe that's such a rare request anyway we can let it go for the sake of back compat?

[tobycheese]

Is it intentional that in Series|DataFrame.replace, regex defaults to False, in Series.str.replace regexp defaults to True (and plain python str.replace does not use regexps)?

That default to True often bites me, maybe we can add a warning?