ENH: add StringMethods (.str accessor) to Index, fixes #9068 #9667

mortada · 2015-03-17T03:06:25Z

fixes #9068

mortada · 2015-04-01T19:02:25Z

@jreback please let me know what you think of this, thanks

jreback · 2015-04-01T19:14:09Z

this looks good
pls add a release note in v0.16.1

jreback · 2015-04-01T19:14:46Z

may also want to update the docs to mention u can do this on Index (I think it's in text.rst iirc)

mortada · 2015-04-02T00:17:54Z

@jreback both release note and text.rst have been updated

shoyer · 2015-04-02T00:28:48Z

pandas/core/base.py

@@ -497,6 +498,19 @@ def searchsorted(self, key, side='left'):
        #### needs tests/doc-string
        return self.values.searchsorted(key, side=side)

+    # string methods
+    def _make_str_accessor(self):
+        if not com.is_object_dtype(self.dtype):


Can you add a check here to exclude string methods on MultiIndex objects, too? They have object dtype, too.

@shoyer you mean a simple check like this?

if isinstance(self, MultiIndex): raise AttributeError(".str accessor is not supported for MultiIndex objects")

Yes, that looks about right

Normally, I would suggest only putting this accessor on the appropriate types directly, but it's a little tricky with MultiIndex because it's a subclass of the generic Index type (which is what string indexes area)

I think better to check the inferred type
which will be string or mixed for s multi index (iirc)

mortada · 2015-04-02T19:16:52Z

pandas/core/base.py

+            raise AttributeError("Can only use .str accessor with string "
+                                 "values (i.e. inferred_type is 'string')")
+        return StringMethods(self)
+


@shoyer @jreback I improved the checking here according to your recommendations - in the case of Series it's the same logic as before and in the case of Index it uses the inferred_type attribute

sinhrks · 2015-04-05T01:13:35Z

Nice, but 2 points:

There are some function returns bool, and Index(dtype=bool) is not usable very much (related to API: Index should support __inverse__ ops #8875). It may be better to return np.array in these cases.
Should prohibit to use return_type="frame" in split ?

jreback · 2015-04-05T23:15:09Z

@sinhrks can you elaborate on the first point (about bool)? an example?

@mortada

I think add a tests for all of the .str methods (e.g. to test the return types issue), I think you will have to raise if return_type='frame' is defined

mortada · 2015-04-06T00:29:41Z

@jreback just added a check and unit test for the case of Index and return_type='frame'

@sinhrks I'm looking into the bool issue you referenced

shoyer · 2015-04-06T01:20:11Z

@jreback @mortada some str methods return booleans, e.g., str.isalpha. For series, it makes sense to return a As @sinhrks points out, we probably don't want to return a new Index object for these values, but rather a plain numpy array.

mortada · 2015-04-06T06:44:50Z

@shoyer, @sinhrks I see, indeed returning np.array in the boolean case seems a lot more useful.

@jreback an example would be:

idx = Index(['a1', 'a2', 'b1', 'b2'])
s = Series(range(4), index=idx)

If we return an np.array then an expression like

s[s.index.str.startswith('a')]

would work naturally, whereas if we return a boolean Index that's not very useful

mortada · 2015-04-06T06:48:30Z

Just added a check for the boolean Index case, it will now return np.array instead. Please take a look, thanks!

shoyer · 2015-04-06T07:33:53Z

doc/source/whatsnew/v0.16.1.txt

@@ -18,6 +18,7 @@ Enhancements
 ~~~~~~~~~~~~

 - Added ``StringMethods.capitalize()`` and ``swapcase`` which behave as the same as standard ``str`` (:issue:`9766`)
+- Added ``StringMethods`` (.str accessor) to ``Index`` (:issue:`9068`)


It would be nice to add an example or two here, both to show off the new feature and to illustrate the behavior with boolean output.

good idea, will do

jreback · 2015-04-06T11:08:16Z

pandas/core/strings.py

+            if isinstance(self.series, Index):
+                # if result is a boolean np.array, return the np.array
+                # instead of wrapping it into a boolean Index (GH 8875)
+                if result.dtype == bool:


use is_bool_dtype(result)

good catch I'll change this

jreback · 2015-04-06T11:11:31Z

@mortada yeh I think we need to do something for

Index(['foo_bar','bah_baz']).str.split('_',return_type='frame')

sinhrks · 2015-04-06T13:15:17Z

@mortada Thanks for quick action:)

jorisvandenbossche · 2015-04-06T13:45:11Z

Didn't look in detail into this, but a quick question: what is the return type? I suppose in general a new index? But what if the method returns multiple elements for each entry? (eg returning a frame for a series)?

mortada · 2015-04-06T17:30:19Z

@jorisvandenbossche the return type is Index unless the dtype is bool, in which case the return type will be np.array.

if the method returns multiple elements it will be an Index of lists, and won't be a DataFrame. If the user specifies return_type='frame' it will raise. Here's an example:

In [1]: from pandas import Index
In [2]: idx = Index(['a b c', 'd e', 'f'])

In [3]: idx.str.split()
Out[3]: Index([['a', 'b', 'c'], ['d', 'e'], ['f']], dtype='object')

In [4]: idx.str.split(return_type='frame')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-d22b631e0f0f> in <module>()
----> 1 idx.str.split(return_type='frame')

/Users/mortada_mehyar/code/github/pandas/pandas/core/strings.py in split(self, pat, n, return_type)
    987     @copy(str_split)
    988     def split(self, pat=None, n=-1, return_type='series'):
--> 989         result = str_split(self.series, pat, n=n, return_type=return_type)
    990         return self._wrap_result(result)
    991

/Users/mortada_mehyar/code/github/pandas/pandas/core/strings.py in str_split(arr, pat, n, return_type)
    652         raise ValueError("return_type must be {'series', 'frame'}")
    653     if return_type == 'frame' and isinstance(arr, Index):
--> 654         raise ValueError("return_type='frame' is not supported for string "
    655                          "methods on Index")
    656     if pat is None:

ValueError: return_type='frame' is not supported for string methods on Index

mortada · 2015-04-06T17:46:41Z

@shoyer added examples to doc
@jreback now using is_bool_dtype

please take a look, thanks.

jorisvandenbossche · 2015-04-07T23:00:37Z

By the way, I think this is a very nice feature, especially useful for cleaning up your column names (stripping whitespace at beginning or end -> typical problem you see on SO, replacing whitespace with _, convert to all lower case, ..) For the docs, maybe it would be useful to also focus on this and give an example where the column names are used (but maybe not needed for this PR)

jreback · 2015-04-08T00:15:43Z

@jorisvandenbossche I don't think its a problem if certain methods which have a return_type='frame' raise if you try to call them on an Index. The only other option is to actually return a frame as I think the purpose is what you mentioned above, to allow cleaning type ops on an Index, which you expect to get an Index.

Another option is to exclude the methods entirely from the Index (meaning ones that have a default return_type='frame')

mortada · 2015-04-08T01:49:27Z

@jreback I added reture_type='index' as a alias for 'series' as you suggested, please take a look

jreback · 2015-04-08T12:05:43Z

lgtm. @jorisvandenbossche

jorisvandenbossche · 2015-04-08T20:17:12Z

@mortada Are you sure you committed it? I still only see the frame/series options (or I am overlooking it completely ..)

mortada · 2015-04-08T21:14:59Z

@jorisvandenbossche oops I had to do another rebase and missed the last commit, thanks for catching!

mortada · 2015-04-09T00:06:05Z

@jorisvandenbossche it's fixed, please take a look thanks

jorisvandenbossche · 2015-04-09T13:43:39Z

pandas/core/strings.py

@@ -632,9 +632,10 @@ def str_split(arr, pat=None, n=None, return_type='series'):
    pat : string, default None
        String or regular expression to split on. If None, splits on whitespace
    n : int, default None (all)
-    return_type : {'series', 'frame'}, default 'series
+    return_type : {'series', 'index', 'frame'}, default 'series'


Maybe make this {'series'/'index', 'frame'} to make it more clear they are aliases?

jorisvandenbossche · 2015-04-09T13:51:12Z

The thing I am thinking now: won't it be confusing for users to get a Index back even if he/she supplies return_type='series' on a Index.str.split, or to get a series back with Series.str.split(.., return_type='index')?

I was also thinking of another option: an expand keyword (or another name), that indicates for False: give same dimension back (so for series/index keep it a series/index), and for True: expand series to dataframe.
This would then be a duplicate for return_type of course. But the return_type was only introduced in 0.15.1 (and for partition it is still in a PR), so if we want to change this: better now than later. Or has this ship sailed?

But maybe we shouldn't block this PR with this discussion and open a new issue for this. So I am OK with merging.

jreback · 2015-04-09T14:47:30Z

@jorisvandenbossche

how about we just do

return_type='same'|'expand' to satisfy this need? (and can be easily back-compat)

but let's do that in another issue / PR

mortada · 2015-04-10T02:53:16Z

@jreback @jorisvandenbossche I'm happy to do another PR for the return_type

jreback · 2015-04-10T13:50:11Z

ok, merging this, but @mortada I would appreciate another issue (and PR!) for the return_type issue.

ENH: add StringMethods (.str accessor) to Index, fixes #9068

jorisvandenbossche · 2015-04-10T14:00:14Z

@mortada really a nice improvement!

I will open an new issue to discuss the return_type thing

mortada · 2015-04-10T18:57:58Z

@jorisvandenbossche thanks!

Another thing you had mentioned was expanding the docs to cover some questions on SO about cleaning up column names. Could you please point me to some of those SO questions? I can add more examples to the docs.

jorisvandenbossche · 2015-04-11T10:57:43Z

@mortada I don't have a specific SO question on my mind, but you often see questions like: "why does df['my_col'] give a KeyError as it is in the columns?" with response -> "check df.columns, maybe you have starting or trailing spaces in the names" (something you don't easily see in the console output, you can only detect it through the alignment)

So for me, this are typical ways to clean up column names:

remove starting/trailing whitespace (eg leftover of reading it in)
convert all columns names to lower case (eg something typical you would want for writing it to certain databases/on certain platforms to not have to deal with case sensitivity issues)
remove alle whitespace in the column names, or convert all whitespaces to underscores ("my col" -> "my_col")

shoyer · 2015-04-11T22:06:40Z

@mortada very nice, thanks!

mortada force-pushed the index_str_methods branch from 9bab07a to 00e6aea Compare April 1, 2015 21:38

shoyer reviewed Apr 2, 2015
View reviewed changes

mortada force-pushed the index_str_methods branch 2 times, most recently from 4b9468b to 78b68cb Compare April 2, 2015 19:13

mortada reviewed Apr 2, 2015
View reviewed changes

mortada force-pushed the index_str_methods branch from 78b68cb to e6b0a4b Compare April 6, 2015 00:18

mortada force-pushed the index_str_methods branch from e6b0a4b to 23cf6b3 Compare April 6, 2015 06:46

shoyer reviewed Apr 6, 2015
View reviewed changes

jreback added Indexing Related to indexing on series/frames, not to indexes themselves API Design Strings String extension data type and string data labels Apr 6, 2015

jreback added this to the 0.16.1 milestone Apr 6, 2015

jreback reviewed Apr 6, 2015
View reviewed changes

mortada force-pushed the index_str_methods branch 2 times, most recently from 4b68a72 to 1b48868 Compare April 6, 2015 17:42

mortada force-pushed the index_str_methods branch from 1b48868 to 8968009 Compare April 8, 2015 01:01

mortada force-pushed the index_str_methods branch from 8968009 to 7d734fb Compare April 8, 2015 17:49

mortada force-pushed the index_str_methods branch from 7d734fb to ed77c72 Compare April 8, 2015 21:14

jorisvandenbossche reviewed Apr 9, 2015
View reviewed changes

This was referenced Apr 9, 2015

DOC/CLN: Revise StringMethods docs #9843

Merged

ENH: Add StringMethods.partition and rpartition #9773

Merged

ENH: add StringMethods (.str accessor) to Index, fixes pandas-dev#9068

f98bcb8

mortada force-pushed the index_str_methods branch from ed77c72 to f98bcb8 Compare April 10, 2015 07:28

jreback added a commit that referenced this pull request Apr 10, 2015

Merge pull request #9667 from mortada/index_str_methods

2734fff

ENH: add StringMethods (.str accessor) to Index, fixes #9068

jreback merged commit 2734fff into pandas-dev:master Apr 10, 2015

jorisvandenbossche mentioned this pull request Apr 10, 2015

API: return_type argument in StringMethods.split() #9847

Closed

mortada mentioned this pull request Apr 11, 2015

DOC: add more examples to StringMethods on Index #9858

Merged

sinhrks mentioned this pull request Apr 24, 2015

API: Index.str follow-ups (extract/get_dummies) #9980

Closed

mortada mentioned this pull request Apr 28, 2015

ENH: Index StringMethods should return MultiIndex when result dimension is more than one #10008

Closed

6 tasks

mortada deleted the index_str_methods branch April 29, 2015 16:56

ENH: add StringMethods (.str accessor) to Index, fixes #9068 #9667

ENH: add StringMethods (.str accessor) to Index, fixes #9068 #9667

Conversation

mortada commented Mar 17, 2015

mortada commented Apr 1, 2015

jreback commented Apr 1, 2015

jreback commented Apr 1, 2015

mortada commented Apr 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sinhrks commented Apr 5, 2015

jreback commented Apr 5, 2015

mortada commented Apr 6, 2015

shoyer commented Apr 6, 2015

mortada commented Apr 6, 2015

mortada commented Apr 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Apr 6, 2015

sinhrks commented Apr 6, 2015

jorisvandenbossche commented Apr 6, 2015

mortada commented Apr 6, 2015

mortada commented Apr 6, 2015

jorisvandenbossche commented Apr 7, 2015

jreback commented Apr 8, 2015

mortada commented Apr 8, 2015

jreback commented Apr 8, 2015

jorisvandenbossche commented Apr 8, 2015

mortada commented Apr 8, 2015

mortada commented Apr 9, 2015

Choose a reason for hiding this comment

jorisvandenbossche commented Apr 9, 2015

jreback commented Apr 9, 2015

mortada commented Apr 10, 2015

jreback commented Apr 10, 2015

jorisvandenbossche commented Apr 10, 2015

mortada commented Apr 10, 2015

jorisvandenbossche commented Apr 11, 2015

shoyer commented Apr 11, 2015