ENH: Allow usecols to accept callable (GH14154) #14234

WillAyd · 2016-09-16T04:08:04Z

[ X] closes ENH: allow usecols to be case insensitive #14154
[ X] tests added / passed
[ X] passes git diff upstream/master | flake8 --diff
[X ] whatsnew entry

codecov-io · 2016-09-16T04:35:50Z

Current coverage is 85.28% (diff: 100%)

Merging #14234 into master will increase coverage by <.01%

@@             master     #14234   diff @@
==========================================
  Files           144        144          
  Lines         50953      50963    +10   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43452      43462    +10   
  Misses         7501       7501          
  Partials          0          0

Powered by Codecov. Last update 4378f82...9b0d112

jreback · 2016-09-16T10:28:35Z

pandas/io/parsers.py


    if usecols is not None:
+        if callable(usecols):
+            return usecols


you can just in-line _evaluate_usecols here, then change _vaidate_usecols to return the usecols and set it

Took a look but don't think that would work. The problem seems to be the names argument that's a requirement for _evaluate_usecols.

Unless I'm missing something I think I'd have to modify _validate_usecols to accept names as an argument, but it's currently being called before the names are even set (unless explicitly passed as an argument to the read function)

jreback · 2016-09-16T10:29:27Z

pandas/io/parsers.py

-usecols : array-like, default None
-    Return a subset of the columns. All elements in this array must either
+usecols : array-like or callable, default None
+    Return a subset of the columns. If array-like, all elements must either
    be positional (i.e. integer indices into the document columns) or strings
    that correspond to column names provided either by the user in `names` or
    inferred from the document header row(s). For example, a valid `usecols`


can you add a mini-example of a callable

jreback · 2016-11-25T14:37:18Z

can you rebase, move whatsnew notes to 0.20.0, and add an example as indicated

WillAyd · 2016-11-25T22:45:25Z

Requested changes should have been made - let me know if there's anything else you would like to tweak for this enhancement

jreback · 2016-11-25T22:48:43Z

doc/source/whatsnew/v0.20.0.txt

@@ -31,6 +31,7 @@ Other enhancements
 ^^^^^^^^^^^^^^^^^^

 - ``pd.read_excel`` now preserves sheet order when using ``sheetname=None`` (:issue:`9930`)
+- The ``usecols`` argument now accepts a callable function as a value  (:issue:`14154`)


The usecols argument in pd.read_csv

jreback · 2016-11-25T22:49:06Z

pandas/io/parsers.py

+    if callable(usecols):
+        return set([i for i, name in enumerate(names)
+                    if com._apply_if_callable(usecols, name)])
+    else:


just return is fine here (no else)

jreback · 2016-11-25T22:49:19Z

pandas/io/parsers.py

-    msg = ("The elements of 'usecols' must "
-           "either be all strings, all unicode, or all integers")
+    msg = ("'usecols' must either be all strings, all unicode, "
+           "all integers or callable")


jreback · 2016-11-25T22:51:16Z

pandas/io/tests/parser/usecols.py

-               "either be all strings, all unicode, or all integers")
+
+        msg = ("'usecols' must either be all strings, all unicode, "
+               "all integers or callable")


jreback · 2016-11-25T22:51:20Z

pandas/io/tests/parser/usecols.py

-        msg = ("The elements of 'usecols' must "
-               "either be all strings, all unicode, or all integers")
+        msg = ("'usecols' must either be all strings, all unicode, "
+               "all integers or callable")


jreback · 2016-11-25T22:51:33Z

pandas/io/tests/parser/usecols.py

@@ -366,3 +367,25 @@ def test_np_array_usecols(self):
        expected = DataFrame([[1, 2]], columns=usecols)
        result = self.read_csv(StringIO(data), usecols=usecols)
        tm.assert_frame_equal(result, expected)
+
+    def test_callable_usecols(self):
+        s = '''AaA,bBb,CCC,ddd


can you add the issue number as a comment

jreback · 2016-11-25T22:54:26Z

pandas/parser.pyx

@@ -437,7 +440,11 @@ cdef class TextReader:
        # suboptimal
        if usecols is not None:
            self.has_usecols = 1
-            self.usecols = set(usecols)
+            if callable(usecols):
+                self.callable_usecols = 1


I would not use this additional variable and instead just do a callable(self.usecalls) I think its more obvious

jreback · 2016-11-25T22:54:45Z

@WillAyd thanks. couple more pretty minor comments. otherwise lgtm.

WillAyd · 2016-11-27T17:51:49Z

All of your requested changes should be accounted for. Let me know if anything else comes to mind.

jorisvandenbossche

Apart from the docstring and whatsnew, can you also update the IO docs in two places:

overview of keywords: http://pandas.pydata.org/pandas-docs/stable/io.html#column-and-index-locations-and-names
section on usecols: http://pandas.pydata.org/pandas-docs/stable/io.html#filtering-columns-usecols (I would add an example of a callable there)

jorisvandenbossche · 2016-11-28T09:10:27Z

pandas/io/parsers.py

-    results in much faster parsing time and lower memory usage.
+    parameter would be [0, 1, 2], ['foo', 'bar', 'baz'] or lambda x: x.upper()
+    in ['AAA', 'BBB', 'DDD']. Using this parameter results in much faster
+    parsing time and lower memory usage.


Can you add before the example some explanation how the callable is used? (I suppose it is called on each individual column name and should return True or False?)

And maybe also use `` around the code example to better indicate that the "in ['AAA', 'BBB', 'DDD']" still belongs with the "lambda x: x.upper()"

jorisvandenbossche · 2016-11-28T09:15:03Z

pandas/io/parsers.py

+def _evaluate_usecols(usecols, names):
+    if callable(usecols):
+        return set([i for i, name in enumerate(names)
+                    if com._apply_if_callable(usecols, name)])


I think the com._apply_if_callable is not needed anymore as you already did if callable(usecols)? (apply_if_callable does exactly the same under the hood)

jorisvandenbossche · 2016-11-28T09:29:50Z

pandas/io/tests/parser/usecols.py

+
+        df = self.read_csv(StringIO(s), usecols=lambda x:
+                           x.upper() in ['AAA', 'BBB', 'DDD'])
+        tm.assert_frame_equal(df, expected)


Can you add a test that lambda x: False return an empty frame? (similar to the test_empty_usecols above)

jorisvandenbossche · 2016-11-28T09:32:56Z

pandas/parser.pyx

-                                             name in self.usecols):
+                usecols = set()
+                if callable(self.usecols):
+                    if com._apply_if_callable(self.usecols, name):


same here, com._apply_if_callable is not needed I think

WillAyd · 2016-11-30T00:02:45Z

I noticed that the Travis-CI build failed for OS X. I didn't see anything useful in the log and checking their website it appears that the failure may have stemmed from issues on their end and not necessarily from the code. Do you have a preferred method to restart that build? Should I close / re-open the PR or do you have another way you'd like to restart that test?

jreback · 2016-11-30T00:04:05Z

I restarted the job.

jreback · 2016-12-04T23:46:41Z

doc/source/io.rst

-  parameter would be [0, 1, 2] or ['foo', 'bar', 'baz']. Using this parameter
-  results in much faster parsing time and lower memory usage.
+  inferred from the document header row(s). For example, a valid array-like
+  `usecols` parameter would be [0, 1, 2] or ['foo', 'bar', 'baz']. If callable,


since this is docs, you can format this to read easier. (use an ipython block), blank line

jreback · 2016-12-04T23:47:44Z

pandas/io/parsers.py

+    inferred from the document header row(s). For example, a valid array-like
+    `usecols` parameter would be [0, 1, 2] or ['foo', 'bar', 'baz']. If
+    callable, the callable function will be evaluated against the column names,
+    returning names where the callable function evaluates to True. An example


you can't format this (like you can an .rst), but maybe a blank line or some bullet points?

jreback · 2016-12-04T23:47:58Z

pandas/io/parsers.py

@@ -976,17 +980,26 @@ def _is_index_col(col):
    return col is not None and col is not False


+def _evaluate_usecols(usecols, names):
+    if callable(usecols):


can you give a mini-doc string here

jreback · 2016-12-04T23:48:57Z

pandas/io/tests/parser/usecols.py

+                           x.upper() in ['AAA', 'BBB', 'DDD'])
+        tm.assert_frame_equal(df, expected)
+
+    def test_callable_usecols_with_false_callable(self):


you can put this test with the above one, just put a 1-line comment to separate them (I think they use the same input as well), to avoid duplication

jreback · 2016-12-04T23:49:30Z

@WillAyd just a couple of minor comments. ping when green.

WillAyd · 2016-12-06T04:56:15Z

@jreback pushed some edits to cover your latest comments. Let me know if there's anything else you'd like on this

jreback · 2016-12-06T11:29:16Z

lgtm. ping on green.

jorisvandenbossche · 2016-12-06T11:38:20Z

@WillAyd Thanks!

jreback requested changes Sep 16, 2016

View reviewed changes

jreback added Enhancement IO CSV read_csv, to_csv labels Sep 16, 2016

jreback requested changes Sep 16, 2016

View reviewed changes

jreback mentioned this pull request Oct 24, 2016

Strip whitespace from column names when usecols in read_csv #14480

Closed

WillAyd force-pushed the callable-usecols branch from 5f17d1d to ddbdb7a Compare November 25, 2016 22:44

jreback reviewed Nov 25, 2016

View reviewed changes

WillAyd force-pushed the callable-usecols branch 2 times, most recently from 1986920 to 5c933db Compare November 26, 2016 23:29

jorisvandenbossche reviewed Nov 28, 2016

View reviewed changes

jorisvandenbossche added this to the 0.20.0 milestone Nov 28, 2016

WillAyd force-pushed the callable-usecols branch from 5c933db to a1a05ca Compare November 29, 2016 03:27

jreback reviewed Dec 4, 2016

View reviewed changes

jreback approved these changes Dec 4, 2016

View reviewed changes

ENH: Allow usecols to accept callable (GH14154)

9b0d112

WillAyd force-pushed the callable-usecols branch from a1a05ca to 9b0d112 Compare December 6, 2016 04:48

jorisvandenbossche approved these changes Dec 6, 2016

View reviewed changes

jorisvandenbossche merged commit 8cdc09e into pandas-dev:master Dec 6, 2016

GGordonGordon mentioned this pull request Jan 2, 2017

ERR: usecols fails to raise error if column doesn't exist but is the same length as headers #14671

Closed

WillAyd deleted the callable-usecols branch December 12, 2017 15:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Allow usecols to accept callable (GH14154) #14234

ENH: Allow usecols to accept callable (GH14154) #14234

WillAyd commented Sep 16, 2016

codecov-io commented Sep 16, 2016 •

edited

Loading

jreback Sep 16, 2016

WillAyd Sep 16, 2016

jreback Sep 16, 2016

jreback commented Nov 25, 2016

WillAyd commented Nov 25, 2016

jreback Nov 25, 2016

jreback Nov 25, 2016

jreback Nov 25, 2016

jreback Nov 25, 2016

jreback Nov 25, 2016

jreback Nov 25, 2016

jreback Nov 25, 2016

jreback commented Nov 25, 2016

WillAyd commented Nov 27, 2016

jorisvandenbossche left a comment

jorisvandenbossche Nov 28, 2016

jorisvandenbossche Nov 28, 2016

jorisvandenbossche Nov 28, 2016

jorisvandenbossche Nov 28, 2016

WillAyd commented Nov 30, 2016

jreback commented Nov 30, 2016

jreback Dec 4, 2016

jreback Dec 4, 2016

jreback Dec 4, 2016

jreback Dec 4, 2016

jreback commented Dec 4, 2016

WillAyd commented Dec 6, 2016

jreback commented Dec 6, 2016

jorisvandenbossche commented Dec 6, 2016

ENH: Allow usecols to accept callable (GH14154) #14234

ENH: Allow usecols to accept callable (GH14154) #14234

Conversation

WillAyd commented Sep 16, 2016

codecov-io commented Sep 16, 2016 • edited Loading

Current coverage is 85.28% (diff: 100%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 25, 2016

WillAyd commented Nov 25, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 25, 2016

WillAyd commented Nov 27, 2016

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Nov 30, 2016

jreback commented Nov 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 4, 2016

WillAyd commented Dec 6, 2016

jreback commented Dec 6, 2016

jorisvandenbossche commented Dec 6, 2016

codecov-io commented Sep 16, 2016 •

edited

Loading