API: multi-line, not inplace eval #11149

chris-b1 · 2015-09-19T18:45:31Z

closes #9297
closes #8664

This is just a proof of concept at this point - the idea is to allow multiple assignments within an eval block, and also include an inplace argument (currently default True to match old behavior, although maybe that should be changed) for chaining.

In [15]: df = pd.DataFrame({'a': np.linspace(0, 100), 'b': range(50)})

In [16]: df.eval("""
    ...: c = a * 2
    ...: d = c / 2.0
    ...: e = a + b + c""")

In [17]: df.head()
Out[17]: 
        a  b          c         d          e
0  0.000000  0   0.000000  0.000000   0.000000
1  2.040816  1   4.081633  2.040816   7.122449
2  4.081633  2   8.163265  4.081633  14.244898
3  6.122449  3  12.244898  6.122449  21.367347
4  8.163265  4  16.326531  8.163265  28.489796

In [18]: df.eval("""
    ...: f = e * 2
    ...: g = f * 1.5""", inplace=False).head()
Out[18]: 
        a  b          c         d          e          f          g
0  0.000000  0   0.000000  0.000000   0.000000   0.000000   0.000000
1  2.040816  1   4.081633  2.040816   7.122449  14.244898  21.367347
2  4.081633  2   8.163265  4.081633  14.244898  28.489796  42.734694
3  6.122449  3  12.244898  6.122449  21.367347  42.734694  64.102041
4  8.163265  4  16.326531  8.163265  28.489796  56.979592  85.469388

jreback · 2015-10-08T11:10:48Z

pandas/computation/tests/test_eval.py

@@ -1256,6 +1256,44 @@ def f():
        expected['c'] = expected['a'] + expected['b']
        assert_frame_equal(df, expected)

+    def test_multi_line_expression(self):
+        df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})


add the issue as a comment

jreback · 2015-10-08T11:15:45Z

@chris-b1 this looks reasonable. I think that we should change the inplace to False by default. That obviously is a breaking change. So what I think we could do is this:

make inplace=None the default
detect if an assignment is being made. if inplace is None, then show a deprecation warning.
this implicity makes inplace=None -> inplace=False for non-assignments (which is what it currently does, and is quite reasonable). To get inplace then assignment you would have to be explict about passing inplace=True, which is what we would want.

yes?

jreback · 2015-10-08T11:16:04Z

cc @shoyer

shoyer · 2015-10-08T17:52:06Z

Yes, this looks great!

I agree that we should change the default behavior to inplace=True in a version or two. I think @jreback has the right idea on the migration path.

jreback · 2015-11-10T01:25:51Z

@chris-b1 going to mark this for 0.18.0 as putting the deprecation warning in-place (pun intended)

jreback · 2015-12-06T19:10:19Z

@chris-b1 want to update / add in deprecation warning as discussed

chris-b1 · 2015-12-06T22:10:49Z

@jreback - this is updated - added a whatsnew and deprecation warnings.

jreback · 2015-12-07T11:37:28Z

pandas/computation/eval.py

+
+        WARNING: inplace=None currently falls back to to True, but
+        in a future version, will default to False.  Use inplace=True
+        explicitly rather than relying on the default.


let's add .assign in the See also (as .assign is basically what df.eval('c=a+b',inplace=False) IS)

jreback · 2015-12-07T11:39:02Z

pls review the eval docs for any updates (in enhancingperf.rst)

jreback · 2015-12-07T11:39:36Z

does this handle #8664 ?

chris-b1 · 2015-12-09T03:13:30Z

@jreback
Updated the enhancingperf docs, and added a fix/test for #8664

jreback · 2015-12-09T15:12:56Z

doc/source/enhancingperf.rst

+identifier.
+
+The ``inplace`` keyword determines whether this assignment will performed
+on the original ``DataFrame`` or return a copy with the new column.


add a versionadded tag for inplace

jreback · 2015-12-16T15:45:36Z

pandas/core/frame.py

@@ -2048,6 +2048,9 @@ def query(self, expr, **kwargs):
            The query string to evaluate.  You can refer to variables
            in the environment by prefixing them with an '@' character like
            ``@a + b``.
+        inplace : bool


add a versionadded

jreback · 2015-12-16T15:47:30Z

lgtm. couple of doc changes. squash & ping when green.

chris-b1 · 2016-01-03T23:54:32Z

Updated for your doc comment and PEP8 - nothing major but did have to add a couple # noqa where a name was only used in an eval block - is that the right approach?

jreback · 2016-01-04T02:17:23Z

merged via 99c2d93

thanks for the nice changes!

yeh the # noqa is fine (I took one out, but otherwise ok).

Deprecated back in 0.18.0 xref pandas-devgh-11149

Deprecated back in 0.18.0 xref pandas-devgh-11149 [ci skip]

Deprecated in 0.18.0. xref pandas-devgh-11149. Also patches bug where we were improperly handling the inplace=False condition, as we were assuming that target input was non-None when that wasn't necessarily enforced.

pandas-dev/pandas#11149

* COMPAT: pandas TimeGrouper xref pandas-dev/pandas#16747 * COMPAT: For pandas 0.21 CategoricalDtype * COMPAT: For pandas 0.21.0 HTML repr changes pandas-dev/pandas#16879 * COMPAT: For pandas 0.21.0 numexpr import pandas-dev/pandas#17710 * COMPAT: Default for inplace is now False in eval pandas-dev/pandas#11149

jreback added the API Design label Sep 21, 2015

jreback added this to the Next Major Release milestone Oct 8, 2015

jreback reviewed Oct 8, 2015
View reviewed changes

chris-b1 force-pushed the eval-inplace branch from dd580c1 to ba84151 Compare October 12, 2015 00:15

jreback modified the milestones: 0.18.0, Next Major Release Nov 10, 2015

chris-b1 force-pushed the eval-inplace branch from ba84151 to 15a423e Compare December 6, 2015 22:10

chris-b1 changed the title ~~Experimental/WIP: multi-line, not inplace eval~~ API: multi-line, not inplace eval Dec 6, 2015

jreback reviewed Dec 7, 2015
View reviewed changes

jreback reviewed Dec 9, 2015
View reviewed changes

jreback mentioned this pull request Dec 9, 2015

BUG: query modifies the frame when you compare with = #8828

Closed

chris-b1 force-pushed the eval-inplace branch from f5cd30b to 765b855 Compare December 13, 2015 19:24

jreback reviewed Dec 16, 2015
View reviewed changes

chris-b1 force-pushed the eval-inplace branch from 3f0e363 to 3465696 Compare January 3, 2016 20:03

jreback closed this Jan 4, 2016

This was referenced Jan 4, 2016

API: DataFrame.eval, inplace option #9297

Closed

BUG: query modifies the frame when you compare with = #8664

Closed

ENH: str.extractall for several matches #11386

Closed

chris-b1 deleted the eval-inplace branch January 15, 2016 01:02

gfyoung added a commit to forking-repos/pandas that referenced this pull request Jun 20, 2017

MAINT: Change default of inplace to False in pd.eval

5b38b03

Deprecated back in 0.18.0 xref pandas-devgh-11149

gfyoung mentioned this pull request Jun 20, 2017

BUG/MAINT: Change default of inplace to False in pd.eval #16732

Merged

gfyoung added a commit to forking-repos/pandas that referenced this pull request Jun 20, 2017

MAINT: Default inplace to False in pd.eval

bf6fbe5

Deprecated back in 0.18.0 xref pandas-devgh-11149 [ci skip]

gfyoung added a commit to forking-repos/pandas that referenced this pull request Jun 20, 2017

MAINT: Default inplace to False in pd.eval

b7cee03

Deprecated back in 0.18.0 xref pandas-devgh-11149 [ci skip]

gfyoung added a commit to forking-repos/pandas that referenced this pull request Jun 20, 2017

MAINT: Default inplace to False in pd.eval

d52e95d

Deprecated back in 0.18.0 xref pandas-devgh-11149 [ci skip]

jreback mentioned this pull request Jul 6, 2017

DEPR: deprecations log for removed issues #13777

Closed

TomAugspurger added a commit to TomAugspurger/dask that referenced this pull request Oct 5, 2017

COMPAT: Default for inplace is now False in eval

286e2aa

pandas-dev/pandas#11149

TomAugspurger added a commit to TomAugspurger/dask that referenced this pull request Oct 5, 2017

COMPAT: Default for inplace is now False in eval

564f46a

pandas-dev/pandas#11149

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: multi-line, not inplace eval #11149

API: multi-line, not inplace eval #11149

chris-b1 commented Sep 19, 2015

jreback Oct 8, 2015

jreback commented Oct 8, 2015

jreback commented Oct 8, 2015

shoyer commented Oct 8, 2015

jreback commented Nov 10, 2015

jreback commented Dec 6, 2015

chris-b1 commented Dec 6, 2015

jreback Dec 7, 2015

jreback commented Dec 7, 2015

jreback commented Dec 7, 2015

chris-b1 commented Dec 9, 2015

jreback Dec 9, 2015

jreback Dec 16, 2015

jreback commented Dec 16, 2015

chris-b1 commented Jan 3, 2016

jreback commented Jan 4, 2016

API: multi-line, not inplace eval #11149

API: multi-line, not inplace eval #11149

Conversation

chris-b1 commented Sep 19, 2015

jreback Oct 8, 2015

Choose a reason for hiding this comment

jreback commented Oct 8, 2015

jreback commented Oct 8, 2015

shoyer commented Oct 8, 2015

jreback commented Nov 10, 2015

jreback commented Dec 6, 2015

chris-b1 commented Dec 6, 2015

jreback Dec 7, 2015

Choose a reason for hiding this comment

jreback commented Dec 7, 2015

jreback commented Dec 7, 2015

chris-b1 commented Dec 9, 2015

jreback Dec 9, 2015

Choose a reason for hiding this comment

jreback Dec 16, 2015

Choose a reason for hiding this comment

jreback commented Dec 16, 2015

chris-b1 commented Jan 3, 2016

jreback commented Jan 4, 2016