DEPR: Deprecate box kwarg for to_timedelta and to_datetime #24486

cbertinato · 2018-12-29T18:06:25Z

closes DEPR: box argument in to_datetime and to_timedelta #24416
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

There are still some tests that are failing on Travis, but they appear to be unrelated. I'll see what CI returns for the PR and address any tests that continue to fail that might be due to the changes here.

jreback

you will have to catch the warning in tests that currently use box

cbertinato · 2018-12-29T18:30:39Z

You mean assert that the warning is raised in those cases?

jreback · 2018-12-29T18:40:32Z

yes need to eliminate all warnings from the test suite - either catch them or remove the box parameter (prob the former is ok here)

jreback · 2018-12-29T18:40:57Z

doc/source/whatsnew/v0.24.0.rst

@@ -1152,6 +1152,7 @@ Deprecations
 - Passing a string alias like ``'datetime64[ns, UTC]'`` as the ``unit`` parameter to :class:`DatetimeTZDtype` is deprecated. Use :class:`DatetimeTZDtype.construct_from_string` instead (:issue:`23990`).
 - In :meth:`Series.where` with Categorical data, providing an ``other`` that is not present in the categories is deprecated. Convert the categorical to a different dtype or add the ``other`` to the categories first (:issue:`24077`).
 - :meth:`Series.clip_lower`, :meth:`Series.clip_upper`, :meth:`DataFrame.clip_lower` and :meth:`DataFrame.clip_upper` are deprecated and will be removed in a future version. Use ``Series.clip(lower=threshold)``, ``Series.clip(upper=threshold)`` and the equivalent ``DataFrame`` methods (:issue:`24203`)
+- The functions :func:`pandas.to_datetime` and :func:`pandas.to_timedelta` have deprecated the ``box`` keyword. Use :attr:`Series.values` and :meth:`Timestamp.to_datetime64`/:meth:`Timedelta.to_timedelta64` instead to get an ndarray of values or ``numpy.timestamp64``/``numpy.timedelta64``, respectively (:issue:`24416`).


recommend .to_numpy()

jreback · 2018-12-29T18:41:24Z

pandas/core/tools/timedeltas.py

@@ -40,6 +43,12 @@ def to_timedelta(arg, unit='ns', box=True, errors='raise'):
        - If True returns a Timedelta/TimedeltaIndex of the results.
        - If False returns a numpy.timedelta64 or numpy.darray of
          values of dtype timedelta64[ns].
+
+        .. deprecated:: 0.24.0
+            Use :attr:`Series.values` or :meth:`Timedelta.to_timedelta64`


here as well

mroeschke · 2018-12-29T19:24:59Z

doc/source/whatsnew/v0.24.0.rst

@@ -1152,6 +1152,7 @@ Deprecations
 - Passing a string alias like ``'datetime64[ns, UTC]'`` as the ``unit`` parameter to :class:`DatetimeTZDtype` is deprecated. Use :class:`DatetimeTZDtype.construct_from_string` instead (:issue:`23990`).
 - In :meth:`Series.where` with Categorical data, providing an ``other`` that is not present in the categories is deprecated. Convert the categorical to a different dtype or add the ``other`` to the categories first (:issue:`24077`).
 - :meth:`Series.clip_lower`, :meth:`Series.clip_upper`, :meth:`DataFrame.clip_lower` and :meth:`DataFrame.clip_upper` are deprecated and will be removed in a future version. Use ``Series.clip(lower=threshold)``, ``Series.clip(upper=threshold)`` and the equivalent ``DataFrame`` methods (:issue:`24203`)
+- The functions :func:`pandas.to_datetime` and :func:`pandas.to_timedelta` have deprecated the ``box`` keyword. Use :attr:`Series.values` and :meth:`Timestamp.to_datetime64`/:meth:`Timedelta.to_timedelta64` instead to get an ndarray of values or ``numpy.timestamp64``/``numpy.timedelta64``, respectively (:issue:`24416`).


numpy.timestamp64 --> numpy.datetime64

codecov · 2018-12-30T05:02:15Z

Codecov Report

Merging #24486 into master will decrease coverage by 49.25%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           master   #24486       +/-   ##
===========================================
- Coverage   92.31%   43.05%   -49.26%     
===========================================
  Files         166      166               
  Lines       52335    52339        +4     
===========================================
- Hits        48313    22535    -25778     
- Misses       4022    29804    +25782

Flag	Coverage Δ
#multiple	`?`
#single	`43.05% <100%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/tools/timedeltas.py	`67.92% <100%> (-30.12%)`	⬇️
pandas/core/tools/datetimes.py	`34.51% <100%> (-51.25%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/core/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
pandas/io/formats/html.py	`0% <0%> (-98.65%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-95.46%)`	⬇️
pandas/io/sas/sas7bdat.py	`0% <0%> (-91.17%)`	⬇️
... and 126 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aeff38d...5b84d9c. Read the comment docs.

codecov · 2018-12-30T05:02:15Z

Codecov Report

Merging #24486 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #24486      +/-   ##
==========================================
- Coverage   91.28%   91.27%   -0.01%     
==========================================
  Files         173      173              
  Lines       52965    52969       +4     
==========================================
+ Hits        48348    48350       +2     
- Misses       4617     4619       +2

Flag	Coverage Δ
#multiple	`89.85% <100%> (-0.01%)`	⬇️
#single	`41.74% <57.14%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/parsers.py	`95.34% <ø> (ø)`	⬆️
pandas/core/indexes/datetimelike.py	`98.52% <100%> (ø)`	⬆️
pandas/core/tools/timedeltas.py	`98.14% <100%> (+0.07%)`	⬆️
pandas/core/tools/datetimes.py	`84.63% <100%> (-0.5%)`	⬇️
pandas/core/dtypes/cast.py	`88.16% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26d991f...4d6e0d3. Read the comment docs.

cbertinato · 2018-12-30T17:36:14Z

Would it make sense to transition all of the internal uses of box here as well?

jreback · 2018-12-30T19:54:10Z

Would it make sense to transition all of the internal uses of box here as well?

yep.

jreback · 2018-12-31T16:22:24Z

pandas/io/parsers.py

                    dayfirst=dayfirst,
                    errors='ignore',
                    infer_datetime_format=infer_datetime_format
-                )
+                ).to_pydatetime()


this is a very lower performance option, rather just do

tools.to_datetime(.....).to_numpy()

though you may actually be able to get away with not doing any conversion and just remove the box=, see if it passes.

Yeah. I had considered the latter, but wasn't sure whether there would be other consequences. I'll give it a try and see if it passes. The test fails with .to_numpy() because we lose the tzinfo.

jreback · 2018-12-31T16:23:39Z

pandas/tests/indexes/datetimes/test_tools.py

-        result = pd.to_datetime(dates, format=fmt, box=box)
-        expected = const(expected_dates)
-        tm.assert_equal(result, expected)
+        with catch_warnings():


don't do this,

use tm.assert_produces_warning

jreback · 2018-12-31T16:24:02Z

pandas/tests/indexes/datetimes/test_tools.py

-        result = to_datetime(arr, box=True)
-        assert result is arr
+        with catch_warnings():
+            simplefilter("ignore", FutureWarning)


same for the rest (in this case you can just remove the keyword entirely)

Got it. Will do.

pep8speaks · 2019-01-03T13:21:36Z

Hello @cbertinato! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-03-12 15:23:12 UTC

jreback · 2019-01-06T16:34:23Z

some lint issues and needs a rebase

pandas/core/dtypes/cast.py

jreback · 2019-01-06T16:39:33Z

pandas/core/indexes/datetimelike.py

@@ -232,7 +232,12 @@ def asobject(self):
        return self.astype(object)

    def _convert_tolerance(self, tolerance, target):
-        tolerance = np.asarray(to_timedelta(tolerance, box=False))
+        tolerance = to_timedelta(tolerance)


see #24653 where this could be much simpler

ideally would do that as a pre-cursor PR

jreback · 2019-01-06T16:40:15Z

pandas/io/parsers.py

@@ -3120,18 +3122,28 @@ def _get_lines(self, rows=None):
 def _make_date_converter(date_parser=None, dayfirst=False,
                         infer_datetime_format=False):
    def converter(*date_cols):
+        from pandas.core.dtypes.common import is_datetime64_dtype


can be imported at the top

pandas/io/parsers.py

jreback · 2019-01-06T16:43:18Z

pandas/tests/indexes/datetimes/test_tools.py

-                      errors='raise')
+            # Assuming all datetimes are in bounds, to_datetime() returns
+            # an array that is equal to Timestamp() parsing
+            tm.assert_numpy_array_equal(


you can just remove the box arg here

jreback · 2019-01-06T16:43:41Z

pandas/tests/indexes/datetimes/test_tools.py

+            # A list of datetimes where the last one is out of bounds
+            dts_with_oob = dts + [np.datetime64('9999-01-01')]
+
+            pytest.raises(ValueError, pd.to_datetime, dts_with_oob,


use the form

with pytest.raises(...): ...

jreback · 2019-01-06T16:43:54Z

pandas/tests/indexes/datetimes/test_tools.py

+            pytest.raises(ValueError, pd.to_datetime, dts_with_oob,
+                          errors='raise')
+
+        with tm.assert_produces_warning(FutureWarning):


remove box arg

jreback · 2019-01-06T16:44:05Z

pandas/tests/indexes/datetimes/test_tools.py

-                [dt.item() for dt in dts_with_oob],
-                dtype='O'
+
+        with tm.assert_produces_warning(FutureWarning):


remove box arg

jreback

should do #24653 as a pre-cursor here to make this simpler

pandas/core/dtypes/cast.py

pandas/io/parsers.py

jreback · 2019-02-27T14:34:13Z

pandas/core/tools/datetimes.py

@@ -444,6 +447,12 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,

        - If True returns a DatetimeIndex or Index-like object
        - If False returns ndarray of values.
+
+        .. deprecated:: 0.24.0


jreback · 2019-02-27T14:34:26Z

pandas/core/tools/timedeltas.py

@@ -40,6 +43,12 @@ def to_timedelta(arg, unit='ns', box=True, errors='raise'):
        - If True returns a Timedelta/TimedeltaIndex of the results.
        - If False returns a numpy.timedelta64 or numpy.darray of
          values of dtype timedelta64[ns].
+
+        .. deprecated:: 0.24.0


pandas/tests/indexes/timedeltas/test_tools.py

jreback · 2019-03-10T22:51:50Z

@cbertinato this is looking close, can you see if get passing

cbertinato · 2019-03-11T12:54:49Z

All green!

WillAyd · 2019-03-11T17:28:45Z

doc/source/whatsnew/v0.25.0.rst

@@ -94,6 +94,7 @@ Deprecations
 ~~~~~~~~~~~~

 - Deprecated the `M (months)` and `Y (year)` `units` parameter of :func: `pandas.to_timedelta`, :func: `pandas.Timedelta` and :func: `pandas.TimedeltaIndex` (:issue:`16344`)
+- The functions :func:`pandas.to_datetime` and :func:`pandas.to_timedelta` have deprecated the ``box`` keyword. Use :attr:`Series.values` and :meth:`Timestamp.to_datetime64`/:meth:`Timedelta.to_timedelta64` instead to get an ndarray of values or ``numpy.timestamp64``/``numpy.timedelta64``, respectively (:issue:`24416`)


Should update .values to .to_numpy()

Ah, yes indeed. Updated.

cbertinato · 2019-03-13T13:22:13Z

All green.

jreback · 2019-03-13T15:42:13Z

thanks @cbertinato nice patch!

* master: (22 commits) Fixturize tests/frame/test_operators.py (pandas-dev#25641) Update ValueError message in corr (pandas-dev#25729) DOC: fix some grammar and inconsistency issues in the User Guide (pandas-dev#25728) ENH: Add public start, stop, and step attributes to RangeIndex (pandas-dev#25720) Make Rolling.apply documentation clearer (pandas-dev#25712) pandas-dev#25707 - Fixed flakiness in stata write test (pandas-dev#25714) Json normalize nan support (pandas-dev#25619) TST: resolve issues with test_constructor_dtype_datetime64 (pandas-dev#24868) DEPR: Deprecate box kwarg for to_timedelta and to_datetime (pandas-dev#24486) BUG: Preserve name in DatetimeIndex.snap (pandas-dev#25585) Fix concat not respecting order of OrderedDict (pandas-dev#25224) CLN: remove pandas.core.categorical (pandas-dev#25655) TST/CLN: Remove more Panel tests (pandas-dev#25675) Pinned pycodestyle (pandas-dev#25701) DOC: update date of 0.24.2 release notes (pandas-dev#25699) BUG: Fix error in replace with strings that are large numbers (pandas-dev#25616) (pandas-dev#25644) BUG: fix usage of na_sentinel with sort=True in factorize() (pandas-dev#25592) BUG: Fix to_string output when using header (pandas-dev#16718) (pandas-dev#25602) CLN: Remove unused test code (pandas-dev#25670) CLN: remove Panel from concat error message (pandas-dev#25676) ... # Conflicts: # doc/source/whatsnew/v0.25.0.rst

jreback requested changes Dec 29, 2018

View reviewed changes

mroeschke reviewed Dec 29, 2018

View reviewed changes

gfyoung added Datetime Datetime data dtype Timedelta Timedelta data type Deprecate Functionality to remove in pandas labels Dec 30, 2018

jreback requested changes Dec 31, 2018

View reviewed changes

jreback requested changes Jan 6, 2019

View reviewed changes

cbertinato force-pushed the 24416-depr-box branch from 51be01f to 2f81938 Compare January 18, 2019 23:41

jreback requested changes Jan 19, 2019

View reviewed changes

pandas/core/dtypes/cast.py Show resolved Hide resolved

pandas/core/dtypes/cast.py Show resolved Hide resolved

pandas/io/parsers.py Show resolved Hide resolved

cbertinato force-pushed the 24416-depr-box branch 2 times, most recently from d62170f to ab05d74 Compare February 26, 2019 13:45

jreback requested changes Feb 27, 2019

View reviewed changes

cbertinato force-pushed the 24416-depr-box branch from ab05d74 to f388126 Compare March 8, 2019 15:34

cbertinato force-pushed the 24416-depr-box branch 3 times, most recently from 63717f5 to 427345d Compare March 11, 2019 01:26

WillAyd requested changes Mar 11, 2019

View reviewed changes

cbertinato force-pushed the 24416-depr-box branch from 427345d to 7893630 Compare March 11, 2019 17:44

DEPR: Deprecate box kwarg for to_timedelta and to_datetime

4d6e0d3

cbertinato force-pushed the 24416-depr-box branch from 7893630 to 4d6e0d3 Compare March 12, 2019 15:22

jreback added this to the 0.25.0 milestone Mar 13, 2019

jsexauer mentioned this pull request Mar 13, 2019

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

jreback approved these changes Mar 13, 2019

View reviewed changes

jreback merged commit 58a4151 into pandas-dev:master Mar 13, 2019

jbrockmendel mentioned this pull request Dec 6, 2019

DEPR: box arg in to_datetime #30111

Merged

jreback mentioned this pull request Dec 6, 2019

DEPR: deprecations log for removed issues #13777

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPR: Deprecate box kwarg for to_timedelta and to_datetime #24486

DEPR: Deprecate box kwarg for to_timedelta and to_datetime #24486

cbertinato commented Dec 29, 2018

jreback left a comment

cbertinato commented Dec 29, 2018

jreback commented Dec 29, 2018

jreback Dec 29, 2018

jreback Dec 29, 2018

mroeschke Dec 29, 2018

codecov bot commented Dec 30, 2018

codecov bot commented Dec 30, 2018 •

edited

Loading

cbertinato commented Dec 30, 2018

jreback commented Dec 30, 2018

jreback Dec 31, 2018

jreback Dec 31, 2018

cbertinato Dec 31, 2018

jreback Dec 31, 2018

jreback Dec 31, 2018

cbertinato Dec 31, 2018

pep8speaks commented Jan 3, 2019 •

edited

Loading

jreback commented Jan 6, 2019

jreback Jan 6, 2019

jreback Jan 6, 2019

jreback Jan 6, 2019

jreback Jan 6, 2019

jreback Jan 6, 2019

jreback Jan 6, 2019

jreback Jan 6, 2019

jreback left a comment

jreback Feb 27, 2019

jreback Feb 27, 2019

jreback commented Mar 10, 2019

cbertinato commented Mar 11, 2019

WillAyd Mar 11, 2019

cbertinato Mar 11, 2019

cbertinato commented Mar 13, 2019

jreback commented Mar 13, 2019

DEPR: Deprecate box kwarg for to_timedelta and to_datetime #24486

DEPR: Deprecate box kwarg for to_timedelta and to_datetime #24486

Conversation

cbertinato commented Dec 29, 2018

jreback left a comment

Choose a reason for hiding this comment

cbertinato commented Dec 29, 2018

jreback commented Dec 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 30, 2018

Codecov Report

codecov bot commented Dec 30, 2018 • edited Loading

Codecov Report

cbertinato commented Dec 30, 2018

jreback commented Dec 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Jan 3, 2019 • edited Loading

Comment last updated at 2019-03-12 15:23:12 UTC

jreback commented Jan 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Mar 10, 2019

cbertinato commented Mar 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbertinato commented Mar 13, 2019

jreback commented Mar 13, 2019

codecov bot commented Dec 30, 2018 •

edited

Loading

pep8speaks commented Jan 3, 2019 •

edited

Loading