Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert change to comparison op with datetime.date objects #21361

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion doc/source/whatsnew/v0.23.1.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,52 @@ and bug fixes. We recommend that all users upgrade to this version.
:local:
:backlinks: none


.. _whatsnew_0231.fixed_regressions:

Fixed Regressions
~~~~~~~~~~~~~~~~~

**Comparing Series with datetime.date**

We've reverted a 0.23.0 change to comparing a :class:`Series` holding datetimes and a ``datetime.date`` object (:issue:`21152`).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use date=datetime64[ns]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh was just saying that users talk about dtypes, rather than Series holding datetimes. no big deal really.

In pandas 0.22 and earlier, comparing a Series holding datetimes and ``datetime.date`` objects would coerce the ``datetime.date`` to a datetime before comapring.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use double back ticks around Series (or :class`Series`

This was inconsistent with Python, NumPy, and :class:`DatetimeIndex`, which never consider a datetime and ``datetime.date`` equal.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should double backtick on Python, NumPy


In 0.23.0, we unified operations between DatetimeIndex and Series, and in the process changed comparisons between a Series of datetimes and ``datetime.date`` without warning.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

double backticks on DTI & Series


We've temporarily restored the 0.22.0 behavior, so datetimes and dates may again compare equal, but restore the 0.23.0 behavior in a future release.

To summarize, here's the behavior in 0.22.0, 0.23.0, 0.23.1:

.. code-block:: python

# 0.22.0... Silently coerce the datetime.date
>>> Series(pd.date_range('2017', periods=2)) == datetime.date(2017, 1, 1)
0 True
1 False
dtype: bool

# 0.23.0... Do not coerce the datetime.date
>>> Series(pd.date_range('2017', periods=2)) == datetime.date(2017, 1, 1)
0 False
1 False
dtype: bool

# 0.23.1... Coerce the datetime.date with a warning
>>> Series(pd.date_range('2017', periods=2)) == datetime.date(2017, 1, 1)
/bin/python:1: FutureWarning: Comparing Series of datetimes with 'datetime.date'. Currently, the
'datetime.date' is coerced to a datetime. In the future pandas will
not coerce, and the values not compare equal to the 'datetime.date'.
To retain the current behavior, convert the 'datetime.date' to a
datetime with 'pd.Timestamp'.
#!/bin/python3
0 True
1 False
dtype: bool

In addition, ordering comparisons will raise a ``TypeError`` in the future.

**Other Fixes**

- Reverted the ability of :func:`~DataFrame.to_sql` to perform multivalue
inserts as this caused regression in certain cases (:issue:`21103`).
Expand Down
30 changes: 30 additions & 0 deletions pandas/core/ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@
"""
# necessary to enforce truediv in Python 2.X
from __future__ import division
import datetime
import operator
import textwrap
import warnings

import numpy as np
import pandas as pd
Expand Down Expand Up @@ -1197,8 +1200,35 @@ def wrapper(self, other, axis=None):
if is_datetime64_dtype(self) or is_datetime64tz_dtype(self):
# Dispatch to DatetimeIndex to ensure identical
# Series/Index behavior
if (isinstance(other, datetime.date) and
not isinstance(other, datetime.datetime)):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this check needed? (I would not expect it ever to be true if it is a datetime.date ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently datetime.datetime subclasses datetime.date

https://bugs.python.org/issue28878

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than inlineing this, I think would be better to move to a module level function, maybe
other = maybe_coerce_other(other, op)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean just the new bit I added, right?

I'm inclined to leave it here since we'll just be deleting it soon (is 0.24.0 too soon?).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently datetime.datetime subclasses datetime.date

Ah, only checked the other way around :-)

Yeah, if this is only temporary code I would just leave it here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, is there an issue to fix this for 0.24.0? (can you add a TODO mention here)

# https://github.com/pandas-dev/pandas/issues/21152
# Compatibility for difference between Series comparison w/
# datetime and date
msg = (
"Comparing Series of datetimes with 'datetime.date'. "
"Currently, the 'datetime.date' is coerced to a "
"datetime. In the future pandas will not coerce, "
"and {future}. "
"To retain the current behavior, "
"convert the 'datetime.date' to a datetime with "
"'pd.Timestamp'."
)

if op in {operator.lt, operator.le, operator.gt, operator.ge}:
future = "a TypeError will be raised"
else:
future = (
"'the values will not compare equal to the "
"'datetime.date'"
)
msg = '\n'.join(textwrap.wrap(msg.format(future=future)))
warnings.warn(msg, FutureWarning, stacklevel=2)
other = pd.Timestamp(other)

res_values = dispatch_to_index_op(op, self, other,
pd.DatetimeIndex)

return self._constructor(res_values, index=self.index,
name=res_name)

Expand Down
40 changes: 40 additions & 0 deletions pandas/tests/series/test_arithmetic.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,46 @@ def test_ser_cmp_result_names(self, names, op):


class TestTimestampSeriesComparison(object):
def test_dt64_ser_cmp_date_warning(self):
# https://github.com/pandas-dev/pandas/issues/21359
# Remove this test and enble invalid test below
ser = pd.Series(pd.date_range('20010101', periods=10), name='dates')
date = ser.iloc[0].to_pydatetime().date()

with tm.assert_produces_warning(FutureWarning) as m:
result = ser == date
expected = pd.Series([True] + [False] * 9, name='dates')
tm.assert_series_equal(result, expected)
assert "Comparing Series of datetimes " in str(m[0].message)
assert "will not compare equal" in str(m[0].message)

with tm.assert_produces_warning(FutureWarning) as m:
result = ser != date
tm.assert_series_equal(result, ~expected)
assert "will not compare equal" in str(m[0].message)

with tm.assert_produces_warning(FutureWarning) as m:
result = ser <= date
tm.assert_series_equal(result, expected)
assert "a TypeError will be raised" in str(m[0].message)

with tm.assert_produces_warning(FutureWarning) as m:
result = ser < date
tm.assert_series_equal(result, pd.Series([False] * 10, name='dates'))
assert "a TypeError will be raised" in str(m[0].message)

with tm.assert_produces_warning(FutureWarning) as m:
result = ser >= date
tm.assert_series_equal(result, pd.Series([True] * 10, name='dates'))
assert "a TypeError will be raised" in str(m[0].message)

with tm.assert_produces_warning(FutureWarning) as m:
result = ser > date
tm.assert_series_equal(result, pd.Series([False] + [True] * 9,
name='dates'))
assert "a TypeError will be raised" in str(m[0].message)

@pytest.mark.skip(reason="GH-21359")
def test_dt64ser_cmp_date_invalid(self):
# GH#19800 datetime.date comparison raises to
# match DatetimeIndex/Timestamp. This also matches the behavior
Expand Down