BUG: read-only values in cython funcs #37613

jbrockmendel · 2020-11-03T20:59:30Z

closes BUG: read-only buffer failures in datetime parsing #34857
closes BUG: another ValueError: buffer source array is read-only #37264
closes BUG: comparison/indexing causes ValueError: buffer source array is read-only #37312
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…adonly

jreback · 2020-11-04T02:06:08Z

pandas/_libs/join.pyx

@@ -268,7 +268,7 @@ ctypedef fused join_t:

 @cython.wraparound(False)
 @cython.boundscheck(False)
-def left_join_indexer_unique(join_t[:] left, join_t[:] right):
+def left_join_indexer_unique(ndarray[join_t] left, ndarray[join_t] right):


maybe these need const?

we can't use const join_t until cython3

jreback · 2020-11-04T02:06:36Z

pandas/tests/tools/test_to_datetime.py

@@ -36,6 +36,14 @@


 class TestTimeConversionFormats:
+    def test_to_datetime_readonly(self):


parameterize over readonly

jreback · 2020-11-04T02:06:41Z

pandas/tests/tools/test_to_timedelta.py

@@ -9,6 +9,14 @@


 class TestTimedeltas:
+    def test_to_timedelta_readonly(self):


…adonly

jreback · 2020-11-04T13:44:12Z

@simonjayhawkins this might be worth backporting

simonjayhawkins · 2020-11-04T14:32:53Z

@simonjayhawkins this might be worth backporting

sure. will wait for #37628 to be merged to avoid merge conflicts in release notes.

…n cython funcs

simonjayhawkins · 2020-11-04T19:53:39Z

pandas/core/arrays/datetimelike.py

@@ -1025,9 +1025,8 @@ def _addsub_object_array(self, other: np.ndarray, op):
        result : same class as self
        """
        assert op in [operator.add, operator.sub]
-        if len(other) == 1:
+        if len(other) == 1 and self.ndim == 1:


@jbrockmendel do you recall if tests were failing without this change, or unrelated to the tests added here.

Yes, a test failed without this change

on the backport tests fail with and without this change. with this change, we get an additional 13 performance warnings. without this change, we don't get performance warnings in 3 tests

without this change, we don't get performance warnings in 3 tests

Probably the same three I saw on master that motivated this edit. I just checked out the backport branch locally, will try to troubleshoot.

im not wild about not-having a better solution, but i think the immediate issue can be addressed by making the condiiton and (self.ndim == 1 or other.shape == self.shape == (1, 1))

Thanks @jbrockmendel for looking into this. I have updated the backport PR with this change for now, pending further discussion there on best approach. #37633 (comment)

the suggested update applied to the backport still failed. 458e0ce

@jbrockmendel I'm struggling with this. (working on 1.1x so maybe differences with master)

can you justify this change other than tests were failing.

somethings changed in the internals dispatch where other in the _addsub_object_array call is now a scalar (in a 2d array with shape 1,1) for the failing test_td64arr_add_sub_object_array test instead of a 2d array on 1.1.x

so not raising a performance warning makes sense if other is now a 'scalar'

(also the repr fails for a 2d timedelta array)

on 1.1.x

(Pdb) other array([[Timedelta('1 days 00:00:00')], [<2 * Days>]], dtype=object) (Pdb) a Traceback (most recent call last): ... ValueError: Value must be Timedelta, string, integer, float, timedelta or convertible, not TimedeltaArray (Pdb) self._data array([[ 86400000000000], [172800000000000]], dtype='timedelta64[ns]') (Pdb)

with the patch applied ( #37633)

c:\users\simon\pandas\pandas\core\arrays\datetimelike.py(1337)_addsub_object_array() -> assert op in [operator.add, operator.sub] (Pdb) other array([[<2 * Days>]], dtype=object) (Pdb) self._data array([[172800000000000]], dtype='timedelta64[ns]') (Pdb)

can you justify this change other than tests were failing.

I never did manage to track down the underlying source of the problem that made this particular edit necessary. Its still early-ish afternoon for me so I can spend some more time today trying again if you think its a priority.

(also the repr fails for a 2d timedelta array)

FWIW that was fixed in #37164

… (#37655) * Moving the file test_frame.py to a new directory * Сreated file test_frame_color.py * Transfer tests of test_frame.py to test_frame_color.py * PEP 8 fixes * Transfer tests of test_frame.py to test_frame_groupby.py and test_frame_subplots.py * Removing unnecessary imports * PEP 8 fixes * Fixed class name * Transfer tests of test_frame.py to test_frame_subplots.py * Transfer tests of test_frame.py to test_frame_groupby.py, test_frame_subplots.py, test_frame_color.py * Changed class names * Removed unnecessary imports * Removed import * catch FutureWarnings (#37587) * TST/REF: collect indexing tests by method (#37590) * REF: prelims for single-path setitem_with_indexer (#37588) * ENH: __repr__ for 2D DTA/TDA (#37164) * CLN: de-duplicate _validate_where_value with _validate_setitem_value (#37595) * TST/REF: collect tests by method (#37589) * TST/REF: move remaining setitem tests from test_timeseries * TST/REF: rehome test_timezones test * move misplaced arithmetic test * collect tests by method * move misplaced file * REF: Categorical.is_dtype_equal -> categories_match_up_to_permutation (#37545) * CLN refactor non-core (#37580) * refactor core/computation (#37585) * TST/REF: share method tests between DataFrame and Series (#37596) * BUG: Index.where casting ints to str (#37591) * REF: IntervalArray comparisons (#37124) * regression fix for merging DF with datetime index with empty DF (#36897) * ERR: fix error message in Period for invalid frequency (#37602) * CLN: remove rebox_native (#37608) * TST/REF: tests.generic (#37618) * TST: collect tests by method (#37617) * TST/REF: collect test_timeseries tests by method * misplaced DataFrame.values tst * misplaced dataframe.values test * collect test by method * TST/REF: share tests across Series/DataFrame (#37616) * Gh 36562 typeerror comparison not supported between float and str (#37096) * docs: fix punctuation (#37612) * REGR: pd.to_hdf(..., dropna=True) not dropping missing rows (#37564) * parametrize set_axis tests (#37619) * CLN: clean color selection in _matplotlib/style (#37203) * DEPR: DataFrame/Series.slice_shift (#37601) * REF: re-use validate_setitem_value in Categorical.fillna (#37597) * PERF: release gil for ewma_time (#37389) * BUG: Groupy dropped nan groups from result when grouping over single column (#36842) * ENH: implement timeszones support for read_json(orient='table') and astype() from 'object' (#35973) * REF/BUG/TYP: read_csv shouldn't close user-provided file handles (#36997) * BUG/REF: read_csv shouldn't close user-provided file handles * get_handle: typing, returns is_wrapped, use dataclass, and make sure that all created handlers are returned * remove unused imports * added IOHandleArgs.close * added IOArgs.close * mostly comments * move memory_map from TextReader to CParserWrapper * moved IOArgs and IOHandles * more comments Co-authored-by: Jeff Reback <jeff@reback.net> * more typing checks to pre-commit (#37539) * TST: 32bit dtype compat test_groupby_dropna (#37623) * BUG: Metadata propagation for groupby iterator (#37461) * BUG: read-only values in cython funcs (#37613) * CLN refactor core/arrays (#37581) * Fixed Metadata Propogation in DataFrame (#37381) * TYP: add Shape alias to pandas._typing (#37128) * DOC: Fix typo (#37630) * CLN: parametrize test_nat_comparisons (#37195) * dataframe dataclass docstring updated (#37632) * refactor core/groupby (#37583) * BUG: set index of DataFrame.apply(f) when f returns dict (#37544) (#37606) * BUG: to_dict should return a native datetime object for NumPy backed dataframes (#37571) * ENH: memory_map for compressed files (#37621) * DOC: add example & prose of slicing with labels when index has duplicate labels (#36814) * DOC: add example & prose of slicing with labels when index has duplicate labels #36251 * DOC: proofread the sentence. Co-authored-by: Jun Kudo <jun-lab@junnoMacBook-Pro.local> * DOC: Fix typo (#37636) "columns(s)" sounded odd, I believe it was supposed to be just "column(s)". * CI: troubleshoot win py38 builds (#37652) * TST/REF: collect indexing tests by method (#37638) * TST/REF: collect tests for get_numeric_data (#37634) * misplaced loc test * TST/REF: collect get_numeric_data tests * REF: de-duplicate _validate_insert_value with _validate_scalar (#37640) * CI: catch windows py38 OSError (#37659) * share test (#37679) * TST: match matplotlib warning message (#37666) * TST: match matplotlib warning message * TST: match full message * pd.Series.loc.__getitem__ promotes to float64 instead of raising KeyError (#37687) * REF/TST: misplaced Categorical tests (#37678) * REF/TST: collect indexing tests by method (#37677) * CLN: only call _wrap_results one place in nanmedian (#37673) * TYP: Index._concat (#37671) * BUG: CategoricalIndex.equals casting non-categories to np.nan (#37667) * CLN: _replace_single (#37683) * REF: simplify _replace_single by noting regex kwarg is bool * Annotate * CLN: remove never-False convert kwarg * TYP: make more internal funcs keyword-only (#37688) * REF: make Series._replace_single a regular method (#37691) * REF: simplify cycling through colors (#37664) * REF: implement _wrap_reduction_result (#37660) * BUG: preserve fold in Timestamp.replace (#37644) * CLN: Clean indexing tests (#37689) * TST: fix warning for pie chart (#37669) * PERF: reverted change from commit 7d257c6 to solve issue #37081 (#37426) * DataFrameGroupby.boxplot fails when subplots=False (#28102) * ENH: Improve error reporting for wrong merge cols (#37547) * Transfer tests of test_frame.py to test_frame_color.py * PEP8 * Fixes for linter * Сhange pd.DateFrame to DateFrame * Move inconsistent namespace check to pre-commit, fixup more files (#37662) * check for inconsistent namespace usage * doc * typos * verbose regex * use verbose flag * use verbose flag * match both directions * add test * don't import annotations from future * update extra couple of cases * 🚚 rename * typing * don't use subprocess * don't type tests * use pathlib * REF: simplify NDFrame.replace, ObjectBlock.replace (#37704) * REF: implement Categorical.encode_with_my_categories (#37650) * REF: implement Categorical.encode_with_my_categories * privatize * BUG: unpickling modifies Block.ndim (#37657) * REF: dont support dt64tz in nanmean (#37658) * CLN: Simplify groupby head/tail tests (#37702) * Bug in loc raised for numeric label even when label is in Index (#37675) * REF: implement replace_regex, remove unreachable branch in ObjectBlock.replace (#37696) * TYP: Check untyped defs (except vendored) (#37556) * REF: remove ObjectBlock._replace_single (#37710) * Transfer tests of test_frame.py to test_frame_color.py * TST/REF: collect indexing tests by method (#37590) * PEP8 * Сhange DateFrame to pd.DateFrame * Сhange pd.DateFrame to DateFrame * Removing imports * Bug fixes * Bug fixes * Fix incorrect merge * test_frame_color.py edit * Transfer tests of test_frame.py to test_frame_color.py, test_frame_groupby.py and test_frame_subplots.py * Removing unnecessary imports * PEP8 * # Conflicts: # pandas/tests/plotting/frame/test_frame.py # pandas/tests/plotting/frame/test_frame_color.py # pandas/tests/plotting/frame/test_frame_subplots.py * Moving the file test_frame.py to a new directory * Transfer tests of test_frame.py to test_frame_color.py, test_frame_groupby.py and test_frame_subplots.py * Removing unnecessary imports * PEP8 * CLN: clean categorical indexes tests (#37721) * Fix merge error * PEP 8 fixes * Fix merge error * Removing unnecessary imports * PEP 8 fixes * Fixed class name * Transfer tests of test_frame.py to test_frame_subplots.py * Transfer tests of test_frame.py to test_frame_groupby.py, test_frame_subplots.py, test_frame_color.py * Changed class names * Removed unnecessary imports * Removed import * TST/REF: collect indexing tests by method (#37590) * TST: match matplotlib warning message (#37666) * TST: match matplotlib warning message * TST: match full message * TST: fix warning for pie chart (#37669) * Transfer tests of test_frame.py to test_frame_color.py * PEP8 * Fixes for linter * Сhange pd.DateFrame to DateFrame * Transfer tests of test_frame.py to test_frame_color.py * PEP8 * Сhange DateFrame to pd.DateFrame * Сhange pd.DateFrame to DateFrame * Removing imports * Bug fixes * Bug fixes * Fix incorrect merge * test_frame_color.py edit * Fix merge error * Fix merge error * Removing unnecessary features * Resolving Commit Conflicts daf999f 365d843 * black fix Co-authored-by: jbrockmendel <jbrockmendel@gmail.com> Co-authored-by: Marco Gorelli <m.e.gorelli@gmail.com> Co-authored-by: Philip Cerles <philip.cerles@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Sven <sven.schellenberg@paradynsystems.com> Co-authored-by: Micael Jarniac <micael@jarniac.com> Co-authored-by: Andrew Wieteska <48889395+arw2019@users.noreply.github.com> Co-authored-by: Maxim Ivanov <41443370+ivanovmg@users.noreply.github.com> Co-authored-by: Erfan Nariman <34067903+erfannariman@users.noreply.github.com> Co-authored-by: Fangchen Li <fangchen.li@outlook.com> Co-authored-by: patrick <61934744+phofl@users.noreply.github.com> Co-authored-by: attack68 <24256554+attack68@users.noreply.github.com> Co-authored-by: Torsten Wörtwein <twoertwein@users.noreply.github.com> Co-authored-by: Jeff Reback <jeff@reback.net> Co-authored-by: Janus <janus@insignificancegalore.net> Co-authored-by: Joel Whittier <rootbeerfriend@gmail.com> Co-authored-by: taytzehao <jtth95@gmail.com> Co-authored-by: ma3da <34522496+ma3da@users.noreply.github.com> Co-authored-by: junk <juntrp0207@gmail.com> Co-authored-by: Jun Kudo <jun-lab@junnoMacBook-Pro.local> Co-authored-by: Alex Kirko <alexander.kirko@gmail.com> Co-authored-by: Yassir Karroum <ukarroum17@gmail.com> Co-authored-by: Kaiqi Dong <kaiqi@kth.se> Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com> Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com>

clouds56 · 2021-09-12T05:30:37Z

to_datetime not works when unit set,

import pandas as pd
import numpy as np

a = np.array([1,2,3])
a.setflags(write=False)
pd.to_datetime(a, utc=True, unit='s')

This would call tslib.array_with_unit_to_datetime eventually.

jbrockmendel added 2 commits November 3, 2020 12:58

BUG: read-only values in cython funcs

cf89507

Merge branch 'master' of https://github.com/pandas-dev/pandas into re…

eb5b6e9

…adonly

jreback requested changes Nov 4, 2020

View reviewed changes

jreback added this to the 1.2 milestone Nov 4, 2020

jreback added Bug Compat pandas objects compatability with Numpy or Python functions Datetime Datetime data dtype labels Nov 4, 2020

jbrockmendel added 3 commits November 3, 2020 19:43

Merge branch 'master' of https://github.com/pandas-dev/pandas into re…

192231a

…adonly

parametrize

dbe99c5

test fixup

8d41c44

jreback approved these changes Nov 4, 2020

View reviewed changes

jreback merged commit 1c6cd01 into pandas-dev:master Nov 4, 2020

jbrockmendel deleted the readonly branch November 4, 2020 16:07

simonjayhawkins pushed a commit to simonjayhawkins/pandas that referenced this pull request Nov 4, 2020

Backport PR pandas-dev#37613 on branch 1.1.x: BUG: read-only values i…

135b215

…n cython funcs

simonjayhawkins mentioned this pull request Nov 4, 2020

Backport PR #37613 on branch 1.1.x: BUG: read-only values in cython funcs #37633

Closed

simonjayhawkins reviewed Nov 4, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: read-only values in cython funcs #37613

BUG: read-only values in cython funcs #37613

jbrockmendel commented Nov 3, 2020

jreback Nov 4, 2020

jbrockmendel Nov 4, 2020

jreback Nov 4, 2020

jreback Nov 4, 2020

jreback commented Nov 4, 2020

simonjayhawkins commented Nov 4, 2020

simonjayhawkins Nov 4, 2020

jbrockmendel Nov 4, 2020

simonjayhawkins Nov 4, 2020

jbrockmendel Nov 4, 2020

jbrockmendel Nov 4, 2020

simonjayhawkins Nov 9, 2020

simonjayhawkins Dec 1, 2020

simonjayhawkins Dec 1, 2020

jbrockmendel Dec 1, 2020

clouds56 commented Sep 12, 2021

		@@ -36,6 +36,14 @@


		class TestTimeConversionFormats:
		def test_to_datetime_readonly(self):

		@@ -9,6 +9,14 @@


		class TestTimedeltas:
		def test_to_timedelta_readonly(self):

BUG: read-only values in cython funcs #37613

BUG: read-only values in cython funcs #37613

Conversation

jbrockmendel commented Nov 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Nov 4, 2020

simonjayhawkins commented Nov 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clouds56 commented Sep 12, 2021