Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#19431] Regression in make_block_same_class (tests failing for new fastparquet release) #19434

Merged
merged 23 commits into from
Jan 29, 2018
Merged

[#19431] Regression in make_block_same_class (tests failing for new fastparquet release) #19434

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
eb55928
udpate test_parquet since fastparquet now handles tz
minggli Jan 28, 2018
2f4fc07
bring back dtype kwarg because it is needed for DatetimeTZBlock
minggli Jan 28, 2018
e18f8e6
version dependence test_datetime_tz
minggli Jan 28, 2018
9a07d97
separate test cases for new and old behaviour of fastparquet
minggli Jan 28, 2018
3d9810d
tidy test_datetime_tz to test old behaviour of fastparquet<0.14
minggli Jan 28, 2018
ee75fdf
rephase reason to skip test case for oldder fp
minggli Jan 28, 2018
68d6324
follow pytest fixture pattern as in pyarrow
minggli Jan 28, 2018
985081d
follow pyarrow test_basic style for fastparquet new behaviour>=0.1.4
minggli Jan 28, 2018
0cfcd37
fastparquet=0.1.3
minggli Jan 28, 2018
0c4a6d7
other api change
minggli Jan 28, 2018
6ce68cf
fix typo
minggli Jan 28, 2018
f414743
deprecation warning for dtype in make_block_same_class.
minggli Jan 28, 2018
0d76fe7
Future warning for dtype in make_block_same_class.
minggli Jan 28, 2018
bb95dc6
update notes as fastparquet nows supports timezone
minggli Jan 28, 2018
d9a2e2a
remove fastparquet pin
minggli Jan 29, 2018
97b17e9
remove other api change as it is internal
minggli Jan 29, 2018
800b741
remove version
minggli Jan 29, 2018
52220b9
remove dtype for make_block and DeprecationWarning on make_block_same…
minggli Jan 29, 2018
c602a76
FutureWarning on make_block_same_class
minggli Jan 29, 2018
ddbbde3
test case for dtype and warning generation
minggli Jan 29, 2018
6e6b5f0
issue number and simplify test
minggli Jan 29, 2018
326394f
misc doc
minggli Jan 29, 2018
77422ba
use pandas warning assert
minggli Jan 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions pandas/core/internals.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ def array_dtype(self):
"""
return self.dtype

def make_block(self, values, placement=None, ndim=None):
def make_block(self, values, placement=None, ndim=None, dtype=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can add this back, but it needs to be deprecated anyhow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then please answer in the original issue how to achieve the same result (see the code example I posted there, to create a new DatetimeTZ block from a numpy array).
But I don't see why this needs to be deprecated. The dtype argument was actually doing something, in contrast to fastpath (which we therefore deprecated)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fp should be using make_block and not make_block_same_class

"""
Create a new block, with type inference propagate any values that are
not specified
Expand All @@ -216,20 +216,21 @@ def make_block(self, values, placement=None, ndim=None):
if ndim is None:
ndim = self.ndim

return make_block(values, placement=placement, ndim=ndim)
return make_block(values, placement=placement, ndim=ndim, dtype=dtype)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe remove dtype here? As fastparquet was not using it (and I don't think anybody else will)
Or otherwise add the same deprecation warning as you did for make_block_same_class


def make_block_scalar(self, values):
"""
Create a ScalarBlock
"""
return ScalarBlock(values)

def make_block_same_class(self, values, placement=None, ndim=None):
def make_block_same_class(self, values, placement=None, ndim=None,
dtype=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deprecate dtype here (add as a FutureWarning)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

""" Wrap given values in a block of same type as self. """
if placement is None:
placement = self.mgr_locs
return make_block(values, placement=placement, ndim=ndim,
klass=self.__class__)
klass=self.__class__, dtype=dtype)

def __unicode__(self):

Expand Down
11 changes: 9 additions & 2 deletions pandas/tests/io/test_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,11 @@ class TestParquetFastParquet(Base):
def test_basic(self, fp, df_full):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make 2 tests here, one for < 0.1.4 and 1 for >

df = df_full

# additional supported types for fastparquet >= 0.1.4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls follow the existing style. see how we have a fixture for pa_ge_070. you need to make similar for fp

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the existing style (see how it is for pyarrow) and what I asked for, so please leave it like this (in this PR).

For the old test we can use a fixture like fp_lt_014 however.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

little confused. But I agree it's better to match existing style in test_basic to avoid unnecessary style variation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a totally incorrect style ATM. So please see what you can fix. skipping INSIDE a test function is not correct, rather a fixture should be used. This must have slipped in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jeff, please leave this. I asked @minggli to do this, and it is the current style. You can do a follow-up PR if you want to change this in all places.

if LooseVersion(fastparquet.__version__) > LooseVersion('0.1.3'):
df['datetime_tz'] = pd.date_range('20130101', periods=3,
tz='US/Eastern')

# additional supported types for fastparquet
df['timedelta'] = pd.timedelta_range('1 day', periods=3)

Expand Down Expand Up @@ -483,10 +488,12 @@ def test_categorical(self, fp):
check_round_trip(df, fp)

def test_datetime_tz(self, fp):
# doesn't preserve tz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same need 2 tests (1 with each fixture)

Copy link
Contributor Author

@minggli minggli Jan 28, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyarrow is using similar style. there are already two tests for datetime_tz, one < 0.1.4 one >.

if LooseVersion(fastparquet.__version__) > LooseVersion('0.1.3'):
pytest.skip("timezone not supported for older fp")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe rather "timezones supported in newer versions of fp"


# fastparquet<0.1.4 doesn't preserve tz
df = pd.DataFrame({'a': pd.date_range('20130101', periods=3,
tz='US/Eastern')})

# warns on the coercion
with catch_warnings(record=True):
check_round_trip(df, fp, expected=df.astype('datetime64[ns]'))
Expand Down