Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataSet.to_numpy() should use numpy dtypes whenever possible #182

Merged
merged 7 commits into from
Sep 7, 2022

Conversation

JBGreisman
Copy link
Member

Pandas DataFrames that contain ExtensionDtypes always default to output data with object dtype when DataFrame.to_numpy() is called. This is suboptimal for MTZ data, which by construction must be compatible with float32, and possibly int32.

This PR wraps the pandas call with DataSet.to_numpy() to assess whether a more sensible default (either float32 or int32) can be used based on the existing data. This should help to avoid cases where data is unnecessarily cast to an object array, which can lead to unexpected behavior downstream.

@JBGreisman JBGreisman added enhancement Improvement to existing feature MTZDtypes Issues related to custom dtypes labels Sep 6, 2022
@JBGreisman JBGreisman changed the title Fix #33: DataSet.to_numpy() should use numpy dtypes whenever possible DataSet.to_numpy() should use numpy dtypes whenever possible Sep 6, 2022
@codecov-commenter
Copy link

codecov-commenter commented Sep 6, 2022

Codecov Report

Merging #182 (7c0264f) into main (4222ffc) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #182      +/-   ##
==========================================
+ Coverage   98.36%   98.37%   +0.01%     
==========================================
  Files          45       45              
  Lines        1772     1783      +11     
==========================================
+ Hits         1743     1754      +11     
  Misses         29       29              
Flag Coverage Δ
unittests 98.37% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
reciprocalspaceship/dataset.py 98.20% <100.00%> (+0.04%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Member

@kmdalton kmdalton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it. I wish we also handled the case where MTZDtype is mixed with Pandas float dtypes. I'm not sure that can be implemented in a way that doesn't cause nasty side effects.

reciprocalspaceship/dataset.py Show resolved Hide resolved
reciprocalspaceship/dataset.py Outdated Show resolved Hide resolved
tests/dtypes/test_dataset_to_numpy.py Outdated Show resolved Hide resolved
tests/dtypes/test_dataset_to_numpy.py Outdated Show resolved Hide resolved
@JBGreisman JBGreisman requested a review from kmdalton September 7, 2022 00:37
@kmdalton kmdalton merged commit d70c5a3 into main Sep 7, 2022
@kmdalton kmdalton deleted the df2numpy branch September 7, 2022 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvement to existing feature MTZDtypes Issues related to custom dtypes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants