Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AAPP L1b reader not to up-cast data to float64 #2893

Merged
merged 3 commits into from
Sep 19, 2024

Conversation

pnuu
Copy link
Member

@pnuu pnuu commented Sep 2, 2024

The aapp_l1b reader was returning the data as float64. This PR forces the data to float32.

  • Closes #xxxx
  • Tests added
  • Fully documented
  • Add your name to AUTHORS.md if not there already

@pnuu pnuu added enhancement code enhancements, features, improvements component:readers labels Sep 2, 2024
@pnuu pnuu self-assigned this Sep 2, 2024
@pnuu
Copy link
Member Author

pnuu commented Sep 2, 2024

Memory and CPU usage graph using main:
Screenshot 2024-09-02 at 11-06-53 Bokeh Plot

the same with this PR:
Screenshot 2024-09-02 at 11-09-45 Bokeh Plot

These were done with 12 composites, nearest neighbour resampling and saving to COG.

The memory usage was reduced by approx. 1.4 GB and processing time by 5 seconds.

Copy link

codecov bot commented Sep 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.05%. Comparing base (084b114) to head (15a0800).
Report is 101 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2893      +/-   ##
==========================================
- Coverage   96.06%   96.05%   -0.02%     
==========================================
  Files         370      370              
  Lines       54320    54334      +14     
==========================================
+ Hits        52185    52191       +6     
- Misses       2135     2143       +8     
Flag Coverage Δ
behaviourtests 3.99% <0.00%> (-0.01%) ⬇️
unittests 96.15% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

satpy/readers/aapp_l1b.py Outdated Show resolved Hide resolved
@coveralls
Copy link

coveralls commented Sep 2, 2024

Pull Request Test Coverage Report for Build 10664568987

Details

  • 32 of 32 (100.0%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall first build on bugfix-aapp-l1b-float32 at 96.152%

Totals Coverage Status
Change from base Build 10654375136: 96.2%
Covered Lines: 52422
Relevant Lines: 54520

💛 - Coveralls

@pnuu
Copy link
Member Author

pnuu commented Sep 19, 2024

@mraspaud ok to merge?

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud mraspaud merged commit d3e6fd4 into pytroll:main Sep 19, 2024
17 of 18 checks passed
@djhoese
Copy link
Member

djhoese commented Sep 21, 2024

@pnuu any idea how long this upcasting had been going on? After this PR some integration tests of mine for my Polar2Grid project are showing ~11% of pixels being slightly different. I don't think I'm using numpy 2 yet so does that mean this upcasting has been going on for many years?

@pnuu pnuu deleted the bugfix-aapp-l1b-float32 branch September 21, 2024 17:47
@pnuu
Copy link
Member Author

pnuu commented Sep 21, 2024

@djhoese float64 seems to originate from here where it was changed from float, so 2 years back.

@pnuu
Copy link
Member Author

pnuu commented Sep 23, 2024

And to look further, prior to that the dtype had been float64 for a while when it was changed to it from np.float in 605bc63 . And after checking the changes backwards in time until mpop era, it has always been either float, np.float or np.float64, which I believe all are/were 64-bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants