Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add select on read for MWA correlator FITS files #1458

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

bhazelton
Copy link
Member

@bhazelton bhazelton commented Jul 6, 2024

Description

Add support for select on read to MWA correlator FITS files to save memory and time. This adds support for most but not all select keywords. The implementation is a bit complicated because of the structure of MWA correlator FITS files, which is also different between the files written by the legacy correlator and MWAX.

Edit: we have benchmarking now and the benchmarks didn't slow down significantly
I am a bit concerned that some of these changes may have slowed down the reading when not selecting on read. I think this needs to be tested/profiled more, would love some help on that front.

I also refactored some selection code for the baseline-time axis and the frequency axis into functions for better code reuse, leading to changes in UVData.select and UVCal.select. And I improved the way the history string is built up.

Motivation and Context

closes #1386

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation change (documentation changes only)
  • Version change
  • Build or continuous integration change

Checklist:

New feature checklist:

  • I have added or updated the docstrings associated with my feature using the numpy docstring format.
  • I have updated the tutorial to highlight my new feature (if appropriate).
  • I have added tests to cover my new feature.
  • All new and existing tests pass.
  • I have updated the CHANGELOG.

@bhazelton bhazelton force-pushed the mwa_corr_fits_select branch 2 times, most recently from cdfc0e3 to ae54798 Compare July 12, 2024 03:56
Copy link

codecov bot commented Jul 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.92%. Comparing base (72dd3bc) to head (0305ab5).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1458      +/-   ##
==========================================
- Coverage   99.92%   99.92%   -0.01%     
==========================================
  Files          61       62       +1     
  Lines       21337    21327      -10     
==========================================
- Hits        21322    21312      -10     
  Misses         15       15              
Flag Coverage Δ
99.92% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Select on read for MWA correlator fits
1 participant