-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add select on read for MWA correlator FITS files #1458
Open
bhazelton
wants to merge
10
commits into
main
Choose a base branch
from
mwa_corr_fits_select
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bhazelton
force-pushed
the
mwa_corr_fits_select
branch
2 times, most recently
from
July 12, 2024 03:56
cdfc0e3
to
ae54798
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1458 +/- ##
==========================================
- Coverage 99.92% 99.92% -0.01%
==========================================
Files 61 62 +1
Lines 21337 21327 -10
==========================================
- Hits 21322 21312 -10
Misses 15 15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
bhazelton
force-pushed
the
mwa_corr_fits_select
branch
from
July 12, 2024 14:06
f4add86
to
a244fce
Compare
refactor some more code into utils methods to allow reuse
bhazelton
force-pushed
the
mwa_corr_fits_select
branch
from
September 24, 2024 20:41
a244fce
to
6ae87bc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add support for select on read to MWA correlator FITS files to save memory and time. This adds support for most but not all select keywords. The implementation is a bit complicated because of the structure of MWA correlator FITS files, which is also different between the files written by the legacy correlator and MWAX.
Edit: we have benchmarking now and the benchmarks didn't slow down significantly
I am a bit concerned that some of these changes may have slowed down the reading when not selecting on read. I think this needs to be tested/profiled more, would love some help on that front.I also refactored some selection code for the baseline-time axis and the frequency axis into functions for better code reuse, leading to changes in
UVData.select
andUVCal.select
. And I improved the way the history string is built up.Motivation and Context
closes #1386
Types of changes
Checklist:
New feature checklist: