CorrDiff: CWB Dataloader cleanups

[nbren12]

Modulus Pull Request

Description

This PR deletes some unused code that made the CWB data loader harder to follow. It also uncomments, the corrdiff dataset tests.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• The CHANGELOG.md is up to date with these changes.
• An issue is linked to this pull request.

Dependencies

[nbren12]

#589

[mnabian]

Need to merge this first for the CI to pass: #594

[jleinonen]

Pretty big changes here, so can you check that you can start training ok, and that inference results with the pretrained checkpoint look ok?

[nbren12]

Pretty big changes here, so can you check that you can start training ok, and that inference results with the pretrained checkpoint look ok?

Hey @Jussmith01. I'm not that spun-up on corrdiff training/inference, but did add a regression test against the real data, and ensured that first item in dataset was bit-for-bit identical throughout the refactor. Perhaps you can add a simple end-to-end test that runs the training and inference on a desktop in < 1 minute. Until such is in the code base, I think it's a bit laborious to make it a requirement for merging. Thoughts @mnabian?

[nbren12]

Ready for re-review. I decided to leave the old reference configs intact since they have the hardcoded validation and other times. We'll probably want to include that in some form later on.

[jleinonen]

Pretty big changes here, so can you check that you can start training ok, and that inference results with the pretrained checkpoint look ok?

I'm not that spun-up on corrdiff training/inference, but did add a regression test against the real data, and ensured that first item in dataset was bit-for-bit identical throughout the refactor. Perhaps you can add a simple end-to-end test that runs the training and inference on a desktop in < 1 minute. Until such is in the code base, I think it's a bit laborious to make it a requirement for merging. Thoughts @mnabian?

Okay, I think that's enough for this PR. Maybe you could check that the normalize_input, normalize_output, denormalize_input, denormalize_output functions also work identically pre/post PR?

[nbren12]

@jleinonen. normalize_input and normalize_output are covered by my existing regression check (since they are called in __getitem__). I will add a test for the denormalize methods.

[mnabian]

/blossom-ci