Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CorrDiff: CWB Dataloader cleanups #593

Merged
merged 13 commits into from
Jul 26, 2024
Merged

CorrDiff: CWB Dataloader cleanups #593

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
234ebae
uncomment tests
nbren12 Jul 15, 2024
364aae5
assert that v1==v2 before cleaning up
nbren12 Jul 15, 2024
b27ef2f
remove v1 and v2 option. these were identical
nbren12 Jul 15, 2024
655f8c1
add regression test
nbren12 Jul 15, 2024
3ccca31
remove unused normalization logic
nbren12 Jul 15, 2024
b3150d2
remove unused options, img_utils.reshape_fields
nbren12 Jul 15, 2024
7a66b02
remove unused options, img_utils.reshape_fields
nbren12 Jul 15, 2024
0ec7f0c
remove FilterTime
nbren12 Jul 15, 2024
c27c8f7
delete unused reference
nbren12 Jul 16, 2024
3940011
Revert "delete unused reference"
nbren12 Jul 16, 2024
a9345c7
test ZarrDataset.{denormalize/normalize}_{input/output}
nbren12 Jul 25, 2024
4d1f9db
Merge remote-tracking branch 'origin/main' into cleanup-normalizations
nbren12 Jul 25, 2024
2bb04ce
Merge branch 'main' into cleanup-normalizations
nbren12 Jul 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions examples/generative/corrdiff/conf/config_generate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,19 +82,16 @@ dataset:
train: False
all_times: True
data_path: /data/data2/2023-01-24-cwb-4years.zarr
n_history: 0
in_channels: [0, 1, 2, 3, 4, 9, 10, 11, 12, 17, 18, 19]
out_channels: [0, 17, 18, 19]
img_shape_x: 448
img_shape_y: 448
roll: false
add_grid: true
ds_factor: 4
min_path: null
max_path: null
global_means_path: null
global_stds_path: null
jleinonen marked this conversation as resolved.
Show resolved Hide resolved
normalization: "v1"

## Deterministic sampler options
sigma_min: null
Expand Down
3 changes: 0 additions & 3 deletions examples/generative/corrdiff/conf/config_generate_patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,19 +82,16 @@ dataset:
train: False
all_times: True
data_path: /data/data2/2023-01-24-cwb-4years.zarr
n_history: 0
in_channels: [0, 1, 2, 3, 4, 9, 10, 11, 12, 17, 18, 19]
out_channels: [0, 17, 18, 19]
img_shape_x: 448
img_shape_y: 448
roll: false
add_grid: true
ds_factor: 4
min_path: null
max_path: null
global_means_path: null
global_stds_path: null
jleinonen marked this conversation as resolved.
Show resolved Hide resolved
normalization: "v1"


## Deterministic sampler options
Expand Down
3 changes: 0 additions & 3 deletions examples/generative/corrdiff/conf/config_train_base.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,19 +87,16 @@ dataset:
type: cwb
train_test_split: false
data_path: /data/data2/2023-01-24-cwb-4years.zarr
n_history: 0
in_channels: [0, 1, 2, 3, 4, 9, 10, 11, 12, 17, 18, 19]
out_channels: [0, 17, 18, 19]
img_shape_x: 448
img_shape_y: 448
roll: false
add_grid: true
ds_factor: 4
min_path: null
max_path: null
global_means_path: null
global_stds_path: null
jleinonen marked this conversation as resolved.
Show resolved Hide resolved
normalization: "v1"

# Validation dataset options
# (need to set dataset.train_test_split == true to have an effect)
Expand Down
29 changes: 0 additions & 29 deletions examples/generative/corrdiff/conf/references/config_data_ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,8 @@ full_field: &FULL_FIELD

batch_size: 1
dt: 1
n_history: 0
img_shape_x: 448
img_shape_y: 448
normalization: v1 #minmax
# era5_data_dir: '/lustre/fsw/sw_climate_fno/cwb-align' #old
# cwb_data_dir: '/lustre/fsw/sw_climate_fno/cwb-rwrf-pad-2212/all_ranges' #old
# relative to CWB_ROOT environment variable
Expand All @@ -40,7 +38,6 @@ full_field_train_crop448_grid_12inchans_fcn_4outchans_4x_6layer: #config for sin
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,9,10,11,12,17,18,19]
out_channels: [0,17,18,19] #[17,18,19]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -53,7 +50,6 @@ full_field_val_crop448_grid_12inchans_fcn_4outchans_4x_6layer: #config for singl
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,9,10,11,12,17,18,19]
out_channels: [0,17,18,19] #[0,9,10,11]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -65,7 +61,6 @@ out_of_sample:
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,9,10,11,12,17,18,19]
out_channels: [0,17,18,19] #[0,9,10,11]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -81,14 +76,12 @@ full_field_train_crop448_grid_12inchans_fcn_4outchans_4x_normv2: #config for sin
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,9,10,11,12,17,18,19]
out_channels: [0,17,18,19] #[17,18,19]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
N_grid_channels: 4
add_grid: True
ds_factor: 4
normalization: v2



Expand All @@ -97,7 +90,6 @@ full_field_train_crop448_grid_12inchans_fcn_4outchans_4x: #config for single gpu
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,9,10,11,12,17,18,19]
out_channels: [0,17,18,19] #[17,18,19]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -110,7 +102,6 @@ full_field_val_crop448_grid_12inchans_fcn_4outchans_4x: &validation #config for
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,9,10,11,12,17,18,19]
out_channels: [0,17,18,19] #[0,9,10,11]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -124,7 +115,6 @@ full_field_train_crop448_grid_20inchans_4outchans_4x: #config for single gpu tra
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [0,17,18,19] #[17,18,19]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -137,7 +127,6 @@ full_field_val_crop448_grid_20inchans_4outchans_4x: #config for single gpu train
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [0,17,18,19] #[17, 18, 19]
roll: !!bool False
patch_size: 448
crop_size_x: 448
crop_size_y: 448
Expand All @@ -151,7 +140,6 @@ full_field_train_crop112_grid_20inchans_4outchans_4x: #config for single gpu tra
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [0,17,18,19] #[17,18,19]
roll: !!bool False
patch_size: 112
crop_size_x: 112
crop_size_y: 112
Expand All @@ -164,7 +152,6 @@ full_field_val_crop112_grid_20inchans_4outchans_4x: #config for single gpu train
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [0,17,18,19] #[17, 18, 19]
roll: !!bool False
patch_size: 112
crop_size_x: 448
crop_size_y: 448
Expand All @@ -178,7 +165,6 @@ full_field_train_crop112_grid_20inchans_19outchans_4x: #config for single gpu tr
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] #[17,18,19]
roll: !!bool False
patch_size: 112
crop_size_x: 112
crop_size_y: 112
Expand All @@ -191,7 +177,6 @@ full_field_val_crop112_grid_20inchans_19outchans_4x: #config for single gpu trai
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] #[17, 18, 19]
roll: !!bool False
patch_size: 112
crop_size_x: 448
crop_size_y: 448
Expand All @@ -206,7 +191,6 @@ full_field_train_crop112_grid_20inchans_4x: #config for single gpu training to c
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17,18,19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 112
crop_size_x: 112
crop_size_y: 112
Expand All @@ -219,7 +203,6 @@ full_field_val_crop112_grid_20inchans_4x: #config for single gpu training to cat
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 112
crop_size_x: 448
crop_size_y: 448
Expand All @@ -233,7 +216,6 @@ full_field_train_crop64_grid_20inchans_4x: #config for single gpu training to ca
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 64
crop_size_y: 64
Expand All @@ -246,7 +228,6 @@ full_field_val_crop64_grid_20inchans_4x: #config for single gpu training to catc
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 448
crop_size_y: 448
Expand All @@ -259,7 +240,6 @@ full_field_train_crop64_grid_20inchans: #config for single gpu training to catch
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 64
crop_size_y: 64
Expand All @@ -271,7 +251,6 @@ full_field_val_crop64_grid_20inchans: #config for single gpu training to catch b
<<: *FULL_FIELD
in_channels: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 448
crop_size_y: 448
Expand All @@ -284,7 +263,6 @@ full_field_train_crop64_grid: #config for single gpu training to catch bugs and
<<: *FULL_FIELD
in_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 64
crop_size_y: 64
Expand All @@ -296,7 +274,6 @@ full_field_val_crop64_grid: #config for single gpu training to catch bugs and ov
<<: *FULL_FIELD
in_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 448
crop_size_y: 448
Expand All @@ -310,7 +287,6 @@ full_field_train_crop64: #config for single gpu training to catch bugs and overf
<<: *FULL_FIELD
in_channels: [17.18,19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 64
crop_size_y: 64
Expand All @@ -320,7 +296,6 @@ full_field_val_crop64: #config for single gpu training to catch bugs and overfit
<<: *FULL_FIELD
in_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 64
crop_size_x: 448
crop_size_y: 448
Expand All @@ -331,7 +306,6 @@ full_field_train_crop112: #config for single gpu training to catch bugs and over
<<: *FULL_FIELD
in_channels: [17.18,19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 112
crop_size_x: 112
crop_size_y: 112
Expand All @@ -341,7 +315,6 @@ full_field_val_crop112: #config for single gpu training to catch bugs and overfi
<<: *FULL_FIELD
in_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 112
crop_size_x: 112
crop_size_y: 112
Expand All @@ -352,7 +325,6 @@ full_field_train_crop224: #config for single gpu training to catch bugs and over
<<: *FULL_FIELD
in_channels: [17.18,19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 224
crop_size_x: 224
crop_size_y: 224
Expand All @@ -362,7 +334,6 @@ full_field_val_crop224: #config for single gpu training to catch bugs and overfi
<<: *FULL_FIELD
in_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
out_channels: [17, 18, 19] #[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
roll: !!bool False
patch_size: 224
crop_size_x: 224
crop_size_y: 224
Expand Down
Loading