Squashed commit of the following:

commit 695539472ffe006a56c8c942dd09a2b819bc9f08
Merge: 86eec0b 4a3ff27
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Tue Jun 28 20:20:23 2022 +0000

    Merge branch 'master' of github.com:stresearch/gaia

commit 4a3ff27
Author: ktrapeznikov <ktrapeznikov@gmail.com>
Date:   Tue Jun 28 16:19:57 2022 -0400

    Dataset (#34)

    * Squashed commit of the following:

    commit 86eec0bcd8acdfb1ad25ca9b8fa2eccf925694f9
    Merge: 2e2f458 4974acc
    Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
    Date:   Tue Jun 28 20:06:52 2022 +0000

        Merge branch '19-fix-mean-bias'

    commit 2e2f45865aee0ed3c0c3c8253ba73aff7339c14b
    Merge: 59be221 f6bedff
    Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
    Date:   Tue Jun 28 20:04:25 2022 +0000

        Merge branch 'master' of github.com:stresearch/gaia

    commit 4974acc399c308d00d12915d6ccca394c48ee689
    Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
    Date:   Tue Jun 28 20:01:30 2022 +0000

        updates

    commit 98b0f42
    Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
    Date:   Wed Jun 8 00:30:51 2022 +0000

        minor fixes

    commit 59be221
    Merge: c160d9f e225ecd
    Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
    Date:   Wed Jun 8 00:18:30 2022 +0000

        Merge branch 'master' of github.com:stresearch/gaia

    commit c160d9f
    Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
    Date:   Wed Jun 8 00:18:22 2022 +0000

        adding changes to dataset

    * delete

    Co-authored-by: Kirill Trapeznikov <kirill.trapeznikov@str.us>

commit 86eec0bcd8acdfb1ad25ca9b8fa2eccf925694f9
Merge: 2e2f458 4974acc
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Tue Jun 28 20:06:52 2022 +0000

    Merge branch '19-fix-mean-bias'

commit 2e2f45865aee0ed3c0c3c8253ba73aff7339c14b
Merge: 59be221 f6bedff
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Tue Jun 28 20:04:25 2022 +0000

    Merge branch 'master' of github.com:stresearch/gaia

commit 4974acc399c308d00d12915d6ccca394c48ee689
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Tue Jun 28 20:01:30 2022 +0000

    updates

commit 98b0f42
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Wed Jun 8 00:30:51 2022 +0000

    minor fixes

commit 59be221
Merge: c160d9f e225ecd
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Wed Jun 8 00:18:30 2022 +0000

    Merge branch 'master' of github.com:stresearch/gaia

commit c160d9f
Author: Kirill Trapeznikov <kirill.trapeznikov@str.us>
Date:   Wed Jun 8 00:18:22 2022 +0000

    adding changes to dataset

create_dataset.py

@@ -0,0 +1,16 @@
+from gaia.data import NCDataConstructor
+
+
+cam4 = "cam4-famip-30m-timestep"
+spcam  = "spcamclbm-nx-16-20m-timestep"
+workers = 64
+cache = "cache"
+
+if __name__=="__main__":
+    # NCDataConstructor.default_data(split="train",  workers  =workers, prefix=cam4, train_years=3, save_location=".", cache = cache)
+    # NCDataConstructor.default_data(split="test",  workers  =workers, prefix=cam4, train_years=3, save_location=".", cache = cache)
+    # NCDataConstructor.default_data(split="train",  workers  =workers, prefix=spcam, train_years=2, save_location=".", cache = cache)
+    NCDataConstructor.default_data(split="test",  workers  =workers, prefix=spcam, train_years=2, save_location=".", cache = cache)
+
+
+    # NCDataConstructor.default_data(split="train")

gaia/data.py

@@ -753,6 +753,13 @@ def load_files_parallel(self, files, num_workers=8, save_file=None):
         #         logger.warning(f"failed {args}, {kwargs}")
         #         return
 
+        logger.info("delete cache files if any")
+
+        os.makedirs(self.cache, exist_ok=True)
+
+        for f in tqdm.tqdm(glob.glob(os.path.join(self.cache,"*"))):
+            os.remove(f)
+
         logger.info("downloading files")
 
         with ProcessPoolExecutor(max_workers=num_workers) as exec: