Updates for version 1.3 #33

kanishkan91 · 2022-10-10T18:52:21Z

Take out transition code to increase memory allocation
Add parallelization for land type kernel density processing
Add script to write ncdfs using xarray based on inputs in config file
Add testing script for new users
Make reading in region, basin mapping file parameterzied so users can read in their own region, basin mapping file
Changes to code cov yml along with addition of badge.
Changes to setup.py (New package dependencies, bumping up python version)
Addition of demeter 1.3.1 workflow which accounts for all changes above.
Adding new docs generated using sphinx

1. Take out transition code to increase memory allocation 2. Add parallelization for land type kernel density processing 3. Add post processing script to generate ncdf files corresponding to demeter runs with each LC as a subdata type 4. Add testing script for new users

1. Assign threads dynamically when applying convultion filter 2. Add ncdf_conversion to native demeter code base 3. Update mapping files for IM3 CLM team 4. Add xarray to setup

codecov-commenter · 2022-10-17T23:54:23Z

Codecov Report

Merging #33 (0ef1c0c) into main (90e4edf) will decrease coverage by 3.47%.
The diff coverage is 43.42%.

@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
- Coverage   73.49%   70.02%   -3.48%     
==========================================
  Files          17       18       +1     
  Lines        1343     1461     +118     
==========================================
+ Hits          987     1023      +36     
- Misses        356      438      +82

Impacted Files	Coverage Δ
demeter/demeter_io/reader.py	`70.83% <ø> (ø)`
setup.py	`0.00% <ø> (ø)`
demeter/ncdf_conversion.py	`12.90% <12.90%> (ø)`
demeter/demeter_io/writer.py	`16.78% <71.42%> (+2.18%)`	⬆️
demeter/config_reader.py	`67.22% <83.33%> (-0.17%)`	⬇️
demeter/process.py	`98.24% <90.90%> (+7.17%)`	⬆️
demeter/change/expansion.py	`90.47% <100.00%> (-0.09%)`	⬇️
demeter/change/intensification.py	`88.70% <100.00%> (-0.10%)`	⬇️
demeter/install_supplement.py	`40.00% <100.00%> (ø)`
demeter/preprocess_data.py	`91.93% <100.00%> (ø)`
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

1. Adding code so that ncdfs can be generated endogenously by setting a constant in config file 2. Bumping up python version to 3.8 3. Adding documentation for handle_single_pft() function

… demeter_1.3.1

marideeweber

As a python amateur, I was able to successfully run demeter and view the outputs, as well as modify the spatial resolution and run it again. There were hiccups at the beginning due to package version issues, but those have been resolved by changing which package versions are required. Great work!

crvernon

Great job @kanishkan91! Just a few inline comments to see to.

Also, for the test data: let's stay away from pickled files and use parquet files instead. Pickled files carry version dependencies.

Thanks!

crvernon · 2023-01-04T12:33:19Z

demeter/ncdf_conversion.py

+        return ds
+
+
+if __name__ == "__main__":


Even though this code will only run from the target script, let's remove the direct call since we are in a package. So everything after 261 should go.

Fixed. Thanks!

crvernon · 2023-01-04T12:33:47Z

demeter/process.py

@@ -57,7 +59,7 @@ def prep_step(self):
                                                           self.s.weights, self.s.spat_ludataharm)

        # create transition array to store data
-        self.transitions = np.zeros(shape=(self.l_spat_region, self.l_order_rules, self.l_order_rules))
+        #self.transitions = np.zeros(shape=(self.l_spat_region, self.l_order_rules, self.l_order_rules))


let's get rid of the commented code

Fixed. Thanks!

crvernon · 2023-01-04T12:34:30Z

demeter/process.py


-            self.config.logger.info("Saving land cover transition files for time step {0}...".format(self.step))
+         #   self.config.logger.info("Saving land cover transition files for time step {0}...".format(self.step))


Remove commented code

Fixed. Thanks!

crvernon · 2023-01-04T12:34:36Z

demeter/process.py


-            wdr.write_transitions(self.s, self.config, self.step, self.transitions)
+          #  wdr.write_transitions(self.s, self.config, self.step, self.transitions)


Remove commented code

Fixed. Thanks!

crvernon · 2023-01-04T12:35:45Z

demeter/process.py

            return wdr.lc_timestep_csv(self.config, self.step, self.s.final_landclasses, self.s.spat_coords, orig_spat_aez,
                                self.s.spat_region, self.s.spat_water, self.s.cellarea, self.s.spat_ludataharm,
-                                self.config.metric, self.config.tabular_units, self.write_outputs)
+                                self.config.metric, self.config.tabular_units, self.write_outputs,write_ncdf,self.sce,self.res,write_csv,self.regrid_res)


to conform to Python's PEP8 style req, leave space after commas. E.g., write_ncdf, self.sce

Fixed. Thanks!

crvernon · 2023-01-04T13:06:51Z

processing_scripts/convert_csv_col_to_ncdf_subdata.py

@@ -0,0 +1,97 @@
+
+import xarray as x


import xarray using it's well known alias to avoid confusion: import xarray as xr

I think this file itself is redundant since this has been functionalized. I'l take this out.

crvernon · 2023-01-04T13:08:07Z

processing_scripts/convert_csv_col_to_ncdf_subdata.py

@@ -0,0 +1,97 @@
+


import order of packages should follow PEP 8 style requirements (https://peps.python.org/pep-0008/#imports)

I think this file itself is redundant since this has been functionalized. I'l take this out.

crvernon · 2023-01-04T13:08:30Z

processing_scripts/demeter_processing.py

@@ -0,0 +1,9 @@
+import demeter


same comment as above

Fixed. Thanks

crvernon · 2023-01-04T13:09:45Z

setup.py

@@ -24,6 +24,7 @@ def readme():
                      'pandas~=1.2.4',


your requirements file has these versions pinned e.g., ==, I prefer this method where >= is used if possible.

Fixed. Thanks!

crvernon · 2023-01-04T13:10:42Z

.github/workflows/build.yml

@@ -31,6 +31,7 @@ jobs:
            pip install pandas==1.2.4


see comment about pinning in setup.py file review

Fixed. Thanks!

kanishkan91 added 3 commits October 10, 2022 14:52

Update processing script

5f0b138

Remove numba dependency

eb60521

kanishkan91 requested a review from crvernon October 10, 2022 19:10

kanishkan91 marked this pull request as draft October 10, 2022 19:10

kanishkan91 requested review from marideeweber and brinday October 10, 2022 19:11

kanishkan91 added 2 commits October 17, 2022 18:07

Updates to demeter processing

7fde097

1. Assign threads dynamically when applying convultion filter 2. Add ncdf_conversion to native demeter code base 3. Update mapping files for IM3 CLM team 4. Add xarray to setup

Reverting changes to regions, basins file

95db8a1

kanishkan91 marked this pull request as ready for review October 18, 2022 00:18

kanishkan91 added 19 commits October 19, 2022 21:10

Move helper function outside class.

9486634

See Summary below for details

03daf04

1. Adding code so that ncdfs can be generated endogenously by setting a constant in config file 2. Bumping up python version to 3.8 3. Adding documentation for handle_single_pft() function

Update setup.py

2064508

Update setup.py

df00bc0

Update build.yml

7ff76fb

Update setup.py

ef8825a

Merge branch 'demeter_1.3.1' of https://github.com/JGCRI/demeter into…

8eb7324

… demeter_1.3.1

Update README.md

35aa2ab

Parameterizing gcam regions, basins file selection

0c4a855

Merge branch 'demeter_1.3.1' of https://github.com/JGCRI/demeter into…

2df3d51

… demeter_1.3.1

Update gcam_regions_32.csv

9f5d5b9

Adding sample supplement for new demeter v 1.3.1 workflow

5e12975

Update install_supplement.py

9fd8e68

Make writing ncdfs and csvs independent

7d4dfb6

Updating pickle files to protocol 4

29f3c28

Update writer.py

d5d0b94

Make write ncdf default and csv optional for testing framework

63fa56f

Adding ability to downscale to CONUS resolution.

5e66104

Add re grid resolution code

b168c43

Update install_supplement.py

73006a3

marideeweber approved these changes Nov 19, 2022

View reviewed changes

crvernon requested changes Jan 4, 2023

View reviewed changes

kanishkan91 added 2 commits January 6, 2023 02:47

Latest documentation using sphinx

3196916

New docs

349779f

kanishkan91 temporarily deployed to github-pages January 6, 2023 07:50 — with GitHub Pages Inactive

kanishkan91 temporarily deployed to github-pages January 6, 2023 10:17 — with GitHub Pages Inactive

Create .nojekyll.txt

95fc51c

kanishkan91 temporarily deployed to github-pages January 6, 2023 10:22 — with GitHub Pages Inactive

Create nojekyll file

88322e5

kanishkan91 temporarily deployed to github-pages January 6, 2023 10:23 — with GitHub Pages Inactive

fix nojekyll

ee9585f

kanishkan91 temporarily deployed to github-pages January 6, 2023 10:36 — with GitHub Pages Inactive

Adding changes based on C Vernon's review

0ef1c0c

kanishkan91 temporarily deployed to github-pages January 8, 2023 04:04 — with GitHub Pages Inactive

kanishkan91 merged commit 9aeac52 into main Jan 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates for version 1.3 #33

Updates for version 1.3 #33

kanishkan91 commented Oct 10, 2022 •

edited

Loading

codecov-commenter commented Oct 17, 2022 •

edited

Loading

marideeweber left a comment

crvernon left a comment

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023

crvernon Jan 4, 2023

kanishkan91 Jan 7, 2023


		self.config.logger.info("Saving land cover transition files for time step {0}...".format(self.step))
		# self.config.logger.info("Saving land cover transition files for time step {0}...".format(self.step))


		wdr.write_transitions(self.s, self.config, self.step, self.transitions)
		# wdr.write_transitions(self.s, self.config, self.step, self.transitions)

Updates for version 1.3 #33

Updates for version 1.3 #33

Conversation

kanishkan91 commented Oct 10, 2022 • edited Loading

codecov-commenter commented Oct 17, 2022 • edited Loading

Codecov Report

marideeweber left a comment

Choose a reason for hiding this comment

crvernon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kanishkan91 commented Oct 10, 2022 •

edited

Loading

codecov-commenter commented Oct 17, 2022 •

edited

Loading