Reimplement: mamba env create -f env.yml #633

davidbrochart · 2020-12-15T13:26:44Z

Currently:

mamba env create -f env.yml

falls back to conda.
We could reimplement it with:

mamba create -n env_name packages_in_env.yml -c channels_in_env.yml

by parsing the YAML file. What do you think?

wolfv · 2020-12-16T17:05:39Z

the issue I can see is that this behavior does not match conda which (afaik) doesn't implement YAML environments under the conda create command.

davidbrochart · 2020-12-16T18:13:08Z

What I mean is that whatever the file specified after -f is, we get the package name, packages and channels, and use them in mamba create. It should be the same behavior, right?

vv111y · 2021-02-24T12:19:14Z

I almost exclusively create envs via conda env create -f foo.yaml
Will mamba provide its functionality for this use case anytime soon?

wolfv · 2021-02-24T12:40:41Z

@vv111y we have the same command. It's just only the solving that is done inside mamba, everything else is done by conda in this case.

Also, we're working towards better command line interfaces in micromamba.

vv111y · 2021-02-24T13:24:06Z

okay so we will see some acceleration?
ADD: just tried it for one case and yes that was much quicker. If this is the typical performance then it definitely makes a difference - Thank you.

akaszynski · 2021-04-02T01:49:32Z

It would still be ideal to create directly from a file, but one alternative is to update an existing env:

mamba env update -n <your-env> --file environment.yml

jucor · 2021-07-05T09:48:44Z

Thanks @akaszynski for the solution, creating an empty environment with the right name then using mamba to update it from the environment.yml worked like a charm.

vnlitvinov · 2021-07-14T11:55:44Z

Isn't it already working as intended? I've just created a new environment running mamba env create -f env.yml with mamba from latest minimamba release...

davidbrochart · 2021-07-14T12:04:45Z

It uses mamba to solve, but conda to download and install the packages.

- install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166.

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in #166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com>

* Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in #166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in #166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in #166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

janxkoci · 2022-02-01T13:01:35Z

Sorry, but if updating an env works (with mamba env update -n foo -f foo.yml), what exactly is missing for mamba env create -f foo.yml to be fully implemented in mamba? Most of the relevant code (e.g. parsing yml files, mentioned in the first post) should already be in place for env update to work, right?

Unless mamba env update also uses conda for the actual download and installation?

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format * remove --disable-epsv option from curl calls Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format * remove --disable-epsv option from curl calls * Feature/eulerian storm track (#312) * added feature/eulerian-storm-track by Jeyavinoth Jeyaratnam * ready to work on eulerian-storm-tracker * changing settings.json for my eulerian-strom-track * copying over new code * working version of eulerian_storm_tracker; needs more finer changes * adding wheeler kiladis to my test run; index.html has a bug when only a single POD is run * trying out eulerian_storm_track * eulerian storm track works, but now have to filter out the topography and create zonal means * create topo file for obs data * added zonal means * added labels * cleaned the eulerian_storm_track.yml file * updated the pdf * removed old front_detection codde * removed old front_detection codde * manually editted the yml env file * updated the documentation * removed etc_composites code from this branch * added MDTF_Doc in remove * removed README.md from main folder * removed eulerian_readme.md * deleted remove/MDTF_.pdf * removed unnecessary files; added rst doc file * removed default_jj.jsonc * removed the seam on the plots using add_cyclic_point * Re-merged the v3 main branch from NOAA MDTF * removed import os from plotter.py * removed unused variables n imports suggested by LGTM * removed commented out codes in all files * removed commented code; removed .yml & default_jj.jsonc * removed mdtf_settings.jsonc * Feature/precip buoy diag (#115) * some changes to framework code * some changes to framework code * Edits * included preprocessing * Early version of working POD * resolved merge conflict * included preprocessing * Early version of working POD * fixed merge issues * removed ghost comment * fixed conda_init.sh * removed hard coded paths * fixed issues from lgtm * except block now does not catch BaseException * fixed BaseExcept handling * removed intermediate driver script * contents of /src now match develop * removed documentation pdf * removed unsused dependencies: networkx, seborn, netCDF4 * added package version numbers * removed nco from dependencies * modified settings file to include dimensions * removed path variables from code; now assuming they are specified in settings file * converted single to double quotes in subprocess.run * comment in settings file about sigma coords * one * updated figures and documentation * Create index.md * Delete docs directory * removed pdf and MDTF-diagnostics directory Co-authored-by: Fiaz Ahmed <fiaz@jupyter.atmos.ucla.edu> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com> Co-authored-by: Jeyavinoth Jeyaratnam <jeyavinoth@users.noreply.github.com> Co-authored-by: Fiaz Ahmed <15623379+ahmedfiaz@users.noreply.github.com> Co-authored-by: Fiaz Ahmed <fiaz@jupyter.atmos.ucla.edu>

* Revert "Update with current development efforts" * Fix FTP links in README.md Github.com does not show hyperlinks with the ftp:// protocol, even if they're explicitly specified. Work around this by writing out the full URL in the document. * Fix FTP links in README.md Fixes NOAA-GFDL/MDTF-diagnostics#7 Github.com does not show hyperlinks with the ftp:// protocol, even if they're explicitly specified. Work around this by writing out the full URL in the document so users can find data. * Merge updates into main branch (#320) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format * remove --disable-epsv option from curl calls Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com> * Correct precipiation variable name Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

* Merge python3_base yaml update into main (#358) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format * remove --disable-epsv option from curl calls * Feature/eulerian storm track (#312) * added feature/eulerian-storm-track by Jeyavinoth Jeyaratnam * ready to work on eulerian-storm-tracker * changing settings.json for my eulerian-strom-track * copying over new code * working version of eulerian_storm_tracker; needs more finer changes * adding wheeler kiladis to my test run; index.html has a bug when only a single POD is run * trying out eulerian_storm_track * eulerian storm track works, but now have to filter out the topography and create zonal means * create topo file for obs data * added zonal means * added labels * cleaned the eulerian_storm_track.yml file * updated the pdf * removed old front_detection codde * removed old front_detection codde * manually editted the yml env file * updated the documentation * removed etc_composites code from this branch * added MDTF_Doc in remove * removed README.md from main folder * removed eulerian_readme.md * deleted remove/MDTF_.pdf * removed unnecessary files; added rst doc file * removed default_jj.jsonc * removed the seam on the plots using add_cyclic_point * Re-merged the v3 main branch from NOAA MDTF * removed import os from plotter.py * removed unused variables n imports suggested by LGTM * removed commented out codes in all files * removed commented code; removed .yml & default_jj.jsonc * removed mdtf_settings.jsonc * Feature/precip buoy diag (#115) * some changes to framework code * some changes to framework code * Edits * included preprocessing * Early version of working POD * resolved merge conflict * included preprocessing * Early version of working POD * fixed merge issues * removed ghost comment * fixed conda_init.sh * removed hard coded paths * fixed issues from lgtm * except block now does not catch BaseException * fixed BaseExcept handling * removed intermediate driver script * contents of /src now match develop * removed documentation pdf * removed unsused dependencies: networkx, seborn, netCDF4 * added package version numbers * removed nco from dependencies * modified settings file to include dimensions * removed path variables from code; now assuming they are specified in settings file * converted single to double quotes in subprocess.run * comment in settings file about sigma coords * one * updated figures and documentation * Create index.md * Delete docs directory * removed pdf and MDTF-diagnostics directory Co-authored-by: Fiaz Ahmed <fiaz@jupyter.atmos.ucla.edu> * Fix Wheeler-Kiladis PDO (#326) * Revert "Update with current development efforts" * Fix FTP links in README.md Github.com does not show hyperlinks with the ftp:// protocol, even if they're explicitly specified. Work around this by writing out the full URL in the document. * Fix FTP links in README.md Fixes NOAA-GFDL/MDTF-diagnostics#7 Github.com does not show hyperlinks with the ftp:// protocol, even if they're explicitly specified. Work around this by writing out the full URL in the document so users can find data. * Merge updates into main branch (#320) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific …

- install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166.

* Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166. * Add temp extremes distshape to CI (NOAA-GFDL#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu>

- install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166.

* Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166. * Add temp extremes distshape to CI (NOAA-GFDL#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu>

- install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166.

* Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166. * Add temp extremes distshape to CI (NOAA-GFDL#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu>

- install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166.

* Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (mamba-org/mamba#633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL#166. * Add temp extremes distshape to CI (NOAA-GFDL#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format * remove --disable-epsv option from curl calls * Initial commit of strat-trop coupling eddy heat flux POD * Add pressure level scalar_coord_template to ta * Minor edits to rst docs * Minor edits to rst docs * Plot title fixes * Updates to documentation regarding interpretation of results * Remove unnecessary imports and pass * Cleaned up variable names and removed hardcoded data paths; added extra comments for users to adapt script for themselves * Added docstrings to functions * Used environment variables for the data paths * Add documentation about the scope of the POD, and the climate/weather implications related to strat-trop coupling * Fix addition of Kidston reference in POD docs Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: lettie_roach <lettie.roach@gmail.com> Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com>

* add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Indicate which settings are required, in website and CLI help * Do more validation on input paths * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting <John.Krasting@noaa.gov> removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach <lettie.roach@gmail.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * parent 22c6dc7ec7fc7834aec5ef8f6dc19d9ef6b3f3e2 author wrongkindofdoctor <20195932+wrongkindofdoctor@users.noreply.github.com> 1642625995 -0500 committer wrongkindofdoctor <20195932+wrongkindofdoctor@users.noreply.github.com> 1663615542 -0400 parent 22c6dc7ec7fc7834aec5ef8f6dc19d9ef6b3f3e2 author wrongkindofdoctor <20195932+wrongkindofdoctor@users.noreply.github.com> 1642625995 -0500 committer wrongkindofdoctor <20195932+wrongkindofdoctor@users.noreply.github.com> 1663605547 -0400 add example case configuration yaml file with ensemble case setup and environment and optional settings in default_tests.jsonc added placeholder comments for future yaml parser calls to cli.py added multirun config template with case list containing ensemble members and a pod list separate from the case list add scratch file for testing multirun routines clean up cli.py fixed data_manager typos fixed spacing and added todos for yaml parser to filesystem.py changed multirun template case names to use synthetic CMIP datasets changed example settings file time frequency to day to work w/synthetic CMIP data remove extraneous call from cli.py reconfigured data_manager DataSourceBase and parent classes to accept a pod name and initialize a POD object with case-related atts rather than a case attribute redefined pod wkdir paths in core.py redefined pods att and fixed env_vars definition in DataSourceBase made a note to get the date_range att properly defined in the pod struct during the pod_setup process added pod name argument to enviroment_manager SubprocessRuntimeManager init method added notes to diagnostic.py add pod_name arguments to ExplicitFileDataSource init and super defs define self.pods in ExplicitFileDataSource init trying to figure out how to handle path_variable and dest_paths for multiple cases messing with path_variable and dest_path instantions. Currently broken add pod args to GFDL_GCP_FileDataSourceBase init method added cases dict to DataSourceBase init method that should prob replate the atts contents added case_varlist dict to Diagnostic class init method to hold varlist data for each case define case_varlist in Diagnostic.setup routine need to propagate case_varlist to variable setup routine and dependencies define case_varlist varlist entries using copy of pod varlist in diagnostic setup redefine self.atts as attribute class from the final case in case list, since we need the convention att later attach varlists for each case and loop through the cases to call setup var on each case varlist to define the dest_path for each case in setup_pod reworked data_manager setup_var call to properly define the date_range att for each case var added date_range key to self.cases in DataSourceBase init working on getting preprocessor to work with varlists attached to each case in wrapped_edit_request need to resolve weirdness with dest_path assignment--having trouble sorting out dummy pod varlist dest_path from case varlist dest_path move pod.case_varlist definition to Diagnostic.from_struct routine instead of copying it from the initialized pod.varlist in the setup reorder loops in data_manager setup to work with pod.case_varlist as varlist objects instead of varlist dicts do some pep8 formatting star reorganizing query_data, iter_vars, and iter_vars_only in data_manager to reference case_varlists add case_name arguments to iter_vars and iter_vars_only added some notes and placeholder test code for getting varlist info ported to case_varlist added notes to core.py replaced pod_name with case_name in main pod log info added comments to diagnostic.py created a CaseVarlist class that inherits from Varlist cleaned up formatting and added comments to util/basic.py added comments to cli.py reformat pod.case_varlist and CaseVarlist class definition in diagnostic.py * revert structural changes to src and sites modules added --data_type cli argument for single run and multi-run modes added self.data_type attribute to to MDTFFramework class added logic block to separate single-run functionality from multirun functionality to core.py main defined data_type = mutli_run in multirun_config_template.jsonc fixed multirun logic in core.py added module for multirun data manager classes and methods cleaned up data_manager.py formatting added notes on super() and MRO info to data_manager_multirun.py created placeholder multirun datasource classes that inherit from data_manager and datasource classes copied case instantiation from single to multi run block in core.py changed setup_multirun to get_pod_config_multirun and set it up to return a pod instance using information from the last case instance in the previous multirun case list loop changed multirun attribute to a boolean defined multirun attribute for pod object need to redirect diagnosticClass.from_config to a multirun pod setup class method need to implement aforementioned class method messing around some more with data_manager trying to break out the varlist added line to test yaml to dump log for debugging change cat to printf in test yaml added varlist and diagnostic entries to multirun module added pod arg to core.case_setup in multirun block added call to get_pod_config_multirun in data_manager multirun setup mode will try adding refs to multirun diagnostic and varlist classes. Eventually want a varlist with list/dicts of paths for each case (I think) added some multirun varlist classes, and tried to make a copy routine for varlist entries that doesn't work moved multirun datasource classes to data_manager so there's no weird circular importing tried modding varlist vars for multirun using botched copy method. Maybe altering the varlist directly. remove unused modules test instantiating a varlist for multirun and replacing path_variable and dest_path atts tried moving varlist path_variable and dest_path to post_init, but this appends empty fields that don't get parsed at runtime * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Indicate which settings are required, in website and CLI help * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting <John.Krasting@noaa.gov> * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate <chiaweih@email.arizona.edu> Co-authored-by: Chia-Wei Hsu <chiaweh2@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: tsjackson-noaa <thomas.jackson@noaa.gov> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson <tom.jackson314@gmail.com> Co-authored-by: Arielle Catalano <acat2@ggr-ch-412-ac.psu.ds.pdx.edu> * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Feature/eulerian storm track (#312) * added feature/eulerian-storm-track by Jeyavinoth Jeyaratnam * ready to work on eulerian-storm-tracker * changing settings.json for my eulerian-strom-track * copying over new code * working version of eulerian_storm_tracker; needs more finer changes * adding wheeler kiladis to my test run; index.html has a bug when only a single POD is run * trying out eulerian_storm_track * eulerian stor…

i thiiiink this should address the mamba-takes-forever-on-github- actions crap -- at least, it worked locally. but double-check; i may need to do the thing of creating an empty env and then updating that from the yaml (see mamba-org/mamba#633 (comment))

* DEP: try pinning skbio >= 0.5.8 #68 * BLD: see about using mamba for CI should be a lot faster, based on experiences with it in wotplot * DEP/BLD: fix install i thiiiink this should address the mamba-takes-forever-on-github- actions crap -- at least, it worked locally. but double-check; i may need to do the thing of creating an empty env and then updating that from the yaml (see mamba-org/mamba#633 (comment)) * DEV/DOC: small updates re: mamba, conda * DEV/DOC: words * DOC: more detailed installation instructions - mention mamba (close #50) - mention install-from-source as backup option * DOC: try to fix table formatting in readme * DOC: you know what maybe we don't need that table * DOC: troubleshooting installation section * DOC: mention #70 in install instruction sec * DOC: words

nelsoncardenas · 2023-10-01T19:44:18Z

I would like to use Mamba to replace Conda as my environment manager, but not having the option to use an environment.yml file is a significant drawback

jonashaag · 2023-10-01T20:42:05Z

???

janxkoci · 2023-10-02T06:50:33Z

@nelsoncardenas I installed mambaforge, where mamba is the default package manager, but it still also includes conda. In fact I'm not aware of any way of installing mamba that won't also give you conda along with it. (Edit: maybe micromamba, but to me that's a separate thing from mamba anyway.)

Personally, I use mamba for everything except those few things that still don't work. That's probably also why they still pack conda with it..

janxkoci · 2023-10-02T06:58:15Z

@NumesSanguis:

When creating a VM with JupyterLab on the Google Cloud Platform, the conda base environment is already full with packages. Trying to install Mamba here results in an over 2 hour wait with the following output:

Can you try with:

conda install --freeze-installed -c conda-forge mamba

This should avoid updating/downgrading existing packages and just give you compatible mamba version (if there is such version).

NumesSanguis · 2023-10-03T00:26:37Z

conda install --freeze-installed -c conda-forge mamba

@janxkoci Currently not on a project that this is needed, but will let you know once I've been able to test that.

nelsoncardenas · 2023-10-05T19:11:09Z

Sorry, I'm a new user and English is not my native language, so I can miss understand the situation. I was having problems when I was trying to create an environment because it took a lot of time.

My solution was to uninstall mamba and conda and install again conda, but I changed the default solver to be based on mamba.

wiwa · 2024-08-26T23:43:55Z

Extremely non-obvious that mamba env create falls back to conda... not even using libmamba solver. I am now using mamba env create -f env.yml --solver=libmamba to get around conda being killed.

jjerphan · 2024-10-08T11:46:37Z

As of mamba 2.0, mamba does not depend on conda anymore, and this issue must have been resolved.

mamba env create -n env_name -f env_specification.yml

can be used.

phue mentioned this issue Feb 19, 2021

Failure resolving conda envs in parallel nextflow-io/nextflow#1819

Closed

phue mentioned this issue Apr 12, 2021

Add support for using mamba with conda nextflow-io/nextflow#2030

Merged

adriendelsalle added the status::need-triage New feature proposal that have not been reviewed. label Jun 4, 2021

tsjackson-noaa mentioned this issue Aug 27, 2021

Improve mamba usage in CI NOAA-GFDL/MDTF-diagnostics#272

Merged

11 tasks

linogaliana mentioned this issue Oct 31, 2021

Utiliser mamba pour accélérer GHA? linogaliana/python-datascientist-docker#2

Closed

spietras mentioned this issue Nov 2, 2021

Mamba spietras/rules_conda#2

Closed

wrongkindofdoctor mentioned this issue Mar 23, 2022

Merge updates into main branch (#320) wrongkindofdoctor/MDTF-diagnostics#43

Merged

11 tasks

tayabsoomro mentioned this issue Jul 10, 2022

use mamaba to create conda environments coadunate/MICAS#67

Open

taylorreiter mentioned this issue Jul 13, 2022

change quick start installation to mamba? dib-lab/charcoal#220

Open

npielawski mentioned this issue Sep 6, 2022

Environment stuck when installing mhamilton723/STEGO#49

Closed

cameronraysmith mentioned this issue Oct 21, 2022

add bioconda package recipe; improve installation in document pinellolab/pyrovelocity#7

Merged

loostrum mentioned this issue May 26, 2023

pre-commit CI and linter updates nlesc-recruit/cudawrappers#176

Merged

mikeymike555 mentioned this issue Dec 28, 2023

Issue on page /intro.html - mamba can't create environment from yaml file fneum/data-science-for-esm#16

Closed

cdalinghaus mentioned this issue Jun 14, 2024

environment_cpu.yaml does not exist (perhaps a typo or a missing channel constantinpape/torch-em#299

Closed

jjerphan closed this as completed Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reimplement: mamba env create -f env.yml #633

Reimplement: mamba env create -f env.yml #633

davidbrochart commented Dec 15, 2020 •

edited

Loading

wolfv commented Dec 16, 2020

davidbrochart commented Dec 16, 2020

vv111y commented Feb 24, 2021 •

edited

Loading

wolfv commented Feb 24, 2021

vv111y commented Feb 24, 2021 •

edited

Loading

akaszynski commented Apr 2, 2021

jucor commented Jul 5, 2021

vnlitvinov commented Jul 14, 2021

davidbrochart commented Jul 14, 2021

janxkoci commented Feb 1, 2022

nelsoncardenas commented Oct 1, 2023

jonashaag commented Oct 1, 2023

janxkoci commented Oct 2, 2023 •

edited

Loading

janxkoci commented Oct 2, 2023

NumesSanguis commented Oct 3, 2023

nelsoncardenas commented Oct 5, 2023

wiwa commented Aug 26, 2024 •

edited

Loading

jjerphan commented Oct 8, 2024

Reimplement: mamba env create -f env.yml #633

Reimplement: mamba env create -f env.yml #633

Comments

davidbrochart commented Dec 15, 2020 • edited Loading

wolfv commented Dec 16, 2020

davidbrochart commented Dec 16, 2020

vv111y commented Feb 24, 2021 • edited Loading

wolfv commented Feb 24, 2021

vv111y commented Feb 24, 2021 • edited Loading

akaszynski commented Apr 2, 2021

jucor commented Jul 5, 2021

vnlitvinov commented Jul 14, 2021

davidbrochart commented Jul 14, 2021

janxkoci commented Feb 1, 2022

nelsoncardenas commented Oct 1, 2023

jonashaag commented Oct 1, 2023

janxkoci commented Oct 2, 2023 • edited Loading

janxkoci commented Oct 2, 2023

NumesSanguis commented Oct 3, 2023

nelsoncardenas commented Oct 5, 2023

wiwa commented Aug 26, 2024 • edited Loading

jjerphan commented Oct 8, 2024

davidbrochart commented Dec 15, 2020 •

edited

Loading

vv111y commented Feb 24, 2021 •

edited

Loading

vv111y commented Feb 24, 2021 •

edited

Loading

janxkoci commented Oct 2, 2023 •

edited

Loading

wiwa commented Aug 26, 2024 •

edited

Loading