possible approaches to ingest workflow in application layer

[uturuncoglu]

I am stating to experiment uwtools as a part of the ufs-coastal-app and just wonder about the approaches to ingest workflow in the application layer. Do I need to go with condo environment to install it (specific version) or just add the workflow tools (main branch) as a submodule in the application level and start creating environment with the copy. Since, I am plaining to start building the specific configuration of ufs coastal (data atmosphere coupled with ocean) model with workflow, I am plaining to implement specific driver/s around it. So, probably I'll do some development also in the uwtools side. Let me know what is the best practice for the development environment. If there is one already for SRW etc. that I could as a reference and follow the same convention, that would be great.

[uturuncoglu]

FYI, @christinaholtNOAA

[uturuncoglu]

I am following https://uwtools.readthedocs.io/en/main/sections/contributor_guide/developer_setup.html to setup development environment. Here are my observations:

• conda install -y -c maddenp -c conda-forge --override-channels condev command is not working somehow and complaining about the Python versions. I could able to skip this by using alternate command conda install maddenp::condev
• Then, I tried to install development shell but this gives error like following bash: jq: command not found on Hercules. I did not try on any other platform yet but jq command seems available through the system. It seems this is related with the condev-shell and maybe conda install maddenp::condev did not install it correctly. Any idea?

[maddenp-noaa]

Hi @uturuncoglu. Thanks for starting the conversation. I'll do what I can to help.

First, on hera, in scratch space, I followed the documented procedure without problem. Here's a transcript of the commands I ran, with only (most of) the output deleted to save space:

/scratch2/BMC/wrfruc/Paul.Madden % wget https://github.com/conda-forge/miniforge/releases/download/24.3.0-0/Miniforge3-Linux-x86_64.sh
/scratch2/BMC/wrfruc/Paul.Madden % bash Miniforge3-Linux-x86_64.sh -bfp ./conda
/scratch2/BMC/wrfruc/Paul.Madden % source conda/etc/profile.d/conda.sh
/scratch2/BMC/wrfruc/Paul.Madden % conda activate
(base) /scratch2/BMC/wrfruc/Paul.Madden % conda install -y -c maddenp -c conda-forge --override-channels condev
(base) /scratch2/BMC/wrfruc/Paul.Madden % git clone https://github.com/maddenp-noaa/uwtools.git
(base) /scratch2/BMC/wrfruc/Paul.Madden % cd uwtools/
(DEV-uwtools) /scratch2/BMC/wrfruc/Paul.Madden/uwtools % uw --version
uw version 2.2.0 build 0

The differences vs the docs are that: 1. I installed conda into the scratch space instead of into my home directory, and 2. I cloned my fork of the uwtools repo instead of the official repo. This should all work the same with the official repo, but you won't be able to push to it. Check out the Fork and PR Model page for more information on this.

So, maybe you can share here step-by-step what you tried and where you ran into trouble.

As for jq, it is installed as a dependency of the condev package, so you should not need to make it available any other way:

(base) /scratch2/BMC/wrfruc/Paul.Madden/uwtools % which jq
/scratch2/BMC/wrfruc/Paul.Madden/conda/bin/jq

[uturuncoglu]

@maddenp-noaa Thanks for your help. Unfortunately, I have no access to Hera. Here are the steps that I am following on Hercules,

module load miniconda3/24.3.0
conda init
bash -- need to open new shell to have changes - the base env will be activated after this
conda create --prefix myenv
conda activate /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/myenv
conda install maddenp::condev
cd sorc/uwtools
export PATH=/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/myenv/bin:$PATH
make devshell

Of course, the patch will be different in your case. Please also note that, I am using the conda installation provided by the system.

[maddenp-noaa]

Ah, sorry, Hercules... Hera... my Greek history is rusty.

module load miniconda3/24.3.0

Unless this is a conda installation that you own, this will not work. You have to be able to write to it. Is this yours, or is it supplied for all users on Hercules? If the latter, you'll need to follow our instructions and install your own Miniforge and use it.

conda init

You should not run this: This is what is placing that code in your ~/.bashrc file to run in every new shell. If you need to use more than one conda installation on a given system, you need to load exactly one of them into any given shell. You can make aliases or bash functions for this if it's helpful, but all that really needs to be done (assuming your shell is bash) is:

source /path/to/conda/installation/etc/profile.d/conda.sh
conda activate

That will put you in the base environment of the conda installation at /path/to/conda/installation. If you want to use a different conda, you can do something similar but in a new shell.

[uturuncoglu]

@maddenp-noaa This is supplied by the system. The steps that I am following is working fine with the system provided conda. It just requires some extra steps that is not found in the documentation. Do you still want me to have my own conda installation?

[maddenp-noaa]

I haven't tried installing condev into a non-base environment, so am happy to hear that that is working for you. I think there are some cases, like building and installing a uwtools conda package locally, that may require that you be working from a base environment, but you may never need to do that, so I don't think you need to do anything differently for now.

[uturuncoglu]

Okay. I had to do export PATH=/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/myenv/bin:$PATH to access jq. I am not sure why but I could use it under my condo environment but not through the make file. So, maybe this needs to be documented somewhere.

[maddenp-noaa]

As mentioned above, jq is installed as a dependency of the condev package, so when you are able to install condev as described in the docs, you'll get jq on your PATH, too.

One thing this procedure assumes is that you do not already have some other conda installation activated in your shell. So, if perhaps your ~/.bashrc or other init files are activating a different conda installation -- maybe one created by a UFS app -- you might consider commenting those statements out, or putting them in a bash function that you can call when you want it. For example, in my ~/.bashrc I have this function:

ca() { source ~/sw/conda/etc/profile.d/conda.sh && conda activate && test -n "$*" && conda $@; }

I can then type ca when I want to activate my personal conda installation. You could consider something similar if you have conda-related code in your init file(s) that should only be executed in specific circumstances.

[uturuncoglu]

Okay. Thanks. I found following in my .bashrc. I think this is different environment. I am not sure how this is installed. Maybe that was the reason of having issue with jq command. I could try again by deactivating it.

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/apps/spack-managed/gcc-11.3.1/miniconda3-24.3.0-avnaftwsbozuvtsq7jrmpmcvf6c7yzlt/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/apps/spack-managed/gcc-11.3.1/miniconda3-24.3.0-avnaftwsbozuvtsq7jrmpmcvf6c7yzlt/etc/profile.d/conda.sh" ]; then
        . "/apps/spack-managed/gcc-11.3.1/miniconda3-24.3.0-avnaftwsbozuvtsq7jrmpmcvf6c7yzlt/etc/profile.d/conda.sh"
    else
        export PATH="/apps/spack-managed/gcc-11.3.1/miniconda3-24.3.0-avnaftwsbozuvtsq7jrmpmcvf6c7yzlt/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

[maddenp-noaa]

Yes, I'd suggesting commenting that out, or putting it inside a shell function that you can run when you want it. That code is installed by conda in certain circumstances, and is probably useful when you've got only a single conda installation you need to work with on a particular machine, and always want it to be available.

[uturuncoglu]

@maddenp-noaa Okay. That sounds fine. Thanks for your help.

[uturuncoglu]

BTW, I am not seeing anything about building specific configuration of the model (i.e. ufs-coastal). Is there any example for it? Maybe I need to create new app under src/uwtools/apps/ but not sure. Maybe new class like COASTAL could include section like build and all the UFS level build commands are triggered from there. Any idea or example might help to go in right direction at this point since there is no any information in the documentation or I could not find it.

[maddenp-noaa]

I'm only partially familiar with the UFS apps, but is ufs-coastal an existing app, or one that you'll be building?

In any case, the general idea is that uwtools will provide tools to help manage configuration files, define workflows (Rocoto now, hopefully ecFlow later), and configure and execute individual NWP programs (what we call "components"). It is the uwtools "drivers" that are responsible for that last part. They're meant to take a specially-formatted YAML input file for configuration, then create a run directory containing all the input and configuration files the component needs to run, and optionally execute the component either locally or by submitting a batch job. We currently have drivers for 10 NWP component programs and are actively writing more.

I suppose that a ufs-coastal app is or will be composed of a number of different executable programs. Which ones? We may have drivers for some, but may need to assist in building more.

Two other crucial components of an app are its build and its workflow.

uwtools isn't involved in the build: The app's executables need to be available, and their paths communicated to uwtools in its configuration files. We suspect that apps will probably also want to supply YAML configuration files for the uwtools drivers that are close to what app users will need, but that may need to be lightly edited for particular experiments -- for example, setting the locations of specific run directories.

As for the workflow, we also currently assume that apps will provide that. uwtools has a tool for generating Rocoto workflow definitions, we we could talk about that as a way forward for ufs-coastal. uwtools also has some simple built-in workflow functionality that could be accessed via custom Python modules that could reside in an app. We could discuss strategies there, too.

[uturuncoglu]

@maddenp-noaa ufs-coastal is a fork of ufs-weather-model but it includes bunch of new ocean model components. You could see it in the following repository: https://github.com/oceanmodeling/ufs-coastal probably we will rename it as ufs-weather-model soon. The build system is totally same with ufs-weather-model. We just define set of new app flags to build new ocean components and their configurations. I think it would be nice to have some level of capability to build the model and create the executable. It would be nice to have it as a part of the workflow but if you think that that needs to be done. It is fine. Then, I think application level that uses workflow to configure the run directory also needs to include extra wrapper around the workflow to cover build part. Anyway, maybe I am missing something in here. So, if there is some prototype for SRW etc. that I could look, it would be helpful to follow exiting conventions.

[uturuncoglu]

@maddenp-noaa BTW, is there any example YAML file that can be supplied to the workflow to run standalone FV3 configuration? That could be a good starting point.

[uturuncoglu]

One more question, how can I make available new driver? I tried to clean existing development environment and creating again (make devshell) but still my custom driver is not shown in uw -h command. Do I need to do anything else?

[uturuncoglu]

Okay. Now I could able to make it available. I also edited src/uwtools/cli.py, src/uwtools/strings.py and created src/uwtools/api/cdeps.py and src/uwtools/drivers/cdeps.py. The CDEPS driver will be responsible to create required namelist files and maybe staging required input but not sure at this point.

[maddenp-noaa]

It sounds like you're making good progress. We should probably talk at some point about the fairly strict code-quality standard in uwtools, which involve auto-formatting, linting, typechecking, unit-testing, and schema validation of input configs. It's a lot, but we want to serve a large number of different use cases and so have to follow some rigorous procedures to keep everything working. If you get to a point where you'd like to show some code, please direct us to a GitHub repo; or we could schedule some time to talk and plan together.

[uturuncoglu]

@maddenp-noaa yes that would be great. BTW, I am still not sure how can I add model build step to the workflow. Is there any example for it? How this will be handled with the SRW etc.

[uturuncoglu]

@maddenp-noaa It seems that execution part is required in the config.yaml. In my case, the CDEPS data component driver is not expected. This is just a component (has its own library) and part of the ufs-weather-model executable. But, CDEPS driver will be responsible to create bunch of nemslist files (like datm_in in NML format and datm.streams in ESMF config format - similar to model_configure or ufs_configure) to execute the executable. So, in my case CDEPS has no executable but it will create just namelist files to prepare the run directory. I wonder if there is a way or approach to handle this kind of tasks. At this point I just put following to the config.yaml to bypass the error but I am not sure it is fine or not,

  execution:
    executable: None

[uturuncoglu]

@maddenp-noaa Okay , I have some progress to bring cdeps support but I am getting some error and might need your help. You could see the additional development to bring cdeps support in here: https://github.com/oceanmodeling/uwtools/pull/new/feature/cdeps

In this case, I am using following config.yaml file,

cdeps:
  execution:
    executable: None
  atm:
    update_values:
      config:
        datamode: ATMMESH
  run_dir: /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools

and issuing following command,

uw cdeps atm --cycle 20240201T18:00:00 --config-file config.yaml

but getting error like following,

[2024-05-03T17:45:39]     INFO Validating config against internal schema cdeps
[2024-05-03T17:45:39]     INFO 0 UW schema-validation errors found
[2024-05-03T17:45:39]     INFO Validating config against internal schema platform
[2024-05-03T17:45:39]     INFO 0 UW schema-validation errors found
Traceback (most recent call last):
  File "/home/tufuk/.conda/envs/DEV-uwtools/bin/uw", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools/src/uwtools/cli.py", line 80, in main
    sys.exit(0 if modes[args[STR.mode]](args) else 1)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools/src/uwtools/cli.py", line 928, in _dispatch_to_driver
    return execute(
           ^^^^^^^^
  File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools/src/uwtools/utils/api.py", line 71, in execute_cycle
    return _execute(
           ^^^^^^^^^
  File "/work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools/src/uwtools/utils/api.py", line 154, in _execute
    getattr(obj, task)()
  File "/home/tufuk/.conda/envs/DEV-uwtools/lib/python3.12/site-packages/iotaa/__init__.py", line 424, in __iotaa_task__
    assets = next(g)
             ^^^^^^^
StopIteration

So, it seems that I am missing something in here since modes[args[STR.mode]](args) returns empty (I put print statement there). At this point, I am expecting to create datm_in file (fortran namelist file) that has only one namelist option like datamode = "ATMMESH". I am not sure how namelist group could be added. My ultimate intermediate goal is to create file looks like following,

&datm_nml
  datamode = "ATMMESH"
  factorfn_data = "null"
  factorfn_mesh = "null"
  flds_co2 = .false.
  flds_presaero = .false.
  flds_wiso = .false.
  iradsw = 1
  model_maskfile = "INPUT/irene_roms_grid_rho_ESMFmesh.nc"
  model_meshfile = "INPUT/irene_roms_grid_rho_ESMFmesh.nc"
  nx_global = 242
  ny_global = 106
  restfilm = "null"
  export_all = .true.
/

So, if above works then I'll extend the implementation to support also other data models and their namelist files. Maybe the issue related with the executable: None in the config file but not sure. I wonder if it is possible to create namelist files without executing anything. I could put something in here to execute a dummy command but not sure.

BTW, is workflow has some level of capability to populate the required data from different sources like FTP etc. I have couple of Python scripts that supports getting data using FTP, wget, AWS S3 etc. So, it would be helpful to have that capability in the workflow but I could also add them to the one level up in the application layer.

[uturuncoglu]

@maddenp-noaa BTW, along the way I had to dig into documentation to find some key information or look around the code. So, there is not strait forward way to find all those information and easy to follow example to bring new driver. Maybe this could be a future improvement in the documentation. Just and idea and feedback.

[uturuncoglu]

@maddenp-noaa The error is disappeared with slightly modified version of your atm() and _model_namelist_file() implementation but I have issue with the consistency between config.yaml and scheme file. I think following scheme file

{
  "properties": {
    "cdeps": {
      "additionalProperties": false,
      "properties": {
        "execution": {
          "$ref": "urn:uwtools:execution"
        },
        "datm_nml": {
          "additionalProperties": false,
          "properties": {
            "datamode": {
              "type": "string"
            },
            "factorfn_data": {
              "type": "string"
            }
          },
          "required": [
            "datamode",
            "factorfn_data"
          ],
          "type": "object"
        },
        "run_dir": {
          "type": "string"
        }
      },
      "required": [
        "execution",
        "datm_nml",
        "run_dir"
      ],
      "type": "object"
    }
  },
  "type": "object"
}

is not consistent with

cdeps:
  execution:
    executable: /usr/bin/date
    batchargs:
      nodes: 1
      stdout: /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools/runscript.out
      walltime: "00:05:00"
  datm_nml:
    datamode: ATMMESH
    factorfn_data: 'null'
  run_dir: /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools

The command works but the result datm_in is empty. I am not sure what is the missing piece even if I check other drivers and their schemes along with config files. Also, based on selection of the data model with command like uw cdeps atm or uw cdeps ocn I need to validate specific version of the scheme file. So, the scheme file will include all the sections for all components (atm, ocn, wav) and the config file could provide only one of them or their combination to be generic since CDEPS provides multiple data component.

[maddenp-noaa]

If the driver does anything beyond complaining about YAML validation errors, then the problem is not the relationship between the .jsonschema and .yaml files. If the YAML does not conform to the schema, uw will stop immediately and not execute any task methods. So, if your driver is creating an empty datm_in file, the problem is either with the content of the YAML or the way the driver code is using it. (It's possible that YAML conforms to the schema, i.e. is syntactically right, but does not provide the right information to the driver, i.e. is semantically wrong.)

So, I think I would need to see your code. I am looking at your branch, and don't see updates to https://github.com/oceanmodeling/uwtools/blob/feature/cdeps/src/uwtools/drivers/cdeps.py -- can you push those so I can have a look?

[uturuncoglu]

@maddenp-noaa There is no any validation error in my case but somehow it does not working. As you said, maybe it is not providing right information. Anyway, I have just pushed the changes. Thanks.

[maddenp-noaa]

I think I see the problem. The _create_user_updated_config() function our Driver class provides, which you call from your _model_namelist_file() method, expects a specific YAML structure, described here (those docs are generic and apply to YAML files, namelist files, etc, so note the comments below about namelist structure specifically.) This is the structure that you'll see enforced in our various existing .jsonschema files, and probably what you want to enforce in cdeps.jsonshcema, too.

Basically, you can have a base_file: and/or an update_values: block under your datm_nml: key. If you provide a base_file: key, its value should be a path to a file, which will be read as a starting point. Then, if you provide an update_values: key, its value will be used to update the config read from base_file:. If you don't provide base_file: in your config.yaml, then the update_values: should be sufficient to build a correct namelist file. If you don't provide update_values:, then base_file: will be used as-is.

Since you're dealing with namelists here, the YAML data under update_values: should have a top level of namelist names, and key-value blocks under each of those. So, for datm_nml, you might have something like

datm_nml:
    update_values:
        nl1:
            k1: v1
            k2: v2
        nl2:
            k3: v3

which would result in a namelist file like

&nl1
  k1=v1
  k2=v2
/
&nl2
  k3=v3
/

Looking at the .jsonschema files for other existing drivers may help clarify this. You could also look at some of the example UW YAML configs under docs/shared and see how these correspond to the JSON Schema files.

You might want to start with a very simple, permissive .jsonschema file, then make it strict later once we have everything working. For example, you might spec datm_nml as just "datm_nml": {"type": "object"} to start with, which will accept any YAML mapping / Python dict structure. Later you can (and we can help) make this more strict, specifying what kind of entries are and are not required, which namelist names and key names are accepted, what kinds of values are accepted, etc.

I hope this helps. I should mention, in case it's not obvious yet, that the task methods the uwtools drivers define are idempotent: Once they create their output, they won't create it again as long it remains "ready" -- which in the case of most disk files just means "the file exists". So, to have e.g. an empty namelist file recreated, you should delete the bad file first, then re-run the driver, which will notice that it is no longer "ready" and create it again -- hopefully with content this time.

[uturuncoglu]

@maddenp-noaa Okay. Thanks for your great help. It is more clear to me. As a baggier user with limited Jason experience, it took time for me to realize requirements. Maybe in the future scheme generation could be more strait forward for the users like me. Anyway, I could able to create the nml file now. Additionally, I add initial data ocean component support. Using following config.yaml,

cdeps:
  execution:
    executable: /usr/bin/date
    batchargs:
      nodes: 1
      stdout: /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/uwtools/runscript.out
      walltime: "00:05:00"
  atm:
    update_values:
      datm_nml:
        datamode: ATMMESH
        factorfn_data: 'null'
  ocn:
    update_values:
      docn_nml:
        datamode:
        model_maskfile: 'INPUT/mesh.nc'

I could able to create both datm_in and docn_in with following commands without any issue,

rm datm_in; uw cdeps atm --cycle 20240201T18:00:00 --config-file config.yaml
rm docn_in; uw cdeps ocn --cycle 20240201T18:00:00 --config-file config.yaml

I also pushed the latest version of the scheme file and driver/cdeps to my branch if you want to see. I think after this point, I'll enhance scheme file to cover all the namelist options as well as their restrictions. As you might notice, I am providing executable: /usr/bin/date to satisfy uwtools requirements at this point. I'll also look at that part based on your suggestions.

Once I finalize this part, I'll also implement support to create other namelist files (in ESMF config format like model_configure) for defining the stream (for your reference: https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/parm/datm.streams.era5.IN) to run CDEPS data components. Anyway, thanks again for your great support. It was impossible to reach this point without your help.

[uturuncoglu]

@maddenp-noaa I just wonder if there is a way to write the namelist file in ESMF config format. The one of the namlist file which is required for CDEPS data component is stream definition file in ESMF config format. In UFS, both model_configure and also ufs_configure are also in ESMF config format. So, if you want to support coupled configurations those needs to be created/modified by workflow. As I see, fv3 driver is using YAML to write out model_configure. This might work for model_configure but not for ufs_configure since it has more sophisticated structure (https://github.com/ufs-community/ufs-weather-model/blob/develop/tests/parm/ufs.configure.s2swa.IN) like run sequences etc. So, I think that needs to be implemented if there is no any existing implementation. Any idea?

[uturuncoglu]

@maddenp-noaa Thanks again for your help. Yes, it makes sense to maintain those templates. I could able to use ninja template to create cdeps data components related ESMF config formatted namelist files. I am working on extending it to all the namelist options and trying to finalize the implementation. I think after those we will have fully functional CDEPS deriver in the workflow. I am not sure what do you think but this will be required for also other coupled applications since CDEPS data components are shared with other configurations under UFS Weather Model. I wonder if you want me to create a draft PR in uwtools side for this additional driver. Anyway, that is totally fine for me to maintain this in out fork too. Just let me know your preference. Thanks again for your great help. I think Jinja templates make easy to handle complex namelist files.

[uturuncoglu]

@maddenp-noaa I wonder if there is a way to merge two task under different task name. At this point, in the driver I have,

    @task
    def atm(self):
        """
        The atm namelist file.
        """
        fn = "datm_in"
        yield self._taskname(f"namelist file {fn}")
        path = self._rundir / fn
        yield asset(path, path.is_file)
        yield None
        path.parent.mkdir(parents=True, exist_ok=True)
        self._model_namelist_file("atm_in", path)

and

    @task
    def atm_stream(self):
        fn = "datm.streams"
        yield self._taskname(f"namelist file {fn}")
        path = self._rundir / fn
        yield asset(path, path.is_file)
        yield file(path=Path(self._driver_config["atm_streams"]["base_file"]))
        render(
            input_file=Path(self._driver_config["atm_streams"]["base_file"]),
            output_file=path,
            values_src=self._driver_config["atm_streams"],
        )

but actually they just both creating different namelist file for data atmosphere. So, it would be nice to run both of them in the same time rather than issuing different commands for each one of them. If so, maybe we don't need to define atm and atm_stream as tasks abut have top level task that calls both.

[maddenp-noaa]

@uturuncoglu To your question about task methods atm and atm_stream, we have been modeling that need -- the need to define several run-time requirements together -- with a task, in each driver, called provisioned_run_directory, which is created using the @tasks decorator (different than @task). Here's an example from another driver, and the @tasks documentation. If, for example, you had, in your CDEPS driver, something like

@tasks
def provisioned_run_directory(self):
    """
    Run directory provisioned with all required content.
    """
    yield self._taskname("provisioned run directory")
    yield [self.atm(), self.atm_stream()]

Then you could run, from the command line,

uw cdeps provisioned_run_directory [other arguments]

To obtain a run directory containing both the namelist file and the configure file. But one could also obtain the namelist-file or configure-file assets independently by running either

uw cdeps atm [other arguments]
uw cdeps atm_stream [other arguments]

Later, you'd also have a run @task method that would require the provisioned_run_directory task (probably among others, like a runscript task) and would execute the component. This way, users can request any subset of the total functionality -- from a complete run (which implies a provisioned run directory and then everything that that directory contains), or just a provisioned run directory without running anything, or just any of the individual assets. In visual form:

I'll answer your other questions as soon as I can.

By the way, you'll find that the uwtools documentation and code is explicit/literal about the word "namelist": It always refers to Fortran namelists, and files containing Fortran namelists. We refer to any other type/format of configuration file with different and specific terminology. I think maybe you mean "namelist" more generically, meaning "some kind of configuration file", but we'll eventually want to be clear about that. We have specific tools/functions/packages, etc. to deal with namelists and namelist files, and different tools/functions/packages for other configuration types/formats. For actual Fortran namelists, we would generally not use Jinja2 templates, but rather create/modify those directly with namelist-specific tools.

[uturuncoglu]

@maddenp-noaa Okay. Thanks for detailed answer. Probably this will solve my issue. Let me try it. Yes, I am using namelist in more general context. It is good to know that it has a specific meaning in uwtools to prevent confusion in my end. Thanks again.

[uturuncoglu]

@maddenp-noaa JFYI, I could able to create stream and namelist files using following template and config file. https://github.com/oceanmodeling/ufs-coastal-app/tree/feature/uw/templates I still need to clean the implementation and also support other data components. I might make them pattern in the JASON scheme file. Thanks again for your help.

[uturuncoglu]

@maddenp-noaa I am trying to combine couple of steps of UFS coastal with UWTools and I just wonder if you have a exiting workflow that coupled couple of steps such as prep and running the code etc. Is there any generic driver or way to trader a python code or command through the workflow. For example, let's assume that I have a script that gets the data for the case, Is it possible to trigger it (and define as a dependency for other steps or derives) without writing any specific driver. I also wonder about the defining dependencies among steps. Anyway, if you could point me couple of place in the documentation, I could try to look at them and try to understand.

[maddenp-noaa]

I think you have a number of options:

• If you have a step that involves data staging, you could consider using the uw file tool. CLI use is documented here, and API (i.e. from within other Python code) use here. There's also a demo video.
• You could use Rocoto to define a workflow that defines steps including prep steps (staging files, perhaps with uw file or with existing scripts; executing drivers like cdeps to provision config files etc. to a run directory, etc.), running a coupled models, doing post-processing, etc. uwtools provides the uw rocoto tool for creating Rocoto XML workflow documents from UW YAML. That's documented here (CLI) and here (API). @christinaholtNOAA has a lot of experience here and may be able to provide additional ideas.
• For exploration or simpler needs, you could define a workflow using the iotaa module, which is included with uwtools and used internally to implement its drivers and its file tool. I'll give a simple example below.

Rocoto is mature, well-supported, and probably a good choice.

Here's an example of a toy workflow using just uwtools (and the iotaa package that it provides as a dependency):

import subprocess
from datetime import datetime
from functools import cache
from pathlib import Path

from iotaa import asset, task, tasks

from uwtools.api import cdeps, config, file, logging

logging.use_uwtools_logger()


@tasks
def cdeps_assets(cfgfile):
    """Provision CDEPS assets to the run directory"""
    cycle = datetime.fromisoformat(cfg(cfgfile)["cycle"])
    cdeps_driver = cdeps.CDEPS(config=cfgfile, cycle=cycle)
    yield "CDEPS assets"
    yield [cdeps_driver.atm_nml(), cdeps_driver.ocn_nml()]


@tasks
def linked_files(cfgfile):
    """Link data files into the run directory"""
    yield "Linked files"
    yield file.Linker(config=cfgfile, keys=["link"]).go()


@tasks
def provisioned_rundir(cfgfile):
    """Provision the run directory will all required content"""
    yield "Provisioned run directory"
    yield [cdeps_assets(cfgfile), linked_files(cfgfile)]


@task
def run(cfgfile):
    """Run"""
    rundir = Path(cfg(cfgfile)["dir"]["run"])
    outfile = rundir / "linecount"
    yield "Run"
    yield asset(outfile, outfile.is_file)
    yield provisioned_rundir(cfgfile)
    subprocess.run("cat %s/* | wc -l >%s" % (rundir, outfile), shell=True)


@cache
def cfg(cfgfile):
    return config.get_yaml_config(cfgfile)

It imports some items from the uwtools API, uses the file module to symlink some data files into the run directory, uses the cdeps module to provision two namelist files, and runs a external programs via a subprocess to count the lines in all the files and write that number to an output file (simulating running something more interesting, like perhaps a coupled executable). The tasks are tied together via dependencies: The run task depends on provisioned_rundir, and provisioned_rundir depends on cdeps_assets and linked_files.

Here's an experiment.yaml to configure it:

cycle: 2024-07-26T12
dir:
  data: /tmp/data
  run: /tmp/run
cdeps:
  atm_in:
    update_values:
      datm_nml:
        datamode: GFS
        export_all: true
        factorfn_data: 'null'
        factorfn_mesh: 'null'
        flds_co2: false
        flds_presaero: false
        flds_wiso: false
        iradsw: 1
        model_maskfile: '{{ dir.run }}/mask1.nc'
        model_meshfile: '{{ dir.run }}/mesh1.nc'
        nx_global: 100
        ny_global: 100
        restfilm: 'null'
  ocn_in:
    update_values:
      docn_nml:
        datamode: some_supported_mode2
        import_data_fields: none
        model_maskfile: '{{ dir.run }}/mask2.nc'
        model_meshfile: '{{ dir.run }}/mesh2.nc'
        nx_global: 150
        ny_global: 150
        restfilm: 'null'
        skip_restart_read: true
        sst_constant_value: -1
  rundir: '{{ dir.run }}'
link:
  '{{ cdeps.atm_in.update_values.datm_nml.model_maskfile }}': '{{ dir.data }}/mask1.nc'
  '{{ cdeps.atm_in.update_values.datm_nml.model_meshfile }}': '{{ dir.data }}/mesh1.nc'
  '{{ cdeps.ocn_in.update_values.docn_nml.model_maskfile }}': '{{ dir.data }}/mask2.nc'
  '{{ cdeps.ocn_in.update_values.docn_nml.model_meshfile }}': '{{ dir.data }}/mesh2.nc'

You can query the workflow for available tasks:

$ iotaa workflow.py --tasks
Tasks in workflow.py:
  cdeps_assets
    Provision CDEPS assets to the run directory
  linked_files
    Link data files into the run directory
  provisioned_rundir
    Provision the run directory will all required content
  run
    Run

Execute the run task:

$ iotaa workflow.py run experiment.yaml
[2024-07-26T23:33:30]     INFO Run: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO Run: Checking requirements
[2024-07-26T23:33:30]     INFO Provisioned run directory: Checking requirements
[2024-07-26T23:33:30]     INFO Validating config against internal schema cdeps
[2024-07-26T23:33:30]     INFO 0 UW schema-validation errors found
[2024-07-26T23:33:30]     INFO CDEPS assets: Checking requirements
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file datm_in: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file datm_in: Checking requirements
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file datm_in: Requirement(s) ready
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file datm_in: Executing
[2024-07-26T23:33:30]     INFO 0 UW schema-validation errors found
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file datm_in: Final state: Ready
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file docn_in: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file docn_in: Checking requirements
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file docn_in: Requirement(s) ready
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file docn_in: Executing
[2024-07-26T23:33:30]     INFO 0 UW schema-validation errors found
[2024-07-26T23:33:30]     INFO 20240726 12Z cdeps namelist file docn_in: Final state: Ready
[2024-07-26T23:33:30]     INFO Linked files: Checking requirements
[2024-07-26T23:33:30]     INFO Validating config against internal schema files-to-stage
[2024-07-26T23:33:30]     INFO 0 UW schema-validation errors found
[2024-07-26T23:33:30]     INFO File links: Checking requirements
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask1.nc -> /tmp/data/mask1.nc: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask1.nc -> /tmp/data/mask1.nc: Checking requirements
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask1.nc -> /tmp/data/mask1.nc: Requirement(s) ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask1.nc -> /tmp/data/mask1.nc: Executing
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask1.nc -> /tmp/data/mask1.nc: Final state: Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh1.nc -> /tmp/data/mesh1.nc: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh1.nc -> /tmp/data/mesh1.nc: Checking requirements
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh1.nc -> /tmp/data/mesh1.nc: Requirement(s) ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh1.nc -> /tmp/data/mesh1.nc: Executing
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh1.nc -> /tmp/data/mesh1.nc: Final state: Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask2.nc -> /tmp/data/mask2.nc: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask2.nc -> /tmp/data/mask2.nc: Checking requirements
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask2.nc -> /tmp/data/mask2.nc: Requirement(s) ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask2.nc -> /tmp/data/mask2.nc: Executing
[2024-07-26T23:33:30]     INFO Link /tmp/run/mask2.nc -> /tmp/data/mask2.nc: Final state: Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh2.nc -> /tmp/data/mesh2.nc: Initial state: Not Ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh2.nc -> /tmp/data/mesh2.nc: Checking requirements
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh2.nc -> /tmp/data/mesh2.nc: Requirement(s) ready
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh2.nc -> /tmp/data/mesh2.nc: Executing
[2024-07-26T23:33:30]     INFO Link /tmp/run/mesh2.nc -> /tmp/data/mesh2.nc: Final state: Ready
[2024-07-26T23:33:30]     INFO Run: Requirement(s) ready
[2024-07-26T23:33:30]     INFO Run: Executing
[2024-07-26T23:33:30]     INFO Run: Final state: Ready

Inspect run directory:

$ tree /tmp/run
/tmp/run
├── datm_in
├── docn_in
├── linecount
├── mask1.nc -> /tmp/data/mask1.nc
├── mask2.nc -> /tmp/data/mask2.nc
├── mesh1.nc -> /tmp/data/mesh1.nc
└── mesh2.nc -> /tmp/data/mesh2.nc

0 directories, 7 files
$ cat /tmp/run/linecount 
26

Running the driver again is a no-op, since all the work is complete:

$ iotaa workflow.py run experiment.yaml
[2024-07-26T23:34:38]     INFO Run: Initial state: Ready

But if a declared output is removed, the workflow can recreate it:

$ rm -v /tmp/run/linecount 
removed '/tmp/run/linecount'
$ iotaa workflow.py run experiment.yaml
[2024-07-26T23:35:35]     INFO Run: Initial state: Not Ready
[2024-07-26T23:35:35]     INFO Run: Checking requirements
[2024-07-26T23:35:35]     INFO Provisioned run directory: Checking requirements
[2024-07-26T23:35:35]     INFO Validating config against internal schema cdeps
[2024-07-26T23:35:35]     INFO 0 UW schema-validation errors found
[2024-07-26T23:35:35]     INFO CDEPS assets: Checking requirements
[2024-07-26T23:35:35]     INFO Linked files: Checking requirements
[2024-07-26T23:35:35]     INFO Validating config against internal schema files-to-stage
[2024-07-26T23:35:35]     INFO 0 UW schema-validation errors found
[2024-07-26T23:35:35]     INFO File links: Checking requirements
[2024-07-26T23:35:35]     INFO Run: Requirement(s) ready
[2024-07-26T23:35:35]     INFO Run: Executing
[2024-07-26T23:35:35]     INFO Run: Final state: Ready

Less work is done than the first time because the namelist files and symlinks are still ready to use.

The above code relies on some features in uwtools that we haven't released yet, but will soon.

Let me know what you think and we can discuss more.

[uturuncoglu]

@maddenp-noaa Thanks. Lots of information in here to digest. Let me look at them and get back to you if I have any question. It seems that a Python script needs to be written for iotaa. Do I also need to write a one for Rocoto? I also wonder how much work do I need to convert exiting workflow from iotaa to Rocoto (or maybe in the future ecflow)?

[maddenp-noaa]

Rocoto requires an XML document to describe workflows. It could be written directly, or you could use the uwtools rocoto tool (CLI: uw rocoto -- docs here, API: uwtools.api.rocoto -- docs here) to construct it from more-familiar YAML. In addition to the workflow document itself, you may (or may not) also need some scripts (bash, Python, whatever) to execute various steps of the workflow.

If you know that you need to use a specific workflow manager eventually, I think you'd be better off starting with it immediately. But I think that defining the workflow steps, figuring out dependencies, etc., can be the hardest part of workflow construction, so I don't think that much work is wated if you move from one workflow engine to another: Steps defined for one engine probably translate, conceptually at least, to another, though the syntax changes.

An interesting property of iotaa is that, once its tasks are defined, you could call them from an external workflow manager, too. For example, suppose that you defined an iotaa module w.py consisting of a prep task and a forecast task that depends on prep. When you run iotaa w.py forecast, the forecast task runs the prep task automatically because it depends on it. But you could also run iotaa w.py prep by itself to do the prep work, then call iotaa w.py forecast: forecast would not re-run prep, because it would see that the prep output was already available. Similarly, you could later upgrade to Rocoto or ecFlow as your workflow engine, and have them call iotaa w.py prep and iotaa w.py forecast separately, re-using the Python code you've already written to execute those tasks, plus get all the additional benefits of those more-powerful engines, like good batch system support, workflow visualization, etc.

[uturuncoglu]

@maddenp-noaa Thanks. I think iotaa is the way to go at this point.

JFYI, I had also issue about installing uwtools from local clone. I waslooking to the following documentation https://uwtools.readthedocs.io/en/main/sections/user_guide/installation.html#build-the-uwtools-package-locally. It is building without issue but then I could not access it. Here are the steps that I am following on Hercules,

module load miniconda3/24.3.0
conda create --prefix python/envs/mybase
conda activate /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/python/envs/mybase
conda install -c conda-forge --override-channels conda-build conda-verify
cd sorc/uwtools/
conda build recipe -c conda-forge --override-channels

Then, conda search -c local --override-channels uwtools command could not find the build. Then, I used conda install -c ../../python/envs/mybase/conda-bld uwtools command to install it (python/envs/mybase is my exiting conda environment). I am not sure but that part of the documentation could be extended little bit to make it more clear. Thanks again for your help.

[maddenp-noaa]

Hi @uturuncoglu Generally, you need to run conda build in the conda installation's base environment, not in an environment you have created, as you've done above. But I think that will not work unless you have write access to the conda installation, which you probably (hopefully!) do not when using a system module.

Two ways forward are:

1. Install your own miniforge as outlined here and use that, in which case the docs you cite above should work as described, or
2. Either install the package via the explicit filesystem path shown toward the end of the conda build output (for me: conda install /home/maddenp/sw/conda/envs/mybase/conda-bld/noarch/uwtools-8.8.8-py_0.tar.bz2) or use a path channel pointing to the conda-bld directory under your mybase environment (for me: conda install -c /home/maddenp/sw/conda/envs/mybase/conda-bld/ uwtools). conda should give priority to the local package vs the one available in the conda-forge channel, but be sure that, when conda says The following NEW packages will be INSTALLED:, it shows that it will be installing the locally-built package.

I'll add something to the docs to try to make this clearer.

[maddenp-noaa]

And if you're installing the package into the same environment in which you built it, you can use the simpler command

conda install -c $CONDA_PREFIX/conda-bld/ uwtools

[uturuncoglu]

@maddenp-noaa Thanks for all your help. I have other couple of minor questions to make DATM+SCHSIM configure working with the workflow.

• Is there any example for NML template. I am providing schism nml file as template but it is not updating dt. Maybe temple file need to have certain syntax to work with uwtools.

schism:
  namelist:
    template_file: /work2/noaa/nems/tufuk/COASTAL/ufs-weather-model/tests/run_dir/coastal_ike_shinnecock_atm2sch_intel/param.nml
    template_values:
      dt: 300
  rundir: '{{ dir.run }}'

• I also need to create ufs.configure and model_configure. I think those were controlled by FV3 driver but not sure. Actually, those files are not tight into any component since they related with top level ESMF/NUOPC driver. This is definitely correct for ufs.configure and partly for model_configure since model_configure has also FV3 specific information. But, I think those are controlled by the top level driver. I am not sure how that will be constructed but do you have any feeling about it. I could copy them over from another directory for now but we might need to configure them through the config.yaml file for more flexibility. Any idea?

[maddenp-noaa]

1. I think I see the issue with the SCHISM driver and will provide a hotfix shortly. You'll need to rebuild the package after a git pull again -- sorry about this, but we really appreciate your efforts here as the first user of this code.
2. We have briefly discussed this question about ufs.configure / model_configure internally but haven't dealt with it yet. I think you're right that this is currently provided by the FV3 driver, and also that it needs to be made more general. I will try to raise this question for discussion at the public uwtools discussion meeting at 11am MDT today. I see that you were invited to that, so if you happen to be able to attend you could provide some input.

[uturuncoglu]

@maddenp-noaa Thanks. Yes, I'll try to join.

[uturuncoglu]

BTW, in the example workflow. The Linker part of code is facing like following,

AttributeError: module 'uwtools.api.file' has no attribute 'Linker'

So, i checked the file and found that there is a link() function. So, I change the code little bit like,

    yield file.link(config=cfgfile, keys=["link"]).go()

But this also requires to provide target_dir but I am not sure how will I get it. In config.yaml I defined like following,

link:
  '{{ dir.data }}/albedo.gr3': albedo.gr3 

and I am expecting to use current run dir but since link part has no information like that maybe I also need to add rundir: '{{ dir.run }}' to link section. BTW, it would be nice to add rundir: '{{ dir.run }}' in the top level. So, all the steps might use the same one if there Is no any specialization under specific task. Just an idea.

[maddenp-noaa]

I think you might have pulled main and built your package just before the PR merged that made this available. At the moment, on main:

>>> from uwtools.api.file import Linker
>>> help(Linker)
Help on class Linker in module uwtools.file:

class Linker(FileStager)
 |  Linker(config: Union[dict, pathlib.Path, NoneType] = None, target_dir: Optional[pathlib.Path] = None, cycle: Optional[datetime.datetime] = None, leadtime: Optional[datetime.timedelta] = None, keys: Optional[list[str]] = None, dry_run: bool = False) -> None
...

I'm sorry, I should have realized, when answering your question about the local conda build above, that you were trying using this latest functionality -- I could have warned you to wait a little. I think if you git pull in main now and rebuild and re-install the package, you should have everything you need to follow the demo I gave above.

In re: the specific construction of the YAML, yes, there are many possibilities there, including a top-level run: key, in which case you might not need dir.run at all. My thought was just to have a top-level dir: section that could contain multiple directories, but they (rundir and datadir) could also just stand alone at the top of the config. It's really up to you how best to organize that for your app.

Our team is continuing to implement features meant to support coupled drivers. I am currently working on a task that would make it possible for the YAML config sections belonging to components to omit some otherwise required items -- like rundir: for all driver types, and the execution: block for standalone (executable) drivers -- and have that information propagated by the block that belongs to the coupled component. For example, cdeps: requires a rundir: entry, but if its code is compiled into a coupled executable, we'd like for the rundir: specified for the coupled executable to also be used for CDEPS, and for the user not to have to keep repeating the same information. (In my demo above, I used Jinja2 to render a shared run-directory value into the cdeps: block, but it would be nice not to have to do that at all.) I mention this because I would hate for you to do a lot of work that we will soon, hopefully, make easier. If you like, we could schedule a short meeting soon to explain what we hope to implement in the next couple/few weeks.

[uturuncoglu]

@maddenp-noaa The link issue is solved after I update uwtools. Thanks.

[uturuncoglu]

@maddenp-noaa Thanks. Yes. I realized that my version have FileLinker rather than Linker but even if I used `FileLinker, it requires target directory. Do you think that it will gone if I update my version? UWtools is evolving too fast (I think this is good) and sometimes it is hard to catch. Yes, please keep me in the loop especially development of the top level driver related things. In the mean time, I could try to go further in the implementation of DATM+SCHSIM as much as possible. I could copy those required config files from some other path for now. Then, update the implementation when UWtools could handle these. Let me know if you have any other suggestion. Thanks again.

[maddenp-noaa]

@uturuncoglu The changes will slow down. We are in that delicate time when some of the drivers, ideas, etc. are getting their first exposure to real-world tests, so we are of course finding bugs and other things that should be improved while we still can. It will be harder to make breaking changes in the future.

For Linker, the target directory should now only be required if any of the destination paths are relative. If they are relative, then we have to know: Relative to what?

>>> from uwtools.api.logging import use_uwtools_logger
>>> use_uwtools_logger(verbose=False)
>>> from uwtools.api.file import Linker
>>> Linker(config={"fstab": "/etc/fstab", "passwd": "/etc/passwd"}).go()
...
uwtools.exceptions.UWConfigError: Relative path 'fstab' requires the target directory to be specified

If target_dir is specified, the relative paths are OK:

>>> Linker(config={"fstab": "/etc/fstab", "passwd": "/etc/passwd"}, target_dir="/tmp/run0").go()
...
[2024-08-01T17:00:36]     INFO Link /tmp/run0/fstab -> /etc/fstab: Final state: Ready
...
[2024-08-01T17:00:36]     INFO Link /tmp/run0/passwd -> /etc/passwd: Final state: Ready
[2024-08-01T17:00:36]     INFO File links: Final state: Ready

(I committed an update to main this morning that allows target_dir to be a plain str value instead of a pathlib Path value, so you might like to update to that. Otherwise, I could have specified target_dir=Path("/tmp/run0").)

It's also OK if absolute destination paths are provided (I'm targeting a new destination directory, /tmp/run1, this time):

>>> Linker(config={"/tmp/run1/fstab": "/etc/fstab", "/tmp/run1/passwd": "/etc/passwd"}).go()
...
[2024-08-01T17:01:35]     INFO Link /tmp/run1/fstab -> /etc/fstab: Final state: Ready
...
[2024-08-01T17:01:35]     INFO Link /tmp/run1/passwd -> /etc/passwd: Final state: Ready
[2024-08-01T17:01:35]     INFO File links: Final state: Ready

Instantiating Copier or Linker objects is useful if you want to use their go() tasks as dependencies of other tasks in a workflow-oriented use case. If you just want to copy or link files from a Python script, you can also use the simpler copy() or link() functions provided in uwtools.api.file. But I think you are indeed working with tasks, for Copier and Linker should be useful.

[uturuncoglu]

@maddenp-noaa I just wonder how can I submit job through the schedular. I define the workflow with iotaa and create a config file to run it but iotaa datm2sch.py run config.yaml command does not submit job for me. Here ar my workflow and config file.

import subprocess
from datetime import datetime
from functools import cache
from pathlib import Path
from iotaa import asset, task, tasks
from uwtools.api import file 
from uwtools.api import cdeps, schism, config, logging

logging.use_uwtools_logger()

@tasks
def cdeps_assets(cfgfile):
    """Provision CDEPS assets to the run directory"""
    cycle = datetime.fromisoformat(cfg(cfgfile)["cycle"])
    cdeps_driver = cdeps.CDEPS(config=cfgfile, cycle=cycle)
    yield "CDEPS assets"
    yield [cdeps_driver.atm_nml(), cdeps_driver.atm_stream()]

@tasks
def schism_assets(cfgfile):
    """Provision SCHISM assets to the run directory"""
    cycle = datetime.fromisoformat(cfg(cfgfile)["cycle"])
    schism_driver = schism.SCHISM(config=cfgfile, cycle=cycle)
    yield "SCHISM assets"
    yield [schism_driver.namelist_file()]

@tasks
def linked_files(cfgfile):
    """Link data files into the run directory"""
    yield "Linked files"
    yield file.Linker(config=cfgfile, keys=["link"]).go()

@tasks
def provisioned_rundir(cfgfile):
    """Provision the run directory will all required content"""
    yield "Provisioned run directory"
    yield [cdeps_assets(cfgfile), schism_assets(cfgfile), linked_files(cfgfile)]

@task
def run(cfgfile):
    """Run"""
    rundir = Path(cfg(cfgfile)["dir"]["run"])
    outfile = rundir / "linecount"
    yield "Run"
    yield asset(outfile, outfile.is_file)
    yield provisioned_rundir(cfgfile)
    subprocess.run("cat %s/* | wc -l >%s" % (rundir, outfile), shell=True)

@cache
def cfg(cfgfile):
    return config.get_yaml_config(cfgfile)

cycle: 2008-08-23T12
dir:
  data: /work2/noaa/nems/tufuk/COASTAL/ufs-weather-model/tests/run_dir/coastal_ike_shinnecock_atm2sch_intel
  run: /work2/noaa/nems/tufuk/COASTAL/datm2sch/run
  model: /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/sorc/ufs-weather-model
  bin: /work2/noaa/nems/tufuk/COASTAL/ufs-coastal-app/install/bin
cdeps:
  atm_in:
    update_values:
      datm_nml:
        datamode: ATMMESH
        export_all: true
        factorfn_data: 'null'
        factorfn_mesh: 'null'
        flds_co2: false
        flds_presaero: false
        flds_wiso: false
        iradsw: 1
        model_maskfile: 'INPUT/esmf_mesh.nc'
        model_meshfile: 'INPUT/esmf_mesh.nc'
        nx_global: 101
        ny_global: 101
        restfilm: 'null'
  atm_streams:
    streams:
      stream01:
        taxmode: limit 
        mapalgo: redist
        tinterpalgo: linear
        readmode: single
        dtlimit: 1.5
        stream_offset: 0
        yearFirst: 2008
        yearLast: 2008
        yearAlign: 2008
        stream_vectors: ['u', 'v']
        stream_mesh_file: 'INPUT/esmf_mesh.nc'
        stream_lev_dimname: 'null'
        stream_data_files: ['INPUT/wind_atm_fin_ch_time_vec_STR_fixed.nc' ]
        stream_data_variables: ['uwnd Sa_u10m', 'vwnd Sa_v10m', 'P Sa_pslv' ]
    template_file: stream.jinja2
  rundir: '{{ dir.run }}'
schism:
  namelist:
    template_file: /work2/noaa/nems/tufuk/COASTAL/ufs-weather-model/tests/run_dir/coastal_ike_shinnecock_atm2sch_intel/param.nml
    template_values:
      dt: 300
  rundir: '{{ dir.run }}'
link:
  '{{ dir.run }}/albedo.gr3': '{{ dir.data }}/albedo.gr3'
  '{{ dir.run }}/bctides.in': '{{ dir.data }}/bctides.in'
  '{{ dir.run }}/fd_ufs.yaml': '{{ dir.model }}/tests/parm/fd_ufs.yaml'
  '{{ dir.run }}/hgrid.gr3': '{{ dir.data }}/hgrid.gr3'
  '{{ dir.run }}/hgrid.ll': '{{ dir.data }}/hgrid.ll'
  '{{ dir.run }}/manning.gr3': '{{ dir.data }}/manning.gr3'
  '{{ dir.run }}/model_configure': '{{ dir.data }}/model_configure'
  '{{ dir.run }}/noahmptable.tbl': '{{ dir.data }}/noahmptable.tbl'
  '{{ dir.run }}/station.in': '{{ dir.data }}/station.in'
  '{{ dir.run }}/ufs.configure': '{{ dir.data }}/ufs.configure'
  '{{ dir.run }}/vgrid.in': '{{ dir.data }}/vgrid.in'
  '{{ dir.run }}/watertype.gr3': '{{ dir.data }}/watertype.gr3'
  '{{ dir.run }}/windrot_geo2proj.gr3': '{{ dir.data }}/windrot_geo2proj.gr3'
execution:
  batchargs:
    cores: 6 
    debug: True
    exclusive: True
    export: NONE
    jobname: my-job
    nodes: 1
    partition: hercules
    queue: batch
    rundir: '{{ dir.run }}'
    shell: /bin/bash
    stderr: '{{ dir.run }}/err'
    stdout: '{{ dir.run }}/out'
    tasks_per_node: 6
    walltime: 00:30:00
  envcmds:
    - module use '{{ dir.model }}/modulefiles'
    - module load ufs_hercules.intel
  executable: '{{ dir.bin }}/ufs_model'
  mpiargs:
    - "--export=ALL"
    - "--ntasks $SLURM_CPUS_ON_NODE"
    - "--account=nems"
  mpicmd: srun
  threads: 1

Maybe I need to also add some other function to my workflow file but not sure.

[uturuncoglu]

I also add following section but still same. I think I need to use uw execute command. I am trying to find an example.

platform:
 account: nems
 scheduler: slurm

[maddenp-noaa]

Is executable: '{{ dir.bin }}/ufs_model' the coupled executable program? If so, this workflow actually looks like a driver. iotaa can be used both inside a driver (for example to define all the items that need to go in a provisioned run directory, to create the component's runscript, to submit the runscript -- and to define the dependencies between them), but then also at a higher level to link multiple drivers together (for example, a workflow might link the execution of one or more prep-related drivers together, then run a model via its driver, then run a post-processing driver). We use iotaa tasks inside the existing uwtools drivers, but haven't yet used it to link drivers together -- so far, existing application have used Rocoto for this.

We have recently done work in uwtools to support creating drivers that are external to the uwtools codebase, and I think maybe that's what we should take advantage of here. In fact, we have had Coastal App in mind as our first use case of an external driver for a coupled executable. By creating an external driver that extends one of the uwtools Driver classes, we get some additional support for creating a runscript, populating it with Slurm directives, and submitting it to the batch system.

I'll take your code and config here and demonstrate this. More soon, but probably not before tomorrow.

[uturuncoglu]

@maddenp-noaa Sorry for late response. Yes, my executable has coupled configuration. It seems that external driver I the way to go at this point. Thanks for your help.

[maddenp-noaa]

@uturuncoglu I've got some time to work more on this. Could attach here your stream.jinja2 referenced by cdeps.atm_streams.template_file in the YAML config, and the SCHISM template file you list /work2/noaa/nems/tufuk/COASTAL/ufs-weather-model/tests/run_dir/coastal_ike_shinnecock_atm2sch_intel/param.nml?

[uturuncoglu]

@maddenp-noaa Thanks. Here is my stream.jinja2,

{% set stream_info = [] -%}
{% for key, val in streams.items() -%}
{% set _ = stream_info.append( key ) -%}
{% endfor -%}
stream_info: {{ stream_info | join(' ') }}
{% for key, val in streams.items() %}
taxmode{{'%02d' % loop.index}}: {{ val['taxmode'] }}
mapalgo{{'%02d' % loop.index}}: {{ val['mapalgo'] }} 
tInterpAlgo{{'%02d' % loop.index}}: {{ val['tinterpalgo'] }}
readMode{{'%02d' % loop.index}}: {{ val['readmode'] }}
dtlimit{{'%02d' % loop.index}}: {{ val['dtlimit'] }}
stream_offset{{'%02d' % loop.index}}: {{ val['stream_offset'] }}
yearFirst{{'%02d' % loop.index}}: {{ val['yearFirst'] }}
yearLast{{'%02d' % loop.index}}: {{ val['yearLast'] }}
yearAlign{{'%02d' % loop.index}}: {{ val['yearAlign'] }}
{% if val['stream_vectors'] is string -%}
stream_vectors{{'%02d' % loop.index}}: {{ val['stream_vectors'] }}
{% else -%}
stream_vectors{{'%02d' % loop.index}}: "{{ val['stream_vectors'] | join(':') }}"
{% endif -%}
stream_mesh_file{{'%02d' % loop.index}}: {{ val['stream_mesh_file'] }}
stream_lev_dimname{{'%02d' % loop.index}}: {{ val['stream_lev_dimname'] }}
{% if val['stream_data_files'] | length > 1 -%}
stream_data_files{{'%02d' % loop.index}}: ""{{ val['stream_data_files'] | join('" "') }}""
{% else -%}
stream_data_files{{'%02d' % loop.index}}: "{{ val['stream_data_files'] | first }}"
{% endif -%}
{% if val['stream_data_variables'] is string -%}
stream_data_variables{{'%02d' % loop.index}}: "{{ val['stream_data_variables'] }}"
{% else -%}
stream_data_variables{{'%02d' % loop.index}}: "{{ val['stream_data_variables'] | join('" "') }}"
{% endif -%}
{% endfor %}

and here is the param.nml

!parameter inputs via namelist convention.
!(1) Use ' ' (single quotes) for chars;
!(2) integer values are fine for real vars/arrays;
!(3) if multiple entries for a parameter are found, the last one wins - please avoid this
!(4) array inputs follow column major (like FORTRAN) and can spill to multiple lines
!(5) space allowed before/after '='

&CORE
!+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
! Core (mandatory) parameters; no defaults
!+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
! Pre-processing option. Useful for checking grid violations.
  ipre = 0 !Pre-processor flag (1: on; 0: off)

! Baroclinic/barotropic option. If ibc=0 (baroclinic model), ibtp is not used.
  ibc = 1 !Baroclinic option
  ibtp = 0 

  rnday = 20. !total run time in days
  dt = 200. !Time step in sec

! Grid for WWM (USE_WWM)
  msc2 = 24     !same as msc in .nml ... for consitency check between SCHISM and WWM
  mdc2 = 30     !same as mdc in .nml

! Define # of tracers in tracer modules (if enabled)
  ntracer_gen = 2 !user defined module (USE_GEN)
  ntracer_age = 4 !age calculation (USE_AGE). Must be =2*N where N is # of age tracers
  sed_class = 5 !SED3D (USE_SED)
  eco_class = 27 !EcoSim (USE_ECO): must be between [25,60]

! Global output controls
  nspool = 9 !output step spool
  ihfskip = 432 !stack spool; every ihfskip steps will be put into 1_*, 2_*, etc...

  nbins_veg_vert = 1
/

&OPT
!+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
! Optional parameters 
!+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
! Starting time
  start_year = 2008 !int
  start_month = 8 !int
  start_day = 23 !int
  start_hour = 0 !double
  utc_start = 0 !double

!-----------------------------------------------------------------------
! Coordinate option: 1: Cartesian; 2: lon/lat (hgrid.gr3=hgrid.ll in this case,
! and orientation of element is outward of earth)
!-----------------------------------------------------------------------
  ics = 2 !Coordinate option

!-----------------------------------------------------------------------
! Hotstart option. 0: cold start; 1: hotstart with time reset to 0; 2: 
! continue from the step in hotstart.nc
!-----------------------------------------------------------------------
  ihot = 0

!-----------------------------------------------------------------------
! Equation of State type used
! ieos_type=0: UNESCO 1980 (nonlinear); =1: linear function of T ONLY, i.e. 
! \rho=eos_b+eos_a*T, where eos_a<=0 in kg/m^3/C
!-----------------------------------------------------------------------
  ieos_type = 0
  ieos_pres = 0 !used only if ieos_type=0. 0: without pressure effects 
  eos_a = -0.1 !needed if ieos_type=1; should be <=0 
  eos_b = 1001. !needed if ieos_type=1

!-----------------------------------------------------------------------
! Main ramp option
!-----------------------------------------------------------------------
!  nramp = 1 !ramp-up option (1: on; 0: off)
  dramp = 1. !needed if nramp=1; ramp-up period in days

!  nrampbc = 0 !ramp-up flag for baroclinic force
  drampbc = 0. !not used if nrampbc=0

!-----------------------------------------------------------------------
! Method for momentum advection. 0: ELM; 1: upwind (not quite working yet)
!-----------------------------------------------------------------------
  iupwind_mom = 0

!-----------------------------------------------------------------------
! Methods for computing velocity at nodes. 
! If indvel=0, conformal linear shape function is used; if indvel=1, averaging method is used.
! For indvel=0, a stabilization method is needed (see below). 
!-----------------------------------------------------------------------
  indvel = 0
 
!-----------------------------------------------------------------------
! 2 stabilization methods, mostly for indvel=0.
! (1) Horizontal viscosity option. ihorcon=0: no viscosity is used; =1: Lapacian;
! =2: bi-harmonic. If ihorcon=1, horizontal viscosity _coefficient_ (<=1/8, related
! to diffusion number) is given in hvis_coef0, and the diffusion # 
! is problem dependent; [0.001-1/8] seems to work well.
! If ihorcon=2, diffusion number is given by hvis_coef0 (<=0.025).
! If indvel=1, no horizontal viscosity is needed. 
! (2) Shapiro filter (see below)
!
! For non-eddying regime applications (nearshore, estuary, river), two easiest options are: 
! (1) indvel=1, ishapiro=ihorcon=0, and any choices of inter_mom; 
! (2) indvel=0, ishapiro=1 (shapiro0=0.5), ihorcon=inter_mom=0.
! For applications that include eddying regime, refer to the manual.
!-----------------------------------------------------------------------
  ihorcon = 0
  hvis_coef0 = 0.025 !const. diffusion # if ihorcon/=0; <=0.025 for ihorcon=2, <=0.125 for ihorcon=1
!  cdh = 0.01 !needed only if ihorcon/=0; land friction coefficient - not active yet

!-----------------------------------------------------------------------
! 2nd stabilization method via Shapiro filter. This should normally be used 
! if indvel=ihorcon=0. To transition between eddying/non-eddying regimes, use
! indvel=0, ihorcon/=0, and ishapiro=-1 (shapiro.gr3).
!-----------------------------------------------------------------------
  ishapiro = 1 !on/off flag
  shapiro0 = 0.5 !Shapiro filter strength, needed only if ishapiro=1; max is 0.5 
  niter_shap = 1 !needed of ishapiro/=0 - # of iterations with Shapiro filter. Suggested: 1

!-----------------------------------------------------------------------
! Implicitness factor (0.5<thetai<=1).
!-----------------------------------------------------------------------
  thetai = 0.6 

!-----------------------------------------------------------------------
! If WWM is used, set coupling/decoupling flag. Not used if USE_WWM is distabled in Makefile
!       0: decoupled so 2 models will run independently; 
!       1: full coupled (elevation, vel, and wind are all passed to WWM); 
!       2: elevation and currents in wwm, no wave force in SCHISM;  
!       3: no elevation and no currents in wwm, wave force in SCHISM;
!       4: elevation but no currents in wwm, wave force in SCHISM;
!       5: elevation but no currents in wwm, no wave force in SCHISM;
!       6: no elevation but currents in wwm, wave force in SCHISM;
!       7: no elevation but currents in wwm, no wave force in SCHISM;
! Note that all these parameters must be present in this file (even though not used).
!-----------------------------------------------------------------------
  icou_elfe_wwm = 0 
  nstep_wwm = 1  !call WWM every this many time steps. If /=1, consider using quasi-steady mode in WWM
  iwbl = 0 !wave boundary layer formulation (used only if USE_WMM and 
           !icou_elfe_wwm/=0 and nchi=1. If icou_elfe_wwm=0, set iwbl=0): 
           !1-modified Grant-Madsen formulation; 2-Soulsby (1997)
  hmin_radstress = 1. !min. total water depth used only in radiation stress calculation [m]
!  nrampwafo = 1       !ramp-up option for the wave forces (1: on; 0: off)
  drampwafo = 1.      !needed if nrampwafo=1; ramp-up period in days
  turbinj = 0.15      !% of depth-induced wave breaking energy injected in turbulence (default: 0.15 (15%), as proposed by Feddersen, 2012)

!-----------------------------------------------------------------------
! Bed deformation option (0: off; 1: vertical deformation only; 2: 3D bed deformation). 
! If imm=1, bdef.gr3 is needed; if imm=2, user needs to update depth info etc
! in the code (not working for ics=2 yet).
!-----------------------------------------------------------------------
  imm = 0
  ibdef = 10 !needed if imm=1; # of steps used in deformation

!-----------------------------------------------------------------------
! Reference latitude for beta-plane approximation when ncor=1 (not used if ics=2)
!-----------------------------------------------------------------------
  slam0 = -124  !lon - not really used
  sfea0 = 45 !lat

!-----------------------------------------------------------------------
! Baroclinicity calculation in off/nearshore. The 'below-bottom' gradient
! is zeroed out if h>=h2_bcc (i.e. like Z) or uses const extrap
! (i.e. like terrain-following) if h<=h1_bcc(<h2_bcc) (and linear
! transition in between based on local depth)
!-----------------------------------------------------------------------
  h1_bcc = 50. ![m]
  h2_bcc = 100. ![m]; >h1_bcc

!-----------------------------------------------------------------------
! Hydraulic model option. If ihydraulics/=0, hydraulics.in 
! is required. This option cannot be used with non-hydrostatic model.
!-----------------------------------------------------------------------
  ihydraulics = 0

!-----------------------------------------------------------------------
! Point sources/sinks option (0: no; 1: on). If =1, needs source_sink.in (list of elements),
! vsource,th, vsink.th, and msource.th. Source/sinks can be specified at an elem more
! than once, and the code will accumulate the volumes, but for mass conc, one of
! the specified values will be taken (no summation).
!-----------------------------------------------------------------------
  if_source = 0
!  nramp_ss = 1 !needed if if_source=1; ramp-up flag for source/sinks
  dramp_ss = 2 !needed if if_source=1; ramp-up period in days

!-----------------------------------------------------------------------
! Horizontal diffusivity option. if ihdif=1, horizontal diffusivity is given in hdif.gr3
!-----------------------------------------------------------------------
  ihdif = 0 

!-----------------------------------------------------------------------
! Bottom friction. 
!           nchi=0: drag coefficients specified in drag.gr3; nchi=-1: Manning's 
!           formulation (even for 3D prisms). 
!           nchi=1: bottom roughness (in meters) specified in rough.gr3 (and in this case, negative
!           or 0 depths in rough.gr3 indicate time-independent Cd, not roughness!).
!           Cd is calculated using the log law, when dzb>=dzb_min; when dzb<dzb_min,
!           Cd=Cdmax*exp[dzb_decay*(1-dzb/dzb_min)], where Cdmax=Cd(dzb=dzb_min),
!           and dzb_decay (<=0) is a decay const specified below. We recommend dzb_decay=0
!           and may remove this constant in the future.
!           If iwbl/=0, nchi must =1.
!-----------------------------------------------------------------------
  nchi = -1
  hmin_man = 1. 

!-----------------------------------------------------------------------
! Coriolis. If ncor=-1, specify "rlatitude" (in degrees); if ncor=0,
! specify Coriolis parameter in "coricoef"; if ncor=1, model uses
! lat/lon in hgrid.ll for beta-plane approximation if ics=1, and in this case,
! the latitude specified in CPP projection ('sfea0') is used. If ncor=1 and ics=2,
! Coriolis is calculated from local latitude, and 'sfea0' is not used.
!-----------------------------------------------------------------------
  ncor = 1 !should usually be 1 if ics=2

!-----------------------------------------------------------------------
! Elevation initial condition flag for cold start only. If ic_elev=1, elev.ic (in *.gr3 format) is needed
! to specify the initial elevations; otherwise elevation is initialized to 0 everywhere 
!-----------------------------------------------------------------------
  ic_elev = 0

!-----------------------------------------------------------------------
! Elevation boundary condition ramp-up flag. =0: ramp up from 0; =1: ramp up from
! elev. values read in from elev.ic or hotstart.nc - if neither is present, from 0.
! This flag is mainly used to start the simulation from non-zero elev.
! The ramp-up period is same as 'dramp' below.
!-----------------------------------------------------------------------
  nramp_elev = 0

!-----------------------------------------------------------------------
! Optional inverse barometric effects on the elev. b.c.
! If inv_atm_bnd=1, the elev.'s at boundary are corrected by the difference
! between the actual atmos. pressure and a reference pressure (prmsl_ref below)
!-----------------------------------------------------------------------
  inv_atm_bnd = 0 !0: off; 1: on
  prmsl_ref = 101325. !reference atmos. pressure on bnd [Pa]

!-----------------------------------------------------------------------
! Initial condition for T,S. This value only matters for ihot=0 (cold start).
! If flag_ic(1:2)=1, the initial T,S field is read in from temp.ic and salt.ic (horizontally varying).
! If flag_ic(1:2)=2, the initial T,S field is read in from ts.ic (vertical varying).
! If ihot=0 && flag_ic(1:2)=2 || ibcc_mean=1, ts.ic is used for removing mean density profile.
! flag_ic(1) must =flag_ic(2)
!-----------------------------------------------------------------------
  flag_ic(1) = 1 !T
  flag_ic(2) = 1 !S

! initial conditions for other tracers.
! 1: needs inputs [MOD]_hvar_[1,2,...].ic ('1...' is tracer id); format of each file is similar to salt.ic;
!    i.e. horizontally varying i.c. is used for each tracer.
! 2: needs [MOD]_vvar_[1,2,...].ic. Format of each file (for each tracer in tis MOD) is similar to ts.ic
!    (i.e. level #, z-coord., tracer value). Verically varying i.c. is used for each tracer.
! 0: model sets own i.c. (EcoSim; TIMOR)
  flag_ic(3) = 1 !GEN (user defined module)
  flag_ic(4) = 1 !Age
  flag_ic(5) = 1 !SED3D
  flag_ic(6) = 1 !EcoSim
  flag_ic(7) = 1 !ICM
  flag_ic(8) = 1 !CoSINE
  flag_ic(9) = 1 !FIB
  flag_ic(10) = 1 !TIMOR
  flag_ic(11) = 1 !FABM

!-----------------------------------------------------------------------
! Settling vel [m/s] for GEN module (positive downward)
!-----------------------------------------------------------------------
  gen_wsett = 1.e-4

!-----------------------------------------------------------------------
! Mean T,S profile option. If ibcc_mean=1 (or ihot=0 and flag_ic(1)=2), mean profile
! is read in from ts.ic, and will be removed when calculating baroclinic force.
! No ts.ic is needed if ibcc_mean=0.
!-----------------------------------------------------------------------
  ibcc_mean = 0 

!-----------------------------------------------------------------------
! Max. horizontal velocity magnitude, used mainly to prevent problem in 
! bulk aerodynamic module
!-----------------------------------------------------------------------
  rmaxvel = 5.

!-----------------------------------------------------------------------
!  Following 2 parameters control backtracking
!-----------------------------------------------------------------------
!-----------------------------------------------------------------------
!  min. vel for invoking btrack and for abnormal exit in quicksearch
!-----------------------------------------------------------------------
  velmin_btrack = 1.e-4
!-----------------------------------------------------------------------
! Nudging factors for starting side/node - add noise to avoid underflow
! The starting location is nudged to: old*(1-btrack_nudge)+btrack_nudge*centroid
! Suggested value: btrack_nudge=9.013e-3
!-----------------------------------------------------------------------
  btrack_nudge= 9.013e-3 

!-----------------------------------------------------------------------
! Wetting and drying. 
! - if ihhat=1, \hat{H} is made non-negative to enhance robustness near 
! wetting and drying; if ihhat=0, no retriction is imposed for this quantity. 
! - inunfl=0 is used for normal cases and inunfl=1 is used for more accurate wetting
! and drying if grid resolution is sufficiently fine.
! - if shorewafo=1, we impose radiation stress R_s = g*grad(eta) (balance between radiation stress
! gradients and the barotropic gradients) at the numerical shoreline (boundary between
! dry and wet elements). This option ensures that the shallow depth in dry elements does not
! create unphysical and very high wave forces at the shoreline (advised for morphodynamics runs).
!-----------------------------------------------------------------------
  ihhat = 1 
  inunfl = 0
  h0 = 0.01     !min. water depth for wetting/drying [m]
  shorewafo = 0 !Matters only if USE_WWM

!-----------------------------------------------------------------------
! Solver options
! USE_PETSC controls the solver type. If it's diabled, the default JCG 
! solver is used. If it's enabled, use PetSc lib. Some of the parameters
! have different meanings under these 2 options. Also with PetSc one can
! use cmd line options to choose solver etc.
!-----------------------------------------------------------------------
  moitn0 = 50 !output spool for solver info; used only with JCG
  mxitn0 = 1500 !max. iteration allowed
  rtol0 = 1.e-12 !error tolerance

!-----------------------------------------------------------------------
! Advection (ELM) option. If nadv=1, backtracking is done using Euler method; 
! nadv=2, using 2nd order Runge-Kutta; if nadv=0, advection in momentum 
! is turned off/on in adv.gr3 (the depths=0,1, or 2 also control methods 
! in backtracking as above). dtb_max/min are the max/min steps allowed -
! actual step is calculated adaptively based on local gradient.
!-----------------------------------------------------------------------
  nadv = 1
  dtb_max = 30. !in sec
  dtb_min = 10.

!-----------------------------------------------------------------------
! If inter_mom=0, linear interpolation is used for velocity at foot of char. line.
! If inter_mom=1 or -1, Kriging is used, and the choice of covariance function is
! specified in 'kr_co'. If inter_mom=1, Kriging is applied to whole domain;
! if inter_mom=-1, the regions where Kriging is used is specified in krvel.gr3 
! (depth=0: no kriging; depth=1: with kriging). 
!-----------------------------------------------------------------------
  inter_mom = 0 
  kr_co = 1 !not used if inter_mom=0

!-----------------------------------------------------------------------
! Transport method. 
! If itr_met=1, upwind method is used. If itr_met>=2, TVD or WENO method is used
! on an element/prism if the total depth (at all nodes of the elem.)>=h_tvd and the flag in
! tvd.prop = 1 for the elem. (tvd.prop is required in this case); 
! otherwise upwind is used for efficiency. 
! itr_met=3 (horizontal TVD) or 4 (horizontal WENO): implicit TVD in the vertical dimension. 
! Also if itr_met==3 and h_tvd>=1.e5, some parts of the code are bypassed for efficiency
! Controls for WENO are not yet in place
!-----------------------------------------------------------------------
  itr_met = 3
  h_tvd = 5. !used only if itr_met>=2; cut-off depth (m) 
  !If itr_met=3 or 4, need the following 2 tolerances of convergence. The convergence
  !is achieved when sqrt[\sum_i(T_i^s+1-T_i^s)^2]<=eps1_tvd_imp*sqrt[\sum_i(T_i^s)^2]+eps2_tvd_imp
  eps1_tvd_imp = 1.e-4 !suggested value is 1.e-4, but for large suspended load, need to use a smaller value (e.g. 1.e-9)
  eps2_tvd_imp = 1.e-14  

!-----------------------------------------------------------------------
! Atmos. option. nws=3 is reserved for coupling with atmospheric model.
! If nws=0, no atmos. forcing is applied. If nws=1, atmos.
! variables are read in from wind.th. If nws=2, atmos. variables are
! read in from sflux_ files.
! If nws=4, ascii format is used for wind and atmos. pressure at each node (see source code).
! If nws>0, 'iwindoff' can be used to scale wind speed (with windfactor.gr3).
!
! Stress calculation:
! If nws=2, ihconsv=1 and iwind_form=0, the stress is calculated from heat exchange
! routine; 
! Otherwise if iwind_form=-1, the stress is calculated from Pond & Pichard formulation;
! if iwind_form=1, Hwang (2018) formulation (Cd tapers off at high wind).
! If WWM is enabled and icou_elfe_wwm>0 and iwind_form=-2, stress is overwritten by WWM.
! In all cases, if USE_ICE the stress in ice-covered portion is calculated by ICE routine.
!-----------------------------------------------------------------------
  nws = 2 
  wtiminc = 200. !time step for atmos. forcing
!  nrampwind = 1 !ramp-up option for atmos. forcing
  drampwind = 1. !needed if nrampwind/=0; ramp-up period in days
  iwind_form = -1 !needed if nws/=0
  !If impose_net_flux/=0 and nws=2, read in net _surface_ heat flux as var 'dlwrf' 
  !(Downward Long Wave) in sflux_rad (solar radiation is still used separately),
  !and if PREC_EVAP is on, also read in net P-E as 'prate' (Surface Precipitation Rate) in sflux_prc.

!-----------------------------------------------------------------------
! Heat and salt exchange. isconsv=1 needs ihconsv=1; ihconsv=1 needs nws=2.
! If isconsv=1, need to compile with precip/evap module turned on.
!-----------------------------------------------------------------------
  ihconsv = 0 !heat exchange option
  isconsv = 0 !evaporation/precipitation model

!-----------------------------------------------------------------------
! Turbulence closure.
!-----------------------------------------------------------------------
  itur = 0 
  dfv0 = 0 !needed if itur=0
  dfh0 = 1.e-4 !needed if itur=0

!-----------------------------------------------------------------------
! Sponge layer for elevation and vel.
! If inu_elev=0, no relaxation is applied to elev.
! If inu_elev=1, relax. constants are specified in elev_nudge.gr3
! and applied to eta=0 (thus a depth=0 means no relaxation).
! Similarly for inu_uv (with input uv_nudge.gr3)
!-----------------------------------------------------------------------
  inu_elev = 0
  inu_uv = 0

!-----------------------------------------------------------------------
! Nudging options for tracers. If inu_[MOD]=0, no nudging is used. If inu_tr=1,
! nudge to initial condition according to relaxation constants specified.
! If inu_tr=2, nudge to values in [MOD]_nu.nc (with step 'step_nu_tr').
! The relaxation constants = [horizontal relax (specified in [MOD]_nudge.gr3) +
! vertical relax] times dt, where vertical relax is a linear function of 
! vnh[1,2] and vnf[1,2], and [MOD] are tracer model names.
!-----------------------------------------------------------------------
  inu_tr(1) = 0 !T
  inu_tr(2) = 0 !S
  inu_tr(3) = 0 !GEN
  inu_tr(4) = 0 !Age
  inu_tr(5) = 0 !SED3D
  inu_tr(6) = 0 !EcoSim 
  inu_tr(7) = 0 !ICM 
  inu_tr(8) = 0 !CoSINE 
  inu_tr(9) = 0 !FIB
  inu_tr(10) = 0 !TIMOR 
  inu_tr(11) = 0 !FABM 

  vnh1 = 400 !vertical nudging depth 1
  vnf1 = 0. !vertical relax \in [0,1]
  vnh2 = 401 !vertical nudging depth 2 (must >vnh1)
  vnf2 = 0. !vertical relax

  step_nu_tr = 86400. !time step [sec] in all [MOD]_nu.nc (for inu_[MOD]=2)

!-----------------------------------------------------------------------
! Cut-off depth for cubic spline interpolation near bottom when computing horizontal gradients
! e.g. using hgrad_nodes() (radiation stress, and gradients of qnon and qhat in non-hydro model). 
! If depth > h_bcc1 ('deep'),
! a min. (e.g. max bottom z-cor for the element) is imposed in the spline and so a more
! conservative method is used without extrapolation beyond bottom; 
! otherwise constant extrapolation below bottom is used.
!-----------------------------------------------------------------------
  h_bcc1 = 100. !h_bcc1

!-----------------------------------------------------------------------
! Dimensioning parameters for inter-subdomain btrack. 
! If error occurs like 'bktrk_subs: overflow' or 'MAIN: nbtrk > mxnbt'
! gradually increasing these will solve the problem
!-----------------------------------------------------------------------
  s1_mxnbt = 0.5
  s2_mxnbt = 3.5

!-----------------------------------------------------------------------
! Flag for harmonic analysis for elevation. If used , need to turn on cpp flags
! in Makefile first. Otherwise set it to 0.
!-----------------------------------------------------------------------
  iharind = 0

!-----------------------------------------------------------------------
! Conservation check option. If iflux=1, some fluxes are computed
! in regions specified in fluxflag.prop (regional number from -1 to an arbitrary integer).
!-----------------------------------------------------------------------
  iflux = 0

!-----------------------------------------------------------------------
! Test flags for debugging. These flags should be turned off normally.
!-----------------------------------------------------------------------
! Williamson test #5 (zonal flow over an isolated mount); if
! on, ics must =2
!-----------------------------------------------------------------------
  izonal5 = 0 !"0" - no test; otherwise on

!-----------------------------------------------------------------------
! Rotating Gausshill test with stratified T,S (1: on; 0: off)
! Surface T,S read in from *.ic; code generates stratification
!-----------------------------------------------------------------------
  ibtrack_test = 0

!-----------------------------------------------------------------------
! Rouse profile test (1: on; 0: off)
! If on, must turn on USE_TIMOR
!-----------------------------------------------------------------------
  irouse_test = 0

!-----------------------------------------------------------------------
! Flag to choose FIB model for bacteria decay (used with USE_FIB)
! flag_fib = 1 - Constant decay rate (/day) in .gr3 format
!                (kkfib_1.gr3 and kkfib_2.gr3)
! flag_fib = 2 - Decay rate computed from Canteras et al., 1995
! flag_fib = 3 - Decay rate computed from Servais et al., 2007
!----------------------------------------------------------------------
  flag_fib = 1

!----------------------------------------------------------------------
! Marsh model parameters (only if USE_MARSH is on)
!----------------------------------------------------------------------
  slr_rate = 120. !sea-level rise rate in mm/yr, times morphological acceleration if used

!----------------------------------------------------------------------
! Vegetation model
! If iveg=1, need 3 extra inputs: (1) sav_D.gr3 (depth is stem diameter in meters);
! (2) sav_N.gr3 (depth is # of stems per m^2);
! (3) sav_h.gr3 (height of canopy in meters). 
! If one of these depths=0 at a node, the code will set all to 0. 
! If USE_MARSH is on and iveg=1, all .gr3 must have constant depths!
!----------------------------------------------------------------------
  iveg = 0 !on/off flag


!----------------------------------------------------------------------
! Coupling step with ICE module.
!----------------------------------------------------------------------
  nstep_ice = 1 !call ice module every nstep_ice steps of SCHISM

!+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
! Physical constants
!+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
!-----------------------------------------------------------------------
! Earth's radii at pole and equator (to define an ellipsoid)
!-----------------------------------------------------------------------
  rearth_pole = 6378206.4
  rearth_eq = 6378206.4

!-----------------------------------------------------------------------
! Specific heat of water (C_p) in J/kg/K
!-----------------------------------------------------------------------
  shw = 4184.d0

!-----------------------------------------------------------------------
! Reference water density for Boussinesq approximation
!-----------------------------------------------------------------------
  rho0 = 1000.d0 !kg/m^3
/

&SCHOUT
!-----------------------------------------------------------------------
! Output section - all optional
!-----------------------------------------------------------------------
! Option for hotstart outputs
!-----------------------------------------------------------------------
  nhot = 0 !1: output *_hotstart every 'hotout_write' steps
  nhot_write = 8640 !must be a multiple of ihfskip if nhot=1

!-----------------------------------------------------------------------
! Station output option. If iout_sta/=0, need output skip (nspool_sta) and
! a station.in. If ics=2, the cordinates in station.in must be in lon., lat,
! and z (positive upward; not used for 2D variables). 
!-----------------------------------------------------------------------
  iout_sta = 0
  nspool_sta = 10 !needed if iout_sta/=0; mod(nhot_write,nspool_sta) must=0

!-----------------------------------------------------------------------
! Global output options
!-----------------------------------------------------------------------
  iof_hydro(1) = 1 !0: off; 1: on - elev. [m]
  iof_hydro(2) = 1 !air pressure [Pa]
  iof_hydro(3) = 0 !air temperature [C]
  iof_hydro(4) = 0 !Specific humidity [-]
  iof_hydro(5) = 0 !solar (shortwave) radiation [W/m/m]
  iof_hydro(6) = 0 !sensible flux (positive upward) [W/m/m] 
  iof_hydro(7) = 0 !latent heat flux (positive upward) [W/m/m]
  iof_hydro(8) = 0 !upward longwave radiation (positive upward) [W/m/m]
  iof_hydro(9) = 0 !downward longwave radiation (positive downward) [W/m/m]
  iof_hydro(10) = 0 !total flux=-flsu-fllu-(radu-radd) [W/m/m]
  iof_hydro(11) = 0 !evaporation rate [kg/m/m/s]
  iof_hydro(12) = 0 !precipitation rate [kg/m/m/s]
  iof_hydro(13) = 0 !Bottom drag coefficient [-]
  iof_hydro(14) = 1 !wind velocity vector [m/s]
  iof_hydro(15) = 0 !wind stress vector [m^2/s/s]
  iof_hydro(16) = 1 !depth-averaged vel vector [m/s]
  iof_hydro(17) = 0 !vertical velocity [m/s]
  iof_hydro(18) = 0 !water temperature [C]
  iof_hydro(19) = 0 !water salinity [PSU]
  iof_hydro(20) = 0 !water density [kg/m^3]
  iof_hydro(21) = 0 !eddy diffusivity [m^2/s]
  iof_hydro(22) = 0 !eddy viscosity [m^2/s]
  iof_hydro(23) = 0 !turbulent kinetic energy
  iof_hydro(24) = 0 !turbulent mixing length [m]
  iof_hydro(25) = 0 !horizontal vel vector [m/s]

  iof_hydro(26) = 0 !horizontal vel vector defined @side [m/s]
  iof_hydro(27) = 0 !vertical vel. @elem [m/s]
  iof_hydro(28) = 0 !T @prism centers [C]
  iof_hydro(29) = 0 !S @prism centers [PSU]
  iof_hydro(30) = 0 !Barotropic pressure gradient force vector (m.s-2) @side centers 

/

definitely we need better template but I could work on it once I have initial version of workflow.

[maddenp-noaa]

@uturuncoglu I have something for you to look at / test. We recently created a new ufs-community repo called uw-coupled-drivers to collect ongoing work on coupled drivers. I've forked that repo here and have a branch with coastal/ subdirectory containing what I got from you supplemented with some contributions from me. You should be able to clone this repo, check out the branch, and work with the code.

You'll need to build a local uwtools package using the latest (as of this morning) code on main. After that, you can e.g.

$ conda env create -y -f environment.yml 
$ conda activate coastal
$ uw execute --module coastal.py --classname Coastal --task provisioned_rundir --config-file coastal.yaml --cycle 2024-08-05T12 --batch

This will create a run/ directory populated with symlinks, namelist files, the stream file, and a runscript.coastal script similar to what you could submit to Slurm. The symlinks in run/ all point to some empty files I added to the repo and would eventually remove, useful for a demo. Hopefully it will be clear how everything works -- and what you'll need to update to get an actual run -- but please follow up with questions.

Note that, if the contents of run/ were all correct (which hopefully they would be with appropriate updates to coastal.yaml), specifying --task run instead of --task provisioned_rundir in the above uw command would actually submit the job to the batch system for you.

You can also run make format && make test in the activated coastal conda environment to format and do some basic checks on the coastal.py module.

Please have a look when you have time and let me know how it goes. I can make and push changes that you can git pull, and I could also add you as a collaborator on my fork if you'd like to push changes directly.

[maddenp-noaa]

Hi @uturuncoglu -- I merged that uw-coupled-drivers PR, and we've also done a v2.4.0 release of uwtools, so I think everything you need to make progress should be available now. Please let us know if you see any problems.

[uturuncoglu]

@maddenp-noaa Thanks. That is great. I am preparing slides to present initial version of the workflow to UFS Coastal team. Then, I'll start working on it and extend the workflow to cover pieces to create input files and downloading new forcing. BTW, the users could run their case through the workflow with minimal help. Anyway, I'll update you about it. Thanks again for your help.

[maddenp-noaa]

It may be helpful, when designing, to differentiate between drivers and workflows. Drivers should have exactly what is needed to configure and run a single executable program. The current coastal.py is, of course, a good example of a driver.

To run multiple executable programs in a defined sequence, with multiple drivers, a workflow (and a workflow engine) becomes interesting. Here you might, again, want to consider Rocoto or ecFlow or even a simple workflow driven by iotaa. I think @christinaholtNOAA was going to start a discussion about workflow design, so we could continue discussion about workflows there.

If you're happy that we've reached a good stopping point for this specific discussion (I think we have), I think you can mark in "Answered" so we know that new discussions should probably happen in a new thread.

[uturuncoglu]

@maddenp-noaa Yes. Thanks for the clarification. I think we could go with iota for now. I am not sure how hard would be to switch rococo and this point and I am not sure about using ecflow along with uwtools. Let me mark this as answered and close the discussion and we could continue to discuss using another one if we need.