Building a workflow with uwtools

[christinaholtNOAA]

@uturuncoglu

It is a bit overdue, but I want to follow up on the topic of building workflows with uwtools that started as part of Discussion #478.

Currently, uwtools has support for defining a Rocoto workflow using the UW YAML language that maps directly onto the Rocoto XML language. This YAML interface means that we can use the uw config realize tools to define arbitrary tasks, add/remove/modify pre-existing tasks, and generally build up sets of tasks that we want to run as part of any given experiment.

The MPAS App is currently doing this with the uw rocoto realize tool, and the approach taken in SRW is very similar, although that app has not yet adopted the uw rocoto tool. Here, I'll describe the basics of how the experiment is built in the MPAS app. I'll use the HFIP_2024 branch as an example because it's been morphing into something more extensive, and we plan to merge the features from this branch back to the main branch once development for HFIP settles down.

The MPAS App keeps a set of default workflow definitions in the parm/wflow directory. We've broken chunks of the workflow down into separate files like cold_start.yaml and post.yaml. I'm also working on a new addition that will add graphics.yaml and tracker.yaml. Each of these files represents a set of tasks that work nicely together and may easily be turned on or off by just including the name of the yaml file to be included in the full workflow definition. This switch is defined in the ush/defaults_config.yaml file as user.workflow_blocks: where the default is just to run the forecast without any post-processing. For example, the default might be the optimal case for a model test suite, whereas a scientific case study might turn on all the post-processing, TC tracking, graphics, etc.

These default workflow definitions files follow the UW YAML Rocoto syntax and can reference any other sections in your experiment configuration if you manage all of that together in the app-level experiment generation script.

If a user wants to turn on post-processing, their user yaml might include the block:

user:
  workflow_blocks:
    - cold_start.yaml
    - post.yaml

The ush/experiment_gen.py script works through this list and prepares a full workflow configuration file. Check out that code block here.

I'd like to pause here in the discussion to see where I might fill in any gaps.

[uturuncoglu]

@christinaholtNOAA That is really cool. Thanks for sharing all those informations. Let me go through them and try to understand. I think the best thing is to try in UFS Coastal App side to see what is missing. At this point, our initial workflow is using iotaa but it would be nice to test rococo. BTW, is there any development on generating top level ufs.configure and model_configure files. As I know they are not used by MPAS but probably used by SRW app.

[christinaholtNOAA]

@uturuncoglu No development just yet on the generation of ufs.configure files. I think we should be able to move toward a very straightforward YAML for the model_configure. Still, it would require some very small changes to the default files in the weather model and would mostly impact the HAFS configurations with a non-standard way of handling sections for their various domains.