Skip to content

NGen Tutorial

Jump to bottom
Nels edited this page Oct 20, 2023 · 63 revisions

NGen Tutorial

This Tutorial will walk through the steps to build NGen and execute several formulations on a small example catchment. We will use a terminal to download, compile, and run the NGen framework.

Building NGen

A few dependencies are required to build and run NGen, those are documented in the dependencies document. For this tutorial, the minium required depdnencies are C/C++ compiler, CMake, and Boost. Additionally, to use the Fortran BMI adapter, a Fortran Compiler is required to to build the middleware, gfortran should be sufficient.

The first step after ensuring your environment has the required dependencies is to clone the NGen repository.

git clone https://github.com/noaa-owp/ngen
cd ngen

Setting up the external components

For this tutorial, we are going to use several external components, and we need to make sure they are properly set up.

  1. First, make sure all the project submodules have been retrieved/updated. 
    git submodule update --init
  • It is possible to do this for an individual submodule as well, by appending -- <submodule_path> to the above command. The submodules paths can be found in the .gitmodules file.
  1. Obtain any needed external BMI modules. Several external modules have had configurations to build shared library files added within the ngen repo. Those can be built using the following commands (note on some systems you may need to use cmake3 instead of cmake). 
    cmake -B extern/cfe/cmake_build -S extern/cfe
make -C extern/cfe/cmake_build
cmake -B extern/topmodel/cmake_build -S extern/topmodel
make -C extern/topmodel/cmake_build
cmake -B extern/noah-owp-modular/cmake_build -S extern/noah-owp-modular
make -C extern/noah-owp-modular/cmake_build
cmake -B extern/evapotranspiration/cmake_build -S extern/evapotranspiration/evapotranspiration/
make -C extern/evapotranspiration/cmake_build
pushd extern/sloth
git submodule update --init
popd
cmake -B extern/sloth/cmake_build -S extern/sloth
make -C extern/sloth/cmake_build
  • These examples are for convenience; depending on your realization configuration, you may not need to build all or any of theses.
  • What matters is that the external module location (meaning the shared library file path for C/C++/Fortran) matches the realization configuration.
  1. To use the noah-owp-modular fortran library, we need to build the iso_c_fortran_bmi middleware 
    cmake -B extern/iso_c_fortran_bmi/cmake_build -S extern/iso_c_fortran_bmi
make -C extern/iso_c_fortran_bmi/cmake_build
  • While this is necessary at the time of this writing, this step should no longer be needed once Issue #333 is resolved.

Configuring the NGen build

NGen builds are managed by the CMake build system. To create a build configuration to compile NGen, use a command like the following (assumed to be run from within the ngen repo directory):

cmake -B cmake_build -S . -DNGEN_WITH_PYTHON:BOOL=OFF -DNGEN_WITH_BMI_FORTRAN:BOOL=ON -DUDUNITS_QUIET:BOOL=ON

This configures NGen to compile without Python support, and with Fortran BMI support, which is needed for Noah-OWP respectively. Note that Python support in particular must be explicitly set to OFF to deactivate it.

If you need Python, you can set it to ON, but you may need to create a local virtual environment directory at .venv and install numpy.

Note -DUDUNITS_QUIET:BOOL=ON will silence any unit conversion warnings that UDUnits emits

Now we can compile NGen

make -j 8 -C cmake_build

Testing the NGen build

Before continuing, it might be a good idea to run the unit tests for NGen. To test the BMI features, we need to build the external test models

   cmake -B extern/test_bmi_c/cmake_build -S extern/test_bmi_c
   make -C extern/test_bmi_c/cmake_build
   cmake -B extern/test_bmi_fortran/cmake_build -S extern/test_bmi_fortran
   make -C extern/test_bmi_fortran/cmake_build
./cmake_build/test/test_unit

Then we can test BMI integration

./cmake_build/test/test_bmi_c
./cmake_build/test/test_bmi_fortran
./cmake_build/test/test_bmi_python # Only if you activated Python, though

TL;DR;

Copy and paste the following into your shell. If you are on a system with both cmake and cmake3, set an alias to make sure you use cmake3.

alias cmake=cmake3

Then run

git clone https://github.com/noaa-owp/ngen &&
cd ngen &&
git submodule update --init &&
cmake -B extern/cfe/cmake_build -S extern/cfe && 
make -C extern/cfe/cmake_build && 
cmake -B extern/topmodel/cmake_build -S extern/topmodel && 
make -C extern/topmodel/cmake_build && 
cmake -B extern/noah-owp-modular/cmake_build -S extern/noah-owp-modular && 
make -C extern/noah-owp-modular/cmake_build &&
cmake -B extern/evapotranspiration/cmake_build -S extern/evapotranspiration/evapotranspiration/
make -C extern/evapotranspiration/cmake_build
make -C extern/evapotranspiration/cmake_build
pushd extern/sloth
git submodule update --init
popd
cmake -B extern/sloth/cmake_build -S extern/sloth
make -C extern/sloth/cmake_build
cmake -B extern/iso_c_fortran_bmi/cmake_build -S extern/iso_c_fortran_bmi &&
make -C extern/iso_c_fortran_bmi/cmake_build &&
cmake -B cmake_build -S . -DNGEN_ACTIVATE_PYTHON:BOOL=ON -DBMI_C_LIB_ACTIVE:BOOL=ON -DBMI_FORTRAN_ACTIVE:BOOL=ON &&
make -j 8 -C cmake_build &&
cmake -B extern/test_bmi_c/cmake_build -S extern/test_bmi_c && 
make -C extern/test_bmi_c/cmake_build && 
cmake -B extern/test_bmi_fortran/cmake_build -S extern/test_bmi_fortran && 
make -C extern/test_bmi_fortran/cmake_build &&
./cmake_build/test/test_unit && 
./cmake_build/test/test_bmi_c && 
./cmake_build/test/test_bmi_fortran

Get a quick coffee...Then you should see

[       OK ] Bmi_Fortran_Formulation_Test.determine_model_time_offset_0_c (16 ms)
[----------] 11 tests from Bmi_Fortran_Formulation_Test (283 ms total)

[----------] Global test environment tear-down
[==========] 90 tests from 2 test suites ran. (334 ms total)
[  PASSED  ] 90 tests.

Everything built correctly!

Important For a more information on the ngen build process, see Building.

Running Formulations with NGen

Now that NGen is compiled, we can run some simple formulation tests. The input to ngen looks like

ngen <catchment_data_path> <catchment subset ids> <nexus_data_path> <nexus subset ids> <realization_config_path>

Important For a more information on ngen usage, see Usage.

Important For a more information on ngen's use of formulations/BMI, see Formulations and BMI.

Running CFE

CFE with PET

CFE requires a Potential EvapoTranspiration input. This input needs to be supplied from some module. We could use SLoTH to prvoide a fake value to CFE, or we can use the evapotranspiration BMI model to compute a real PET value. We will use the PET module built in the previous steps to supply values to CFE.

The following commands create a space to store the simulation outputs and then links the relevant directories needed for forcing and libraries. Run ngen, pointing it to the hydrofabric files and selecting just cat-27 to run.

TODO Need to create a custom realization for these next steps

The realization file configures the relevant BMI settings required to connect to the CFE fomulation. Note that there are two possible configs, one for linux and one for macos, since these two systems name shared library files slightly different. Pick the file based on your platform, and replace the <linux/macos> in the command below with just the name, e.g. example_realization_config_w_bmi_c__linux.json

mkdir cfe
cd cfe
ln -s ../data
ln -s ../extern

../cmake_build/ngen data/catchment_data.geojson cat-27 data/nexus_data.geojson nex-26 data/example_realization_config_w_bmi_c__<linux/macos>.json

Now you should see in your directory the outfiles cat-27.csv and nex-26_output.csv. The cat-27.csv file contains a table of the model's OUTPUT_VARS for each time step. nex-26_output.csv contains the nexus output, which is the modules main_output_variable (as defined in the realization configuration file). This is currently multiplied by the catchments area (meters squared) as determined from the hydrofabric.

Note, this is using a generic config file to initialize the model.

Running Topmodel

For this model run, we will repeat the basic steps above:

mkdir topmod
cd topmod
ln -s ../data
ln -s ../extern

But this time, we need to edit the config file, so copy it to this directory:

cp data/example_realization_config_w_bmi_c__<linux/macos>.json ./realization_config.json

Now copy some test data for topmodel

cp ./extern/topmodel/topmodel/data/* data

Edit the config file realization_config.json, specifically the formulation for cat-27.

Set the following variables to these values:

  • "model_type_name": "bmi_c_topmodel"
  • "library_file": "./extern/topmodel/cmake_build/libtopmodelbmi.<dylib/so>"
  • "init_config": "./data/topmod.run"
  • "main_output_variable": "Qout"
  • "registration_function": "register_bmi_topmodel"

Additionally, TOPMODEL writes its own output files unless specifically told not to. If you want to only have Nextgen output, please set both the stand_alone and output flags to 0 in topmod.run and subcat.dat, respectively. More info here and here and here.

Now run the model

../cmake_build/ngen data/catchment_data.geojson cat-27 data/nexus_data.geojson nex-26 realization_config.json

Once again, you can check the outputs in cat-27.csv and nex-26_output.csv.

Running Noah-OWP-Modular

Setup a space to run the model:

mkdir noahowp
cd noahowp
ln -s ../data
ln -s ../extern

Copy the bmi fortran realization config to the directory:

cp data/example_realization_config_w_bmi_c__<linux/macos>.json ./realization_config.json

Now copy the namelist (model options) for Noah-OWP-Modular:

cp extern/noah-owp-modular/noah-owp-modular/run/namelist.input .

Open the newly copied namelist.input file and modify the parameter_dir to include the path for the parameter tables. For this example, that is ./extern/noah-owp-modular/noah-owp-modular/parameters/.

Edit the config file realization_config.json, specifically the formulation for cat-27.

Set the following variables to these values:

  • "name": "bmi_fortran",
  • "model_type_name": "bmi_fortran_noahowp"
  • "library_file": "./extern/noah-owp-modular/cmake_build/libsurfacebmi.<dylib/so>"
  • "init_config": "namelist.input"
  • "main_output_variable": "QINSUR"
  • "registration_function": "register_bmi"
  • And replace the entire variables_names_map with 
    "variables_names_map" : {
    "PRCPNONC" : "atmosphere_water__liquid_equivalent_precipitation_rate",
    "Q2" : "atmosphere_air_water~vapor__relative_saturation",
    "SFCTMP" : "land_surface_air__temperature",
    "UU" : "land_surface_wind__x_component_of_velocity",
    "VV" : "land_surface_wind__y_component_of_velocity",
    "LWDN" : "land_surface_radiation~incoming~longwave__energy_flux",
    "SOLDN" : "land_surface_radiation~incoming~shortwave__energy_flux",
    "SFCPRS" : "land_surface_air__pressure"
}

Now run the model

../cmake_build/ngen data/catchment_data.geojson cat-27 data/nexus_data.geojson nex-26 realization_config.json

Once again, you can check the outputs in cat-27.csv and nex-26_output.csv.

Home

Tutorial

Getting Started

Configuration

Technical References

Clone this wiki locally