Improve dfm modelbuilder workflow

[veenstrajelmer]

Initialize service:

• Login to EDITO datalab: https://datalab.digitaltwinocean.edito.eu/
• Service Catalog >> Ocean modelling >> Dfm-modelbuilder >> Launch
• Launch
• this will initialize a Python environment with the JupyterLab IDE and dfm_tools installed

Run the modelbuilder:

• Open modelbuilder_example.ipynb
• optionally, update spatial/temporal extent in block [2]
• optionally, adjust the min/max grid resolution with dxy in block [2] and min_edge_size in block [5].
• restart kernel and run all cells by clicking
• generates a folder with D-FlowFM model input files for the desired area/period
• the last cell of the notebook uploads the model to your S3 bucket

run the D-FlowFM model with another service:

• make sure you have the "Vietnam_model" folder in your s3 bucket (including run_docker.sh)
• go to the EDITO Service Catalog
• Process Catalog >> Ocean modelling >> Delft3d-fm >> Launch (is this still the correct service? Delft3dfm-demo dissapeared)
• After approximately 3 minutes, this results in a DFM_OUTPUT_Vietnam folder in your s3 bucket

Remaining issues (helm chart):

• replace with custom service that includes init and credentials already: https://gitlab.mercator-ocean.fr/pub/edito-infra/helm-charts/playground/-/merge_requests/26
• add note about credentials to the service initialization
• update notes about deletion
• set minimal requirements (memory, cpu amount, cpu clocktime, diskspace, bandwith, maybe more)

Remaining issues (init.sh and extend_modelbuilder_notebook.py)

• in init.sh: after copying ipynb, add cell that calls a to be created upload_model_to_s3(dir_model) function and supplying the correct foldername/dir_model to that function
• ipynb/py files are downloaded twice. Seems like init script is executed twice, which might be a bug in edito
• clear notebook output with jupyter nbconvert --clear-output --inplace my_notebook.ipynb
• stretchtype is manually to zero in extra notebook cell until mdu stretchtype should be -1 or 0 instead of 1 Deltares/HYDROLIB-core#691 is fixed
• install from pypi instead of main branch (do this when the modelbuilder is complete)

Remaining issues (modelbuilder, notebook and run_docker.sh):

• flexible (or generic) model input folder name
• create_model_exec_files is not used yet since multicore dimr cannot be parsed and sequential dimr returns nproc=0 instead of 1: dimr_config for parallel run is incorrect Deltares/HYDROLIB-core#562 >> require hydrolib-core 0.8.0 Deltares/dfm_tools#890 >> require minimal dfmt version and simplify upload script
• https://gitlab.mercator-ocean.fr/pub/edito-infra/service-helm-charts/-/merge_requests/27/
• improve run_docker.sh Deltares/dfm_tools#729
• parallel run fails, are we allowed to use multiple cores on EDITO? See also comment in current dockerfile about how to run it, like ulimit and shm-size, maybe we have to demand n cores here also?
• maybe merge partitions (decoimpact needs this) with delft3dfm mapmerge in run_docker.sh. Can the d-ecoimpact service find both sequential/merged nc file since they have different names?
• add temperature=1 and salinity=1 (including heatfluxmodel comment), check runtime
• add waq processes to model: Improve modelbuilder notebook Deltares/dfm_tools#515

Remaining issues (s3 upload):

• flexible (or generic) model input folder name
• convert to function so we can call it from the ipynb notebook directly
• prevent the need for adding run_docker.sh in this script, first solve dimr_config for parallel run is incorrect Deltares/HYDROLIB-core#562
• simplify the s3 upload commands by avoiding the need for a temp directory

Remaining issues (running delft3dfm):

• no access rights for JV
• currently fails since job.yaml expects the folder Vietnam_model to exist. upload_model.py now always renames this to DFM_model, which might not be the best name (delft3dfm_model would be a better alternative).
• https://gitlab.mercator-ocean.fr/pub/edito-infra/process-helm-charts/-/merge_requests/10 fixes this, and it also renames the output folder from something like DFM_OUTPUT_Vietnam to DFM_OUTPUT.
• run with new docker image from containers.deltares.nl: https://issuetracker.deltares.nl/browse/PWES-265
• crash/message if crash, instead of just running service and no output (add process text to screen like dfm-modelbuilder)
• add post-processing notebook/service?