fix: improve workflow list and add --ext-codes flag to workflow run

docs/source/workflow-run.rst

@@ -27,7 +27,7 @@ Workflow Output
 * ``simpleaf_workflow_log.json``: This file records the meta and logging information of the workflow execution. For example, the runtime of each executed command and the ``Step`` of the start and terminating command. If ``--resume`` is set, _simpleaf_ will try to find this file in the provided output directory to decide which step(command) to start.
 * ``workflow_execution_log.json``: This file is a modified version of the workflow manifest JSON discussed above. The only modification is that in this file, the ``Active`` of the successfully invoked commands (return code 0) becomes `false`.  
 
-The results generated by successfully invoked commands might or might not be exported to the output directory specified in ``--output``, as the output directory of each invoked command is defined in the workflow template passed to ``--config-file``, rather than by ``simpleaf workflow``. Most of the workflow templates developed by the simpleaf team will store all commands' results to the output directory passed to ``--output`` if no output directory is provided in the template.
+The results generated by successfully invoked commands might or might not be exported to the output directory specified in ``--output``, as the output directory of each invoked command is defined in the workflow template passed to ``--template``, rather than by ``simpleaf workflow``. Most of the workflow templates developed by the simpleaf team will store all commands' results to the output directory passed to ``--output`` if no output directory is provided in the template.
 
 Full Usage
 ^^^^^^^^^^
@@ -36,30 +36,34 @@ The relevant options (which you can obtain by running ``simpleaf workflow run -h
 
 .. code-block:: console
 
-    Parse an instantiated workflow template and execute the corresponding commands
+    Parse an instantiated workflow template and invoke the workflow commands
 
     Usage: simpleaf workflow run [OPTIONS] --template <TEMPLATE> --output <OUTPUT>
 
     Options:
-    -t, --template <TEMPLATE>    path to an instantiated simpleaf workflow template
-    -o, --output <OUTPUT>        output directory for log files and the workflow outputs that have no explicit output directory
-    -l, --lib-paths <LIB_PATHS>  comma separated library search paths when processing the (custom) workflow configuration file. (right-most wins)
-    -h, --help                   Print help
-    -V, --version                Print version
+    -t, --template <TEMPLATE>  path to an instantiated simpleaf workflow template
+    -o, --output <OUTPUT>      output directory for log files and the workflow outputs that have no explicit output directory
+    -h, --help                 Print help
+    -V, --version              Print version
 
     Control Flow:
     -n, --no-execution           return after converting the config file to JSON foramt without executing the commands
     -s, --start-at <START_AT>    Start the execution from a specific Step. All previous steps will be ignored [default: 1]
-    -r, --resume                 resume execution from the termination step of a previous run. To use this flag, the output directory must contains the JSON file generated from a previous run
+    -r, --resume                 resume execution from the termination step of a previous run. To use this flag, the output directory must
+                                contains the JSON file generated from a previous run
         --skip-step <SKIP_STEP>  comma separated integers indicating which steps (commands) will be skipped during the execution
 
+    Jsonnet:
+    -e, --ext-codes <EXT_CODES>  comma separated string passing to internal Jsonnet engine as --ext-code flags
+    -j, --jpaths <JPATHS>        comma separated library search paths passing to internal Jsonnet engine as --jpath flags
+
 
 
 The procedure of parsing a simpleaf workflow template
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 In ``simpleaf workflow``, we use the `Jrsonnet <https://github.com/CertainLach/jrsonnet>`_, a rust implementation of Jsonnet, to parse the instantiated workflow template passed. Any valid `Jsonnet <https://jsonnet.org/>`_  program and JSON file is a valid simpleaf workflow template as long as it can produce a valid workflow manifest.
-When calling Jrsonnet, ``simpleaf workflow`` automatically passes the following arguments in addition to the provided template. This also means that any custom configuration program can access the ``output`` and ``utils`` variables in the Jsonnet program using ``std.extVar("output")`` and ``std.extVar("utils")``. Note that the path to the parent directory of the file passed to ``--config-file`` is an additional library search directory in Jrsonnet by default.
+When calling Jrsonnet, ``simpleaf workflow`` automatically passes the following built-in arguments in addition to the provided template. This also means that any custom configuration program can access the ``__output`` and ``__utils`` variables in the Jsonnet program using ``std.extVar("__output")`` and ``std.extVar("__utils")``. Note that the path to the parent directory of the file passed to ``--template`` is an additional library search directory in Jrsonnet by default.
 
 1) The output directory passed to ``--output`` as the external variable ``output``.
 2) The workflow utility library from the protocol estuary as the external variable ``utils``.

src/main.rs

@@ -173,8 +173,9 @@ fn main() -> anyhow::Result<()> {
                     no_execution,
                     start_at,
                     resume,
-                    lib_paths,
+                    jpaths,
                     skip_step,
+                    ext_codes,
                 } => run_workflow(
                     af_home_path.as_path(),
                     WorkflowCommands::Run {
@@ -183,8 +184,9 @@ fn main() -> anyhow::Result<()> {
                         no_execution,
                         start_at,
                         resume,
-                        lib_paths,
+                        jpaths,
                         skip_step,
+                        ext_codes,
                     },
                 ),
 

src/simpleaf_commands.rs

@@ -383,15 +383,25 @@ pub enum WorkflowCommands {
         )]
         resume: bool,
 
-        /// comma separated library search paths when processing the (custom) workflow configuration file. (right-most wins)
+        /// comma separated library search paths passing to internal Jsonnet engine as --jpath flags.
         #[arg(
             short,
             long,
             display_order = 6,
             value_delimiter = ',',
             help_heading = "Jsonnet"
         )]
-        lib_paths: Option<Vec<PathBuf>>,
+        jpaths: Option<Vec<PathBuf>>,
+
+        /// comma separated string passing to internal Jsonnet engine as --ext-code flags.
+        #[arg(
+            short,
+            long,
+            display_order = 6,
+            value_delimiter = ',',
+            help_heading = "Jsonnet"
+        )]
+        ext_codes: Option<Vec<String>>,
 
         /// comma separated integers indicating which steps (commands) will be skipped during the execution.
         #[arg(

src/simpleaf_commands/workflow.rs

@@ -38,21 +38,30 @@ pub fn list_workflows(af_home_path: &Path) -> anyhow::Result<()> {
     // get the corresponding workflow directory path
     let workflow_path = protocol_estuary.protocols_dir.as_path();
     let workflows = fs::read_dir(workflow_path)?;
+    let mut print_na_cap = false;
+    let na_string = String::from("N/A*");
     let mut workflow_entries = vec![];
     for prot in workflows {
         if let Ok(prot) = prot {
-            let v = workflow_utils::get_template_version(prot.path(), &protocol_estuary.utils_dir)?;
+            let version =
+                workflow_utils::get_template_version(prot.path(), &protocol_estuary.utils_dir)?;
+            if version == na_string {
+                print_na_cap = true;
+            }
             let n = format!("{:?}", prot.file_name());
             workflow_entries.push(WorkflowTemplate {
                 registry: String::from("COMBINE-lab/protocol-estuary"),
                 name: n,
-                version: v,
+                version,
             })
         } else {
             warn!("Cannot traverse directory {:?}", workflow_path)
         }
     }
     println!("{}", Table::new(workflow_entries).with(Style::rounded()));
+    if print_na_cap {
+        println!("* : could not parse uninstantiated template to attempt extracting the version, please see [url about parsing uninstantiated templates in our tutorial] for further details");
+    }
     Ok(())
 }
 
@@ -226,8 +235,9 @@ pub fn run_workflow(af_home_path: &Path, workflow_cmd: WorkflowCommands) -> anyh
             no_execution,
             start_at,
             resume,
-            lib_paths,
+            jpaths,
             skip_step,
+            ext_codes,
         } => {
             run_fun!(mkdir -p $output)?;
 
@@ -251,7 +261,8 @@ pub fn run_workflow(af_home_path: &Path, workflow_cmd: WorkflowCommands) -> anyh
                 af_home_path,
                 template.as_path(),
                 output.as_path(),
-                &lib_paths,
+                &jpaths,
+                &ext_codes,
             )?;
 
             // write complete workflow json to output folder

src/utils/jrsonnet_main.rs

@@ -1,7 +1,7 @@
 // This crate is a modified version of jrsonnet cli.
 // https://github.com/CertainLach/jrsonnet/blob/master/cmds/jrsonnet/src/main.rs
 
-use anyhow::anyhow;
+use anyhow::{anyhow, Context};
 use clap::Parser;
 use jrsonnet_cli::{ConfigureState, GeneralOpts, ManifestOpts, OutputOpts, TraceOpts};
 use jrsonnet_evaluator::{
@@ -59,7 +59,9 @@ pub fn parse_jsonnet(
     config_file_path: &Path,
     output: &Path,
     utils_dir: &Path,
-    lib_paths: &Option<Vec<PathBuf>>,
+    jpaths: &Option<Vec<PathBuf>>,
+    ext_codes: &Option<Vec<String>>,
+    instantiated: bool,
 ) -> anyhow::Result<String> {
     // define jrsonnet argumetns
     // config file
@@ -72,8 +74,9 @@ pub fn parse_jsonnet(
     //     .to_str()
     //     .expect("Could not convert the parent dir of the config file to str.");
     // external code
-    let ext_output = format!(r#"output='{}'"#, output.display());
-    let ext_utils_file_path = r#"utils=import 'simpleaf_workflow_utils.libsonnet'"#;
+    let ext_output = format!(r#"__output='{}'"#, output.display());
+    let ext_utils_file_path = r#"__utils=import 'simpleaf_workflow_utils.libsonnet'"#;
+    let ext_instantiated = format!(r#"__instantiated='{}'"#, instantiated);
 
     // af_home_dir
     let jpath_pe_utils = utils_dir
@@ -88,17 +91,29 @@ pub fn parse_jsonnet(
         &ext_output,
         "--ext-code",
         ext_utils_file_path,
+        "--ext-code",
+        &ext_instantiated,
         "--jpath",
         jpath_pe_utils,
         // "--jpath",
         // jpath_config_file_path,
     ];
 
     // if the user provides more lib search path, then assign it.
-    if let Some(lib_paths) = lib_paths {
-        for lib_path in lib_paths {
+    if let Some(jpaths) = jpaths {
+        for lib_path in jpaths {
             jrsonnet_cmd_vec.push("--jpath");
-            jrsonnet_cmd_vec.push(lib_path.to_str().expect("Could not convert path to "));
+            jrsonnet_cmd_vec.push(lib_path.to_str().with_context(|| {
+                format!("Could not convert the following path to str {:?}", lib_path)
+            })?);
+        }
+    }
+
+    // if the user provides ext-code, then assign it.
+    if let Some(ext_codes) = ext_codes {
+        for ext_code in ext_codes {
+            jrsonnet_cmd_vec.push("--ext-code");
+            jrsonnet_cmd_vec.push(ext_code.as_str());
         }
     }
 
@@ -160,9 +175,7 @@ fn main_real(s: &State, opts: Opts) -> Result<String, Error> {
     let manifest_format = opts.manifest.configure(s)?;
 
     let input = opts.input.input.ok_or(Error::MissingInputArgument)?;
-    let val = s
-        .import(input)
-        .expect("Cannot import workflow config file.");
+    let val = s.import(input)?;
 
     let val = apply_tla(s.clone(), &tla, val)?;
 

src/utils/workflow_utils.rs

@@ -42,12 +42,17 @@ pub fn get_template_version(template_dir: PathBuf, utils_dir: &Path) -> anyhow::
     };
 
     // Then we call Jrsonnet to get JSON string
-    let workflow_json_string = parse_jsonnet(
+    let workflow_json_string = match parse_jsonnet(
         &template_path,
         PathBuf::from(".").as_path(),
         utils_dir,
         &None,
-    )?;
+        &None,
+        false,
+    ) {
+        Ok(v) => v,
+        Err(_) => return Ok(String::from("N/A*")),
+    };
 
     let workflow_json_value: Value = serde_json::from_str(workflow_json_string.as_str())?;
 
@@ -56,13 +61,13 @@ pub fn get_template_version(template_dir: PathBuf, utils_dir: &Path) -> anyhow::
             if let Some(v) = version_value.as_str() {
                 v.to_string()
             } else {
-                String::from("")
+                String::from("missing")
             }
         } else {
-            String::from("")
+            String::from("missing")
         }
     } else {
-        String::from("")
+        String::from("missing")
     };
 
     Ok(v)
@@ -780,7 +785,8 @@ pub fn parse_workflow_config(
     af_home_path: &Path,
     config_file_path: &Path,
     output: &Path,
-    lib_paths: &Option<Vec<PathBuf>>,
+    jpaths: &Option<Vec<PathBuf>>,
+    ext_codes: &Option<Vec<String>>,
 ) -> anyhow::Result<String> {
     // get protocol_estuary path
     let protocol_estuary = get_protocol_estuary(af_home_path, false)?;
@@ -791,7 +797,9 @@ pub fn parse_workflow_config(
         config_file_path,
         output,
         &protocol_estuary.utils_dir,
-        lib_paths,
+        jpaths,
+        ext_codes,
+        true,
     ) {
         Ok(js) => Ok(js),
         Err(e) => Err(anyhow!(