Updated docs

mlcommons · Feb 23, 2024 · 1f9de76 · 1f9de76
1 parent c4f3106
commit 1f9de76
Show file tree

Hide file tree

Showing 10 changed files with 375 additions and 18 deletions.
diff --git a/cm-mlops/script/app-mlperf-inference-reference/README.md b/cm-mlops/script/app-mlperf-inference-reference/README.md
@@ -823,7 +823,7 @@ ___
   1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-reference/customize.py)***
   1. ***Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-reference/_cm.yaml)***
      * remote,run,cmds
-       * `if (CM_SSH_RUN_COMMANDS  == on)`
+       * `if (CM_ASSH_RUN_COMMANDS  == on)`
        * CM names: `--adr.['remote-run-cmds']...`
        - CM script: [remote-run-commands](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/remote-run-commands)
   1. ***Run native script if exists***

diff --git a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/README.md b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/README.md
@@ -117,6 +117,8 @@ ___
       - Workflow:
     * `_rb6,power`
       - Workflow:
+    * `_rb6,qualcomm`
+      - Workflow:
     * `_rpi4,power`
       - Workflow:
     * `_sapphire-rapids.24c,nvidia`

diff --git a/cm-mlops/script/calibrate-model-for.qaic/README.md b/cm-mlops/script/calibrate-model-for.qaic/README.md
@@ -132,7 +132,7 @@ ___
         - *CM_QAIC_MODEL_BATCH_SIZE*: `#`
         - *CM_CREATE_INPUT_BATCH*: `yes`
       - Workflow:
-    * **`_bs.1`** (default)
+    * `_bs.1`
       - Environment variables:
         - *CM_QAIC_MODEL_BATCH_SIZE*: `1`
         - *CM_CREATE_INPUT_BATCH*: `yes`
@@ -220,10 +220,6 @@ ___
 
     </details>
 
-
-#### Default variations
-
-`_bs.1`
 #### Default environment
 
 <details>

diff --git a/cm-mlops/script/compile-model-for.qaic/README.md b/cm-mlops/script/compile-model-for.qaic/README.md
@@ -107,7 +107,7 @@ ___
       - Environment variables:
         - *CM_COMPILE_BERT*: `on`
         - *CM_QAIC_MODEL_TO_CONVERT*: `calibrate_bert_mlperf`
-        - *CM_QAIC_MODEL_COMPILER_PARAMS_BASE*: `-aic-hw -aic-hw-version=2.0 -execute-nodes-in-fp16=Mul,Sqrt,Div,Add,ReduceMean,Softmax,Sub,Gather,Erf,Pow,Concat,Tile,LayerNormalization -quantization-schema=symmetric_with_uint8 -quantization-precision=Int8 -quantization-precision-bias=Int32 -vvv -compile-only -onnx-define-symbol=batch_size,1 -onnx-define-symbol=seg_length,384 -multicast-weights -combine-inputs=false -combine-outputs=false`
+        - *CM_QAIC_MODEL_COMPILER_PARAMS_BASE*: `-aic-hw -aic-hw-version=2.0 -execute-nodes-in-fp16=Add,Div,Erf,Softmax -quantization-schema=symmetric_with_uint8 -quantization-precision=Int8 -quantization-precision-bias=Int32 -vvv -compile-only -onnx-define-symbol=batch_size,1 -onnx-define-symbol=seg_length,384 -multicast-weights -combine-inputs=false -combine-outputs=false`
         - *CM_QAIC_MODEL_COMPILER_ARGS*: ``
       - Workflow:
         1. ***Read "deps" on other CM scripts***
@@ -117,12 +117,16 @@ ___
     * `_bert-99,offline`
       - Environment variables:
         - *CM_QAIC_MODEL_COMPILER_ARGS*: `-allocator-dealloc-delay=2 -size-split-granularity=1536 -vtcm-working-set-limit-ratio=1`
-        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=1 -mos=1 -ols=3`
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=1 -mos=1 -ols=2`
       - Workflow:
     * `_bert-99,offline,nsp.14`
       - Environment variables:
         - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=1 -mos=1 -ols=3`
       - Workflow:
+    * `_bert-99,offline,nsp.16`
+      - Environment variables:
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=1 -mos=1 -ols=2`
+      - Workflow:
     * `_bert-99,server`
       - Environment variables:
         - *CM_QAIC_MODEL_COMPILER_ARGS*: `-allocator-dealloc-delay=2 -size-split-granularity=1536 -vtcm-working-set-limit-ratio=1`
@@ -141,6 +145,33 @@ ___
       - Environment variables:
         - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=8 -mos=8 -ols=1`
       - Workflow:
+    * `_bert-99.9`
+      - Environment variables:
+        - *CM_COMPILE_BERT*: `on`
+        - *CM_QAIC_MODEL_TO_CONVERT*: `bert_mlperf`
+        - *CM_QAIC_MODEL_COMPILER_PARAMS_BASE*: `-aic-hw -aic-hw-version=2.0 -convert-to-fp16 -vvv -compile-only -onnx-define-symbol=batch_size,1 -onnx-define-symbol=seg_length,384 -combine-inputs=false -combine-outputs=false`
+        - *CM_QAIC_MODEL_COMPILER_ARGS*: ``
+      - Workflow:
+    * `_bert-99.9,offline`
+      - Environment variables:
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=2 -mos=1 -ols=2`
+      - Workflow:
+    * `_bert-99.9,offline,nsp.14`
+      - Environment variables:
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=2 -mos=1 -ols=2`
+      - Workflow:
+    * `_bert-99.9,offline,nsp.16`
+      - Environment variables:
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=2 -mos=1 -ols=2`
+      - Workflow:
+    * `_bert-99.9,server`
+      - Environment variables:
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=4`
+      - Workflow:
+    * `_bert-99.9,server,nsp.14`
+      - Environment variables:
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=4`
+      - Workflow:
     * `_resnet50`
       - Environment variables:
         - *CM_COMPILE_RESNET*: `on`
@@ -150,6 +181,7 @@ ___
     * `_resnet50,multistream`
       - Environment variables:
         - *CM_QAIC_MODEL_COMPILER_ARGS*: ``
+        - *CM_QAIC_MODEL_COMPILER_ARGS_SUT*: `-aic-num-cores=4 -mos=1 -ols=1`
       - Workflow:
     * `_resnet50,multistream,nsp.14`
       - Environment variables:
@@ -404,11 +436,12 @@ ___
 #### New environment keys (filter)
 
 * `CM_ML_MODEL_FILE_WITH_PATH`
-* `CM_QAIC_*`
+* `CM_QAIC_MODEL*`
 #### New environment keys auto-detected from customize
 
 * `CM_ML_MODEL_FILE_WITH_PATH`
 * `CM_QAIC_MODEL_COMPILED_BINARY_WITH_PATH`
+* `CM_QAIC_MODEL_FINAL_COMPILATION_CMD`
 ___
 ### Maintainers
 

diff --git a/cm-mlops/script/get-mlperf-inference-loadgen/README.md b/cm-mlops/script/get-mlperf-inference-loadgen/README.md
@@ -170,9 +170,9 @@ ___
      * get,compiler
        * `if (CM_HOST_OS_TYPE  != windows)`
        * CM names: `--adr.['compiler']...`
-       - CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
        - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
        - CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
+       - CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
      * get,cl
        * `if (CM_HOST_OS_TYPE  == windows)`
        * CM names: `--adr.['compiler']...`

diff --git a/cm-mlops/script/get-onnxruntime-prebuilt/README.md b/cm-mlops/script/get-onnxruntime-prebuilt/README.md
@@ -141,6 +141,7 @@ ___
   1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-onnxruntime-prebuilt/customize.py)***
   1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-onnxruntime-prebuilt/_cm.json)
   1. ***Run native script if exists***
+     * [run.bat](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-onnxruntime-prebuilt/run.bat)
      * [run.sh](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-onnxruntime-prebuilt/run.sh)
   1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-onnxruntime-prebuilt/_cm.json)
   1. ***Run "postrocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-onnxruntime-prebuilt/customize.py)***
@@ -156,8 +157,13 @@ ___
 * `+C_INCLUDE_PATH`
 * `+DYLD_FALLBACK_LIBRARY_PATH`
 * `+LD_LIBRARY_PATH`
+* `+PATH`
+* `CM_ONNXRUNTIME_INCLUDE_PATH`
+* `CM_ONNXRUNTIME_LIB_PATH`
 #### New environment keys auto-detected from customize
 
+* `CM_ONNXRUNTIME_INCLUDE_PATH`
+* `CM_ONNXRUNTIME_LIB_PATH`
 ___
 ### Maintainers
 

diff --git a/cm-mlops/script/install-llvm-prebuilt/README.md b/cm-mlops/script/install-llvm-prebuilt/README.md
@@ -136,9 +136,11 @@ ___
 `cmr "install prebuilt llvm prebuilt-llvm install-prebuilt-llvm"  -j`
 #### New environment keys (filter)
 
+* `+CPLUS_INCLUDE_PATH`
 * `+C_INCLUDE_PATH`
 * `+LD_LIBRARY_PATH`
 * `+PATH`
+* `CM_COMPILER_NAME`
 * `CM_LLVM_*`
 #### New environment keys auto-detected from customize
 

diff --git a/cm-mlops/script/reproduce-mlperf-inference-qualcomm/README.md b/cm-mlops/script/reproduce-mlperf-inference-qualcomm/README.md
@@ -132,16 +132,26 @@ ___
 
     * `_activation-count.#`
       - Environment variables:
-        - *CM_MODEL_BATCH_SIZE*: `#`
         - *CM_MLPERF_QAIC_ACTIVATION_COUNT*: `#`
       - Workflow:
+    * `_bert-99,offline`
+      - Workflow:
     * `_bert-99,qaic`
       - Workflow:
         1. ***Read "deps" on other CM scripts***
            * compile,qaic,model,_bert-99,_pc.99.9980
              * `if (CM_MLPERF_SKIP_RUN  != True)`
              * CM names: `--adr.['qaic-model-compiler', 'bert-99-compiler']...`
              - CM script: [compile-model-for.qaic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/compile-model-for.qaic)
+    * `_bert-99.9,offline`
+      - Workflow:
+    * `_bert-99.9,qaic`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * compile,qaic,model,_bert-99.9
+             * `if (CM_MLPERF_SKIP_RUN  != True)`
+             * CM names: `--adr.['qaic-model-compiler', 'bert-99.9-compiler']...`
+             - CM script: [compile-model-for.qaic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/compile-model-for.qaic)
     * `_bert_,network-client`
       - Environment variables:
         - *CM_BENCHMARK*: `NETWORK_BERT_CLIENT`
@@ -152,6 +162,7 @@ ___
       - Workflow:
     * `_bert_,qaic`
       - Environment variables:
+        - *kilt_model_batch_size*: `1`
         - *kilt_input_format*: `UINT32,1,384:UINT32,1,8:UINT32,1,384:UINT32,1,384`
         - *kilt_input_formata*: `UINT32,1,384:UINT32,1,384:UINT32,1,384`
         - *kilt_output_formatia*: `UINT8,1,384:UINT8,1,384`
@@ -161,11 +172,11 @@ ___
       - Environment variables:
         - *kilt_model_batch_size*: `1`
       - Workflow:
-    * `_dl2q.24xlarge,bert-99,offline`
+    * `_dl2q.24xlarge,bert-99.9,offline`
       - Environment variables:
         - *qaic_activation_count*: `14`
       - Workflow:
-    * `_dl2q.24xlarge,bert-99,server`
+    * `_dl2q.24xlarge,bert-99.9,server`
       - Environment variables:
         - *qaic_activation_count*: `14`
       - Workflow:
@@ -190,9 +201,84 @@ ___
         - *CM_QAIC_DEVICES*: `0`
         - *qaic_activation_count*: `1`
       - Workflow:
-    * `_loadgen-batch-size.#`
+    * `_nsp.16`
+      - Workflow:
+    * `_num-devices.4`
       - Environment variables:
-        - *CM_MLPERF_LOADGEN_BATCH_SIZE*: `#`
+        - *CM_QAIC_DEVICES*: `0,1,2,3`
+      - Workflow:
+    * `_pro`
+      - Environment variables:
+        - *qaic_queue_length*: `10`
+      - Workflow:
+    * `_pro,num-devices.4,bert-99,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `16`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * set,device,qaic,_vc.15
+             - CM script: [set-device-settings-qaic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/set-device-settings-qaic)
+    * `_pro,num-devices.4,bert-99.9,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `8`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * set,device,qaic,_vc.17
+             - CM script: [set-device-settings-qaic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/set-device-settings-qaic)
+    * `_pro,num-devices.4,bert-99.9,server`
+      - Environment variables:
+        - *qaic_activation_count*: `16`
+      - Workflow:
+    * `_pro,num-devices.4,resnet50,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `4`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * set,device,qaic,_vc.16
+             - CM script: [set-device-settings-qaic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/set-device-settings-qaic)
+    * `_pro,num-devices.4,resnet50,server`
+      - Environment variables:
+        - *qaic_activation_count*: `4`
+      - Workflow:
+    * `_pro,num-devices.4,retinanet,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `16`
+      - Workflow:
+        1. ***Read "deps" on other CM scripts***
+           * set,device,qaic,_vc.17
+             - CM script: [set-device-settings-qaic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/set-device-settings-qaic)
+    * `_pro,num-devices.4,retinanet,server`
+      - Environment variables:
+        - *qaic_activation_count*: `16`
+      - Workflow:
+    * `_pro,num-devices.4,singlestream`
+      - Environment variables:
+        - *CM_QAIC_DEVICES*: `0`
+        - *qaic_activation_count*: `1`
+      - Workflow:
+    * `_rb6,bert-99,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `9`
+      - Workflow:
+    * `_rb6,resnet50,multistream`
+      - Environment variables:
+        - *qaic_activation_count*: `2`
+      - Workflow:
+    * `_rb6,resnet50,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `2`
+      - Workflow:
+    * `_rb6,retinanet,multistream`
+      - Environment variables:
+        - *qaic_activation_count*: `8`
+      - Workflow:
+    * `_rb6,retinanet,offline`
+      - Environment variables:
+        - *qaic_activation_count*: `9`
+      - Workflow:
+    * `_rb6,singlestream`
+      - Environment variables:
+        - *qaic_activation_count*: `1`
       - Workflow:
     * `_resnet50,uint8`
       - Environment variables:
@@ -206,6 +292,10 @@ ___
         - *kilt_input_format*: `UINT8,1,3,800,800`
         - *kilt_output_format*: `INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,1000:INT8,1,4,1000:INT8,14,1000:INT8,1,4,1000:INT8,1,4,1000:INT8,1,4,1000`
       - Workflow:
+    * `_singlestream,resnet50`
+      - Workflow:
+    * `_singlestream,retinanet`
+      - Workflow:
 
     </details>
 
@@ -218,6 +308,10 @@ ___
       - Environment variables:
         - *kilt_model_batch_size*: `#`
       - Workflow:
+    * `_bs.0`
+      - Environment variables:
+        - *kilt_model_batch_size*: `1`
+      - Workflow:
 
     </details>
 
@@ -286,6 +380,18 @@ ___
     </details>
 
 
+  * Group "**loadgen-batch-size**"
+    <details>
+    <summary>Click here to expand this section.</summary>
+
+    * `_loadgen-batch-size.#`
+      - Environment variables:
+        - *CM_MLPERF_LOADGEN_BATCH_SIZE*: `#`
+      - Workflow:
+
+    </details>
+
+
   * Group "**loadgen-scenario**"
     <details>
     <summary>Click here to expand this section.</summary>
@@ -406,6 +512,8 @@ ___
     <details>
     <summary>Click here to expand this section.</summary>
 
+    * `_fp16`
+      - Workflow:
     * `_fp32`
       - Environment variables:
         - *CM_IMAGENET_ACCURACY_DTYPE*: `float32`
@@ -445,6 +553,11 @@ ___
         - *CM_QAIC_DEVICES*: `0,1,2,3,4,5,6,7`
         - *qaic_queue_length*: `4`
       - Workflow:
+    * `_rb6`
+      - Environment variables:
+        - *CM_QAIC_DEVICES*: `0`
+        - *qaic_queue_length*: `6`
+      - Workflow:
 
     </details>
 
@@ -503,7 +616,7 @@ These keys can be updated via `--env.KEY=VALUE` or `env` dictionary in `@input.j
 * CM_KILT_REPO_URL: `https://github.com/GATEOverflow/kilt-mlperf`
 * CM_QAIC_DEVICES: `0`
 * kilt_max_wait_abs: `10000`
-* verbosity: `1`
+* verbosity: `0`
 * loadgen_trigger_cold_run: `0`
 
 </details>

diff --git a/cm-mlops/script/run-mlperf-inference-app/README.md b/cm-mlops/script/run-mlperf-inference-app/README.md
@@ -236,7 +236,7 @@ ___
 * --**division** MLPerf division {open,closed} (*open*)
 * --**category** MLPerf category {edge,datacenter,network} (*edge*)
 * --**device** MLPerf device {cpu,cuda,rocm,qaic} (*cpu*)
-* --**model** MLPerf model {resnet50,retinanet,bert-99,bert-99.9,3d-unet-99,3d-unet-99.9,rnnt,dlrm-v2-99,dlrm-v2-99.9,gptj-99,gptj-99.9,sdxl,llama2-70b-99,llama2-70b-99.9,mobilenet,efficientnet} (*retinanet*)
+* --**model** MLPerf model {resnet50,retinanet,bert-99,bert-99.9,3d-unet-99,3d-unet-99.9,rnnt,dlrm-v2-99,dlrm-v2-99.9,gptj-99,gptj-99.9,sdxl,llama2-70b-99,llama2-70b-99.9,mobilenet,efficientnet} (*resnet50*)
 * --**precision** MLPerf model precision {float32,float16,bfloat16,int8,uint8}
 * --**implementation** MLPerf implementation {reference,mil,nvidia-original,intel-original,qualcomm,tflite-cpp} (*reference*)
 * --**backend** MLPerf framework (backend) {onnxruntime,tf,pytorch,deepsparse,tensorrt,glow,tvm-onnx} (*onnxruntime*)
@@ -261,7 +261,7 @@ ___
 * --**compliance** Whether to run compliance tests (applicable only for closed division) {yes,no} (*no*)
 * --**dashboard_wb_project** W&B dashboard project (*cm-mlperf-dse-testing*)
 * --**dashboard_wb_user** W&B dashboard user (*cmind*)
-* --**hw_name** MLPerf hardware name (from [here](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-description/hardware))
+* --**hw_name** MLPerf hardware name (for example "gcp.c3_standard_8", "nvidia_orin", "lenovo_p14s_gen_4_windows_11", "macbook_pro_m1_2", "thundercomm_rb6" ...)
 * --**multistream_target_latency** Set MultiStream target latency
 * --**offline_target_qps** Set LoadGen Offline target QPS
 * --**quiet** Quiet run (select default values for all questions) (*True*)
@@ -311,6 +311,7 @@ r=cm.access({... , "division":...}
 * `--output_dir=value`  &rarr;  `OUTPUT_BASE_DIR=value`
 * `--output_summary=value`  &rarr;  `MLPERF_INFERENCE_SUBMISSION_SUMMARY=value`
 * `--output_tar=value`  &rarr;  `MLPERF_INFERENCE_SUBMISSION_TAR_FILE=value`
+* `--performance_sample_count=value`  &rarr;  `CM_MLPERF_LOADGEN_PERFORMANCE_SAMPLE_COUNT=value`
 * `--power=value`  &rarr;  `CM_SYSTEM_POWER=value`
 * `--precision=value`  &rarr;  `CM_MLPERF_MODEL_PRECISION=value`
 * `--preprocess_submission=value`  &rarr;  `CM_RUN_MLPERF_SUBMISSION_PREPROCESSOR=value`