Docs update to indicate use of conda-merge to generate install files

docker/conda/environments/cuda11.8_examples.yml

@@ -18,7 +18,10 @@
 # env with this file. ex:
 #   mamba env create -n  morpheus --file docker/conda/environments/cuda11.8_dev.yml
 #   conda activate morpheus
-#   mamba env update -n morpheus --file docker/conda/environments/cuda11.8_examples.yml
+#   mamba install -n base -c conda-forge conda-merge
+#   conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+#     docker/conda/environments/cuda${CUDA_VER}_examples.yml  > .tmp/merged.yml \
+#     && mamba env update -n morpheus --file ./merged.yml
 channels:
     - rapidsai
     - nvidia

docs/README.md

@@ -22,7 +22,11 @@ Additional packages required for building the documentation are defined in `./co
 ## Install Additional Dependencies
 From the root of the Morpheus repo:
 ```bash
-mamba env update -f docs/conda_docs.yml
+export CUDA_VER=11.8
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  docs/conda_docs.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
 ```
 
 ## Build Morpheus and Documentation

examples/digital_fingerprinting/production/Dockerfile

@@ -31,7 +31,10 @@ COPY ./conda_env.yml ./
 
 # Install DFP dependencies
 RUN source activate morpheus \
-    && mamba env update -n morpheus -f ./conda_env.yml
+    && mamba install -n base -c conda-forge conda-merge \
+    && conda run -n base --live-stream conda-merge /workspace/docker/conda/environments/cuda11.8_dev.yml \
+      ./conda_env.yml > ./merged.yml \
+    && mamba env update -n morpheus --file ./merged.yml
 
 # Set the tracking URI for mlflow
 ENV MLFLOW_TRACKING_URI="http://mlflow:5000"

examples/digital_fingerprinting/production/morpheus/benchmarks/README.md

@@ -19,51 +19,55 @@
 ### Set up Morpheus Dev Container
 
 If you don't already have the Morpheus Dev container, run the following to build it:
-```
+```bash
 ./docker/build_container_dev.sh
 ```
 
 Now run the container:
-```
+```bash
 ./docker/run_container_dev.sh
 ```
 
 Note that Morpheus containers are tagged by date. By default, `run_container_dev.sh` will try to use current date as tag. Therefore, if you are trying to run a container that was not built on the current date, you must set the `DOCKER_IMAGE_TAG` environment variable. For example,
-```
+```bash
 DOCKER_IMAGE_TAG=dev-221003 ./docker/run_container_dev.sh
 ```
 
 In the `/workspace` directory of the container, run the following to compile Morpheus:
-```
+```bash
 ./scripts/compile.sh
 ```
 
 Now install Morpheus:
-```
+```bash
 pip install -e /workspace
 ```
 
 Install additonal required dependencies:
-```
+```bash
 export CUDA_VER=11.8
-mamba env update -n morpheus --file docker/conda/environments/cuda${CUDA_VER}_examples.yml
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  docker/conda/environments/cuda${CUDA_VER}_examples.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
 ```
 
+
 Fetch input data for benchmarks:
-```
+```bash
 ./examples/digital_fingerprinting/fetch_example_data.py all
 ```
 
 ### Start MLflow
 
 MLflow is used as the model repository where the trained DFP models will be published and used for inference by the pipelines. Run the following to start MLflow in a host terminal window (not container):
 
-```
+```bash
 # from root of Morpheus repo
 cd examples/digital_fingerprinting/production
 ```
 
-```
+```bash
 docker compose up mlflow
 ```
 

examples/gnn_fraud_detection_pipeline/README.md

@@ -21,7 +21,11 @@ limitations under the License.
 Prior to running the GNN fraud detection pipeline, additional requirements must be installed in to your Conda environment. A supplemental requirements file has been provided in this example directory.
 
 ```bash
-mamba env update -n ${CONDA_DEFAULT_ENV} -f examples/gnn_fraud_detection_pipeline/requirements.yml
+export CUDA_VER=11.8
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  examples/gnn_fraud_detection_pipeline/requirements.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
 ```
 
 ## Running

examples/llm/agents/README.md

@@ -95,9 +95,14 @@ export SERPAPI_API_KEY="<YOUR_SERPAPI_API_KEY>"
 Install the required dependencies.
 
 ```bash
-mamba env update -n morpheus --file ${MORPHEUS_ROOT}/docker/conda/environments/cuda11.8_examples.yml
+export CUDA_VER=11.8
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  docker/conda/environments/cuda${CUDA_VER}_examples.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
 ```
 
+
 ### Running the Morpheus Pipeline
 
 The top level entrypoint to each of the LLM example pipelines is `examples/llm/main.py`. This script accepts a set

examples/llm/completion/README.md

@@ -35,9 +35,11 @@ limitations under the License.
 The primary goal of this example is to showcase the creation of a pipeline that integrates an LLM service with Morpheus. Although this example features a single implementation, the pipeline and its components are versatile and can be adapted to various scenarios with unique requirements. The following highlights different customization points within the pipeline and the specific choices made for this example:
 
 #### LLM Service
+
 - The pipeline is designed to support any LLM service that adheres to our LLMService interface. Compatible services include OpenAI, NeMo, or even local execution using llama-cpp-python. In this demonstration, we focus on utilizing NeMo as the LLM service, highlighting the advantages it offers over other LLM services and the seamless integration with the NeMo ecosystem. Furthermore, the pipeline can accommodate more complex configurations using NeMo + Inform without necessitating changes to the core pipeline.
 
 #### Downstream Tasks
+
 - Post LLM execution, the model's output can be leveraged for various tasks, including model training, analysis, or simulating an attack. In this particular example, we have simplified the implementation and focused solely on the LLMEngine.
 
 ### Pipeline Implementation
@@ -64,9 +66,14 @@ Before running the pipeline, ensure that the `NGC_API_KEY` environment variable
 Install the required dependencies.
 
 ```bash
-mamba env update -n morpheus --file ${MORPHEUS_ROOT}/docker/conda/environments/cuda11.8_examples.yml
+export CUDA_VER=11.8
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  docker/conda/environments/cuda${CUDA_VER}_examples.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
 ```
 
+
 #### Setting up NGC API Key
 
 For this example, we utilize the NeMo Service within NGC. To gain access, an NGC API key is required. Follow the
@@ -75,7 +82,6 @@ generate your NGC API key.
 
 Configure the following environment variables, with NGC_ORG_ID being optional:
 
-
 ```bash
 export NGC_API_KEY=<YOUR_API_KEY>
 export NGC_ORG_ID=<YOUR_NGC_ORG_ID>
@@ -105,7 +111,7 @@ python examples/llm/main.py completion [OPTIONS] COMMAND [ARGS]...
 
 - `--pipeline_batch_size INTEGER RANGE`
     - **Description**: Internal batch size for the pipeline. Can be much larger than the model batch size.
-    Also used for Kafka consumers.
+      Also used for Kafka consumers.
     - **Default**: `1024`
 
 - `--model_max_batch_size INTEGER RANGE`
@@ -123,7 +129,6 @@ python examples/llm/main.py completion [OPTIONS] COMMAND [ARGS]...
 - `--help`
     - **Description**: Show the help message with options and commands details.
 
-
 ### Running Morpheus Pipeline with OpenAI LLM service
 
 ```bash

models/training-tuning-scripts/fraud-detection-models/README.md

@@ -23,13 +23,17 @@ limitations under the License.
 
 Install packages for training GNN model.
 
-```
-mamba env update -n ${CONDA_DEFAULT_ENV} -f requirements.yml
+```bash
+export CUDA_VER=11.8
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  requirements.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
 ```
 
 ### Options for training and tuning models.
 
-```
+```bash
 python training.py --help
 Usage: training.py [OPTIONS]
 

morpheus/llm/services/nemo_llm_service.py

@@ -24,7 +24,10 @@
 
 IMPORT_ERROR_MESSAGE = (
     "NemoLLM not found. Install it and other additional dependencies by running the following command:\n"
-    "`mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`")
+    "`mamba install -n base -c conda-forge conda-merge`\n"
+    "`conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml "
+    "  docker/conda/environments/cuda${CUDA_VER}_examples.yml"
+    "  > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml`")
 
 try:
     import nemollm

morpheus/llm/services/openai_chat_service.py

@@ -25,7 +25,10 @@
 IMPORT_ERROR_MESSAGE = (
     "OpenAIChatService & OpenAIChatClient require the openai package to be installed. "
     "Install it by running the following command:\n"
-    "`mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`")
+    "`mamba install -n base -c conda-forge conda-merge`\n"
+    "`conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml "
+    "  docker/conda/environments/cuda${CUDA_VER}_examples.yml"
+    "  > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml`")
 
 try:
     import openai

morpheus/stages/input/arxiv_source.py

@@ -36,7 +36,10 @@
 
 IMPORT_ERROR_MESSAGE = (
     "ArxivSource requires additional dependencies to be installed. Install them by running the following command: "
-    "`mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`")
+    "`mamba install -n base -c conda-forge conda-merge`\n"
+    "`conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml "
+    "  docker/conda/environments/cuda${CUDA_VER}_examples.yml"
+    "  > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml`")
 
 
 @register_stage("from-arxiv")
@@ -45,7 +48,10 @@ class ArxivSource(PreallocatorMixin, SingleOutputSource):
     Source stage that downloads PDFs from arxiv and converts them to dataframes.
 
     This stage requires several additional dependencies to be installed. Install them by running the following command:
-    `mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`
+    mamba install -n base -c conda-forge conda-merge
+    conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml
+      docker/conda/environments/cuda${CUDA_VER}_examples.yml
+      > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml
 
     Parameters
     ----------

tests/benchmarks/README.md

@@ -23,12 +23,12 @@ Pull Docker image from NGC (https://ngc.nvidia.com/catalog/containers/nvidia:tri
 
 Example:
 
-```
+```bash
 docker pull nvcr.io/nvidia/tritonserver:23.06-py3
 ```
 
 ##### Start Triton Inference Server container
-```
+```bash
 cd ${MORPHEUS_ROOT}/models
 
 docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v $PWD:/models nvcr.io/nvidia/tritonserver:23.06-py3 tritonserver --model-repository=/models/triton-model-repo --model-control-mode=explicit --load-model sid-minibert-onnx --load-model abp-nvsmi-xgb --load-model phishing-bert-onnx
@@ -50,32 +50,32 @@ Once Triton server finishes starting up, it will display the status of all loade
 ### Set up Morpheus Dev Container
 
 If you don't already have the Morpheus Dev container, run the following to build it:
-```
+```bash
 ./docker/build_container_dev.sh
 ```
 
 Now run the container:
-```
+```bash
 ./docker/run_container_dev.sh
 ```
 
 Note that Morpheus containers are tagged by date. By default, `run_container_dev.sh` will try to use current date as tag. Therefore, if you are trying to run a container that was not built on the current date, you must set the `DOCKER_IMAGE_TAG` environment variable. For example,
-```
+```bash
 DOCKER_IMAGE_TAG=dev-221003 ./docker/run_container_dev.sh
 ```
 
 In the `/workspace` directory of the container, run the following to compile Morpheus:
-```
+```bash
 ./scripts/compile.sh
 ```
 
 Now install Morpheus:
-```
+```bash
 pip install -e /workspace
 ```
 
 Fetch input data for benchmarks:
-```
+```bash
 ./scripts/fetch_data.py fetch validation
 ```
 
@@ -102,7 +102,7 @@ Morpheus configurations for each workflow are managed using `e2e_test_configs.js
 
 Benchmarks for an individual workflow can be run using the following:
 
-```
+```bash
 cd tests/benchmarks
 
 pytest -s --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_pipelines.py::<test-workflow>
@@ -118,12 +118,12 @@ The `--benchmark-warmup` and `--benchmark-warmup-iterations` options are used to
 - `test_cloudtrail_ae_e2e`
 
 For example, to run E2E benchmarks on the SID NLP workflow:
-```
+```bash
 pytest -s --run_benchmark --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_pipelines.py::test_sid_nlp_e2e
 ```
 
 To run E2E benchmarks on all workflows:
-```
+```bash
 pytest -s --run_benchmark --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_pipelines.py
 ```
 
@@ -188,7 +188,11 @@ Note that the `test_cloudtrail_ae_e2e` benchmarks measure performance of a pipel
 
 You can use the same Dev container created here to run the Production DFP benchmarks. You would just need to install additional dependencies as follows:
 
-```
+```bash
 export CUDA_VER=11.8
-mamba env update -n morpheus --file docker/conda/environments/cuda${CUDA_VER}_examples.yml
+mamba install -n base -c conda-forge conda-merge
+conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml \
+  docker/conda/environments/cuda${CUDA_VER}_examples.yml > .tmp/merged.yml \
+  && mamba env update -n ${CONDA_DEFAULT_ENV} --file .tmp/merged.yml
+
 ```

tests/examples/llm/common/conftest.py

@@ -21,8 +21,11 @@
 @pytest.fixture(name="nemollm", autouse=True, scope='session')
 def nemollm_fixture(fail_missing: bool):
     """
-    All of the tests in this subdir require nemollm
+    All the tests in this subdir require nemollm
     """
     skip_reason = ("Tests for the WebScraperStage require the langchain package to be installed, to install this run:\n"
-                   "`mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`")
+                   "`mamba install -n base -c conda-forge conda-merge`\n"
+                   "`conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml "
+                   "  docker/conda/environments/cuda${CUDA_VER}_examples.yml"
+                   "  > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml`")
     yield import_or_skip("langchain", reason=skip_reason, fail_missing=fail_missing)

tests/llm/conftest.py

@@ -27,7 +27,10 @@ def nemollm_fixture(fail_missing: bool):
     Fixture to ensure nemollm is installed
     """
     skip_reason = ("Tests for the NeMoLLMService require the nemollm package to be installed, to install this run:\n"
-                   "`mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`")
+                   "`mamba install -n base -c conda-forge conda-merge`\n"
+                   "`conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml "
+                   "  docker/conda/environments/cuda${CUDA_VER}_examples.yml"
+                   "  > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml`")
     yield import_or_skip("nemollm", reason=skip_reason, fail_missing=fail_missing)
 
 
@@ -37,7 +40,10 @@ def openai_fixture(fail_missing: bool):
     Fixture to ensure openai is installed
     """
     skip_reason = ("Tests for the OpenAIChatService require the openai package to be installed, to install this run:\n"
-                   "`mamba env update -n ${CONDA_DEFAULT_ENV} --file docker/conda/environments/cuda11.8_examples.yml`")
+                   "`mamba install -n base -c conda-forge conda-merge`\n"
+                   "`conda run -n base --live-stream conda-merge docker/conda/environments/cuda${CUDA_VER}_dev.yml "
+                   "  docker/conda/environments/cuda${CUDA_VER}_examples.yml"
+                   "  > .tmp/merged.yml && mamba env update -n morpheus --file .tmp/merged.yml`")
     yield import_or_skip("openai", reason=skip_reason, fail_missing=fail_missing)