Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix environment yaml paths #1551

Merged
merged 5 commits into from
Mar 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions conda/environments/examples_cuda-121_arch-x86_64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,5 @@ dependencies:
- milvus==2.3.5
- nemollm
- pymilvus==2.3.6
- python-logging-loki
name: examples_cuda-121_arch-x86_64
21 changes: 10 additions & 11 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,15 @@ ARG MORPHEUS_ROOT_HOST
ARG CUDA_MAJOR_VER
ARG CUDA_MINOR_VER

COPY ${MORPHEUS_ROOT_HOST}/conda/environments/runtime_cuda-${CUDA_MAJOR_VER}${CUDA_MINOR_VER}_arch-x86_64.yaml ./conda/environments/runtime.yaml
# Only copy specific files/folders over that are necessary for runtime
COPY "${MORPHEUS_ROOT_HOST}/conda/environments/*.yaml" "./conda/environments/"
COPY "${MORPHEUS_ROOT_HOST}/docker" "./docker"
COPY "${MORPHEUS_ROOT_HOST}/docs" "./docs"
COPY "${MORPHEUS_ROOT_HOST}/examples" "./examples"
COPY "${MORPHEUS_ROOT_HOST}/models" "./models"
COPY "${MORPHEUS_ROOT_HOST}/scripts" "./scripts"
COPY "${MORPHEUS_ROOT_HOST}/*.md" "./"
COPY "${MORPHEUS_ROOT_HOST}/LICENSE" "./"

# Mount Morpheus conda package build in `conda_bld_morpheus`
RUN --mount=type=bind,from=conda_bld_morpheus,source=/opt/conda/conda-bld,target=/opt/conda/conda-bld \
Expand All @@ -254,18 +262,9 @@ RUN --mount=type=bind,from=conda_bld_morpheus,source=/opt/conda/conda-bld,target
-c defaults \
morpheus &&\

/opt/conda/bin/conda env update --solver=libmamba -n morpheus --file conda/environments/runtime.yaml && \
/opt/conda/bin/conda env update --solver=libmamba -n morpheus --file conda/environments/runtime_cuda-${CUDA_MAJOR_VER}${CUDA_MINOR_VER}_arch-x86_64.yaml && \
conda clean -afy

# Only copy specific files/folders over that are necessary for runtime
COPY "${MORPHEUS_ROOT_HOST}/docker" "./docker"
COPY "${MORPHEUS_ROOT_HOST}/docs" "./docs"
COPY "${MORPHEUS_ROOT_HOST}/examples" "./examples"
COPY "${MORPHEUS_ROOT_HOST}/models" "./models"
COPY "${MORPHEUS_ROOT_HOST}/scripts" "./scripts"
COPY "${MORPHEUS_ROOT_HOST}/*.md" "./"
COPY "${MORPHEUS_ROOT_HOST}/LICENSE" "./"

# Use morpheus by default
CMD [ "morpheus" ]

Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Additional packages required for building the documentation are defined in `./co
## Install Additional Dependencies
From the root of the Morpheus repo:
```bash
conda env update --solver=libmamba -n morpheus --file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune
conda env update --solver=libmamba -n morpheus --file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune
```

## Build Morpheus and Documentation
Expand Down
2 changes: 1 addition & 1 deletion docs/source/developer_guide/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ git submodule update --init --recursive

1. Create the Morpheus Conda environment
```bash
conda env create --solver=libmamba -n morpheus --file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml
conda env create --solver=libmamba -n morpheus --file conda/environments/dev_cuda-121_arch-x86_64.yaml
conda activate morpheus
```

Expand Down
5 changes: 1 addition & 4 deletions examples/digital_fingerprinting/production/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,9 @@ SHELL ["/bin/bash", "-c"]
# Set the workdir to the DFP base folder
WORKDIR /workspace/examples/digital_fingerprinting/

# Copy the conda_env file
COPY ./conda_env.yml ./

# Install DFP dependencies
RUN source activate morpheus \
&& mamba env update -n morpheus --file ../../conda/environments/examples_cuda-121_arch-x86_64.yaml
&& mamba env update -n morpheus --file /workspace/conda/environments/examples_cuda-121_arch-x86_64.yaml

# Set the tracking URI for mlflow
ENV MLFLOW_TRACKING_URI="http://mlflow:5000"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Install additonal required dependencies:
```bash
mamba env update \
-n ${CONDA_DEFAULT_ENV} \
--file ./morpheus/conda/environments/examples_cuda-121_arch-x86_64.yaml
--file ./conda/environments/examples_cuda-121_arch-x86_64.yaml
```


Expand Down
2 changes: 1 addition & 1 deletion examples/gnn_fraud_detection_pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Prior to running the GNN fraud detection pipeline, additional requirements must
export CUDA_VER=11.8
mamba env update \
-n ${CONDA_DEFAULT_ENV} \
--file ./morpheus/conda/environments/examples_cuda-121_arch-x86_64.yaml
--file ./conda/environments/examples_cuda-121_arch-x86_64.yaml
```

## Running
Expand Down
2 changes: 1 addition & 1 deletion examples/llm/agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Install the required dependencies.
```bash
mamba env update \
-n ${CONDA_DEFAULT_ENV} \
--file ./morpheus/conda/environments/examples_cuda-121_arch-x86_64.yaml
--file ./conda/environments/examples_cuda-121_arch-x86_64.yaml
```


Expand Down
2 changes: 1 addition & 1 deletion examples/llm/completion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Install the required dependencies.
```bash
mamba env update \
-n ${CONDA_DEFAULT_ENV} \
--file ./morpheus/conda/environments/examples_cuda-121_arch-x86_64.yaml
--file ./conda/environments/examples_cuda-121_arch-x86_64.yaml
```


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Install packages for training GNN model.
```bash
mamba env update \
-n ${CONDA_DEFAULT_ENV} \
--file ./morpheus/conda/environments/model-utils-121_arch-x86_64.yaml
--file ./conda/environments/model-utils-121_arch-x86_64.yaml
```

### Options for training and tuning models.
Expand Down
2 changes: 1 addition & 1 deletion morpheus/llm/services/nemo_llm_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
IMPORT_ERROR_MESSAGE = (
"NemoLLM not found. Install it and other additional dependencies by running the following command:\n"
"`conda env update --solver=libmamba -n morpheus "
"--file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
"--file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")

try:
import nemollm
Expand Down
2 changes: 1 addition & 1 deletion morpheus/llm/services/openai_chat_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
IMPORT_ERROR_MESSAGE = ("OpenAIChatService & OpenAIChatClient require the openai package to be installed. "
"Install it by running the following command:\n"
"`conda env update --solver=libmamba -n morpheus "
"--file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
"--file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")

try:
import openai
Expand Down
2 changes: 1 addition & 1 deletion tests/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,5 +200,5 @@ You can use the same Dev container created here to run the Production DFP benchm
```bash
mamba env update \
-n ${CONDA_DEFAULT_ENV} \
--file ./morpheus/conda/environments/examples_cuda-121_arch-x86_64.yaml
--file ./conda/environments/examples_cuda-121_arch-x86_64.yaml
```
4 changes: 2 additions & 2 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -1033,7 +1033,7 @@ def nemollm_fixture(fail_missing: bool):
"""
skip_reason = ("Tests for the NeMoLLMService require the nemollm package to be installed, to install this run:\n"
"`conda env update --solver=libmamba -n morpheus "
"--file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
"--file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
yield import_or_skip("nemollm", reason=skip_reason, fail_missing=fail_missing)


Expand All @@ -1044,7 +1044,7 @@ def openai_fixture(fail_missing: bool):
"""
skip_reason = ("Tests for the OpenAIChatService require the openai package to be installed, to install this run:\n"
"`conda env update --solver=libmamba -n morpheus "
"--file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
"--file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
yield import_or_skip("openai", reason=skip_reason, fail_missing=fail_missing)


Expand Down
2 changes: 1 addition & 1 deletion tests/examples/llm/common/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,5 +60,5 @@ def langchain_fixture(fail_missing: bool):

skip_reason = ("Tests for the WebScraperStage require the langchain package to be installed, to install this run:\n"
"`conda env update --solver=libmamba -n morpheus "
"--file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
"--file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
yield import_or_skip("langchain", reason=skip_reason, fail_missing=fail_missing)
2 changes: 1 addition & 1 deletion tests/stages/arxiv/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
SKIP_REASON = ("Tests for the arxiv_source require a number of packages not installed in the Morpheus development "
"environment. To install these run:\n"
"`conda env update --solver=libmamba -n morpheus "
"--file morpheus/conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")
"--file conda/environments/dev_cuda-121_arch-x86_64.yaml --prune`")


@pytest.fixture(name="arxiv", autouse=True, scope='session')
Expand Down
Loading