Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move data dir to models #1099

Merged
merged 7 commits into from
Jul 25, 2023
Merged

Move data dir to models #1099

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
46969f9
Move morpheus/data -> models/data
dagardner-nv Jul 24, 2023
f4fb4e9
Update checking of vocab file to allow for a symlink
dagardner-nv Jul 24, 2023
dbe7a7f
Add symlink morpheus/data --> models/data
dagardner-nv Jul 24, 2023
3662c38
Update symlinks
dagardner-nv Jul 24, 2023
a61df8a
Explicitly install fastjsonschema via conda-forge
dagardner-nv Jul 25, 2023
6fd5e15
Revert "Explicitly install fastjsonschema via conda-forge"
dagardner-nv Jul 25, 2023
4ffbad1
Update ci image versions
dagardner-nv Jul 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pull_request.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
uses: ./.github/workflows/ci_pipe.yml
with:
run_check: ${{ startsWith(github.ref_name, 'pull-request/') }}
container: nvcr.io/ea-nvidia-morpheus/morpheus:morpheus-ci-build-230710
test_container: nvcr.io/ea-nvidia-morpheus/morpheus:morpheus-ci-test-230710
container: nvcr.io/ea-nvidia-morpheus/morpheus:morpheus-ci-build-230725
test_container: nvcr.io/ea-nvidia-morpheus/morpheus:morpheus-ci-test-230725
secrets:
NGC_API_KEY: ${{ secrets.NGC_API_KEY }}
2 changes: 1 addition & 1 deletion ci/scripts/run_ci_local.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ GIT_BRANCH=$(git branch --show-current)
GIT_COMMIT=$(git log -n 1 --pretty=format:%H)

LOCAL_CI_TMP=${LOCAL_CI_TMP:-${MORPHEUS_ROOT}/.tmp/local_ci_tmp}
CONTAINER_VER=${CONTAINER_VER:-230710}
CONTAINER_VER=${CONTAINER_VER:-230725}
CUDA_VER=${CUDA_VER:-11.8}
DOCKER_EXTRA_ARGS=${DOCKER_EXTRA_ARGS:-""}

Expand Down
0 morpheus/data/bert-base-cased-hash.txt → models/data/bert-base-cased-hash.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/bert-base-cased-vocab.txt → models/data/bert-base-cased-vocab.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/bert-base-uncased-hash.txt → models/data/bert-base-uncased-hash.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/bert-base-uncased-vocab.txt → models/data/bert-base-uncased-vocab.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/columns_ae.txt → models/data/columns_ae.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/columns_ae_azure.txt → models/data/columns_ae_azure.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/columns_ae_cloudtrail.txt → models/data/columns_ae_cloudtrail.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/columns_ae_duo.txt → models/data/columns_ae_duo.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/columns_fil.txt → models/data/columns_fil.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/labels_ae.txt → models/data/labels_ae.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/labels_nlp.txt → models/data/labels_nlp.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/labels_phishing.txt → models/data/labels_phishing.txt
View file
Open in desktop
File renamed without changes.
0 morpheus/data/public_suffix_list.dat → models/data/public_suffix_list.dat
View file
Open in desktop
File renamed without changes.
0 morpheus/data/splunk_notable_regex.yaml → models/data/splunk_notable_regex.yaml
View file
Open in desktop
File renamed without changes.
0 morpheus/data/windows_event_regex.yaml → models/data/windows_event_regex.yaml
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion models/training-tuning-scripts/log-parsing-models/resources/bert-base-cased-hash.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
../../../../morpheus/data/bert-base-cased-hash.txt
../../../data/bert-base-cased-hash.txt
2 changes: 1 addition & 1 deletion models/training-tuning-scripts/sid-models/resources/bert-base-cased-hash.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
../../../../morpheus/data/bert-base-cased-hash.txt
../../../data/bert-base-cased-hash.txt
2 changes: 1 addition & 1 deletion models/training-tuning-scripts/sid-models/resources/bert-base-cased-vocab.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
../../../../morpheus/data/bert-base-cased-vocab.txt
../../../data/bert-base-cased-vocab.txt
2 changes: 1 addition & 1 deletion models/training-tuning-scripts/sid-models/resources/bert-base-uncased-hash.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
../../../../morpheus/data/bert-base-uncased-hash.txt
../../../data/bert-base-uncased-hash.txt
2 changes: 1 addition & 1 deletion models/training-tuning-scripts/sid-models/resources/bert-base-uncased-vocab.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
../../../../morpheus/data/bert-base-uncased-vocab.txt
../../../data/bert-base-uncased-vocab.txt
1 change: 1 addition & 0 deletions morpheus/data
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../models/data
6 changes: 3 additions & 3 deletions tests/test_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -715,7 +715,7 @@ def test_pipeline_nlp(self, config, callback_values):
assert isinstance(deserialize, DeserializeStage)

assert isinstance(process_nlp, PreprocessNLPStage)
assert process_nlp._vocab_hash_file == vocab_file_name
assert process_nlp._vocab_hash_file == os.path.realpath(vocab_file_name)
assert process_nlp._truncation
assert process_nlp._do_lower_case
assert not process_nlp._add_special_tokens
Expand Down Expand Up @@ -838,7 +838,7 @@ def test_pipeline_nlp_all(self, config, callback_values, tmp_path, mlflow_uri):
assert dropna._column == 'xyz'

assert isinstance(process_nlp, PreprocessNLPStage)
assert process_nlp._vocab_hash_file == vocab_file_name
assert process_nlp._vocab_hash_file == os.path.realpath(vocab_file_name)
assert process_nlp._truncation
assert process_nlp._do_lower_case
assert not process_nlp._add_special_tokens
Expand Down Expand Up @@ -927,7 +927,7 @@ def test_pipeline_nlp_relative_paths(self, config, callback_values):
# pylint: disable=unused-variable
[file_source, deserialize, process_nlp, triton_inf, monitor, add_class, validation, serialize, to_file] = stages

assert process_nlp._vocab_hash_file == vocab_file_name
assert process_nlp._vocab_hash_file == os.path.realpath(vocab_file_name)

@pytest.mark.replace_callback('pipeline_nlp')
def test_pipeline_nlp_relative_path_precedence(self, config, callback_values, tmp_path):
Expand Down