Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests for gnn_fraud_detection_pipeline & ransomware_detection #904

Merged
merged 185 commits into from
May 15, 2023
Merged

Tests for gnn_fraud_detection_pipeline & ransomware_detection #904

merged 185 commits into from
May 15, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Apr 24, 2023 •
edited
Loading

Description

  • Renamed examples/gnn_fraud_detection_pipeline/requirements.yml to docker/conda/environments/cuda11.8_examples.yml, replacing the original with a symlink.
  • Test stage includes additional packages needed for gnn_fraud_detection_pipeline & ransomware_detection pipelines
  • Locally pytest will skip these if the deps are missing
  • Add a new --fail_missing flag which will cause tests to fail instead of skip on missing deps
  • Remove unused redundant apt install of nodejs & npm
  • Use openjdk-11-jre-headless for Kafka instead of openjdk-11-jdk, removing un-needed deps from the image like alsa and GL.

gnn_fraud_detection_pipeline changes:

ransomware_detection changes:

  • Explicitly exclude ldrmodules_df_path from model_features
  • Document that C++ execution is currently unsupported
  • Move nested methods in CreateFeaturesRWStage to methods on the class allowing them to be tested
  • Perform pre-allocation of needed columns in PreprocessingRWStage
  • Update dependencies for training script, and update due to API changes in Tensorflow

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@dagardner-nv
First 20 lines of examples/abp_pcap_detection/pcap_out.jsonlines pull…
9e9222d 
…ed out here since the original is large and not part of the dataset pulled by the test stage in CI
@dagardner-nv
Remove flow_id and rollup_time cols
16de6f7
@dagardner-nv
Test data
a239f41
@dagardner-nv
Fix type-o
59dc4f9
@dagardner-nv
Add a get_stage_class method to StageInfo
d7e23be
@dagardner-nv
Fixture to reset the PluginManager and GlobalStageRegistry
fb7829e
@dagardner-nv
Add helper method to assist pulling a stage from the plugin manager
454eeea
@dagardner-nv
End-to-end test for AbpPcapPreprocessingStage
e3007b7
@dagardner-nv
Test asserting that needed columns are defined
c354361
@dagardner-nv
Request pre-allocation
3fcb19a
@dagardner-nv
Fix setting get_stage_class in LazyStageInfo
dd41ddc
@dagardner-nv
Merge branch 'branch-23.03' into pcap_preprocessing-pre-allocating-797
1781bd6
@dagardner-nv
Merge branch 'branch-23.07' into pcap_preprocessing-pre-allocating-797
5d4eb0a
@dagardner-nv
reload_modules fixture reloads requested modules before and after the…
493e7fb 
… test
@dagardner-nv
Move test_abp_pcap_preprocessing.py to an examples subdir
81002ac
@dagardner-nv
Fixture to restore sys.path
ad9e578
@dagardner-nv
Add type hints
664b2d7
@dagardner-nv
Add test for pass-thru stages
c8992cd
@dagardner-nv
Fix invocation of reload_modules fixture
f460f66
@dagardner-nv
Deleting redundant copies of bert hash files
7c3a2bd
@dagardner-nv
Update references to files bert-base-cased-hash.txt and bert-base-unc…
e0185a2 
…ased-hash.txt to the version in the morpheus/data dir
@dagardner-nv
Revert "Deleting redundant copies of bert hash files"
219e911 
This reverts commit 7c3a2bd.
@dagardner-nv
Remove redundant copies of bert-base-cased-hash.txt and bert-base-unc…
c49c039 
…ased-hash.txt
@dagardner-nv
Remove redundant copue of bert-base-cased-vocab.txt
af42993
@dagardner-nv
Subset of models/log-parsing-models/log-parsing-config-20220418.json …
59b1e88 
…with just the id2label actually used by LogParsingPostProcessingStage
@dagardner-nv
Fetch the bert cased vocab file for tests
6b52af3
@dagardner-nv
Test for log parsing post proc example
85d2924
@dagardner-nv
Move to sub-dir
4df19b0
@dagardner-nv
Test for https://github.com/rapidsai/cudf/issues/130850
7a7f4d5
@dagardner-nv
Explicitly set the encoding as a work-around for nv-morpheus#859
5985006
@dagardner-nv dagardner-nv mentioned this pull request May 5, 2023
Merged
3 tasks
@mdemoret-nv mdemoret-nv mentioned this pull request May 10, 2023
Closed
@dagardner-nv dagardner-nv mentioned this pull request May 11, 2023
Merged
3 tasks
@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit cef498a into nv-morpheus:branch-23.07 May 15, 2023
rapids-bot bot pushed a commit that referenced this pull request May 15, 2023
@dagardner-nv
Helper scripts for running CI locally (#949)
0c99762 
Usage looks like:
```
ci/scripts/run_ci_local.sh checks
ci/scripts/run_ci_local.sh build
ci/scripts/run_ci_local.sh docs
ci/scripts/run_ci_local.sh test
```

Build artifacts are stored locally in `.tmp/local_ci_tmp` or can be overridden by defining `LOCAL_CI_TMP`

To run the entire CI pipeline:
```
ci/scripts/run_ci_local.sh all
```

To run bash inside of the CI containers:
```
# Runs in the build container
ci/scripts/run_ci_local.sh

# Runs in the test container
USE_GPU=1 ci/scripts/run_ci_local.sh
```

Includes changes from #904
Fixes #948

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #949
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG]: stellargraph requires python < 3.9 breaking the gnn_fraud_detection_pipeline example
2 participants
@dagardner-nv @mdemoret-nv