Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Running the nlp_si_detection example pipeline prints a warning about a slow edge #186

Closed
dagardner-nv opened this issue Jun 23, 2022 · 0 comments
Labels
bug Something isn't working Needs Triage Need team to review and classify

Comments

@dagardner-nv
Copy link
Contributor

Describe the bug
The nlp_si_detection pipeline example prints the following warning message about a slow edge. The culprit appears to be the SerializeStage

W20220623 15:41:12.778241 2000453 node.hpp:271] WARNING: A slow edge connection between C++ nodes '[with T = std::shared_ptr<morpheus::MultiResponseProbsMessage>]' and '[with T = std::shared_ptr<morpheus::MultiMessage>]' has been detected. Performance between these nodes can be improved by registering an EdgeConverter at compile time. Without this, conversion to an intermediate python type will be necessary (i.e. C++ -> Python -> C++).

Steps/Code to reproduce bug

morpheus --log_level=DEBUG \
   `# Run a pipeline with 8 threads and a model batch size of 32 (Must match Triton config)` \
   run --num_threads=8 --pipeline_batch_size=1024 --model_max_batch_size=32 \
   `# Specify a NLP pipeline with 256 sequence length (Must match Triton config)` \
   pipeline-nlp --model_seq_length=256 \
   `# 1st Stage: Read from file` \
   from-file --filename=$MORPHEUS_ROOT/examples/data/pcap_dump.jsonlines \
   `# 2nd Stage: Deserialize from JSON strings to objects` \
   deserialize \
   `# 3rd Stage: Preprocessing converts the input data into BERT tokens` \
   preprocess --vocab_hash_file=$MORPHEUS_ROOT/morpheus/data/bert-base-uncased-hash.txt --do_lower_case=True --truncation=True \
   `# 4th Stage: Send messages to Triton for inference. Specify the model loaded in Setup` \
   inf-triton --model_name=sid-minibert-onnx --server_url=localhost:8000 --force_convert_inputs=True \
   `# 5th Stage: Monitor stage prints throughput information to the console` \
   monitor --description "Inference Rate" --smoothing=0.001 --unit inf \
   `# 6th Stage: Add results from inference to the messages` \
   add-class \
   `# 7th Stage: Filtering removes any messages that did not detect SI` \
   filter \
   `# 8th Stage: Convert from objects back into strings` \
   serialize --exclude '^_ts_' \
   `# 9th Stage: A second Monitor stage to monitor the back-half of the pipeline` \
   monitor --description "Serialize Rate" --smoothing=0.001 \
   `# 10th Stage: Write out the JSON lines to the detections.jsonlines file` \
   to-file --filename=detections.jsonlines --overwrite

Expected behavior
No warnings

Environment overview (please complete the following information)

  • Environment location: Bare-metal
  • Method of Morpheus install: from source
@dagardner-nv dagardner-nv added bug Something isn't working Needs Triage Need team to review and classify labels Jun 23, 2022
dagardner-nv added a commit to dagardner-nv/Morpheus that referenced this issue Jun 24, 2022
+ Performs a non-inplace morpheus build.
+ Adds $ORIGIN to rpath of build binaries, fixing issue where libmorpheus.so was unable to find libmorpheus-utils.so (fixes nv-morpheus#111 )
+ ~~Removes the need for a gpu for the build stage~~ Blocked by Neo/nvml issue nv-morpheus#186
+ Common operations moved to common.sh
+ Fixes ci scripts so that they work locally (fixes nv-morpheus#114 )
+ Improve the way sccache is configured (fixes nv-morpheus#110 )

This pull req incorporates changes from nv-morpheus#62

Authors:
  - David Gardner (https://github.com/dagardner-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Devin Robison (https://github.com/drobison00)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: nv-morpheus#107
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Needs Triage Need team to review and classify
Projects
Status: Done
Development

No branches or pull requests

1 participant