Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kafka integration tests #308

Merged
81 commits merged into from
Sep 7, 2022
Merged

kafka integration tests #308

81 commits merged into from
Sep 7, 2022

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Jul 27, 2022 •
edited
Loading

Breaking changes (#296):

  • Default group_id for KafkaSourceStage changes from "custreamz" to "morpheus"
  • Remove default topic value for KafkaSourceStage.

Other changes:

  • Adds Kafka integration tests for ABP, Hammah, Phishing & Sid validation pipelines.
  • Uses pytest_kafka to add zookeeper & kafka pytest fixtures
  • Configures pytest to only run kafka tests when --run_kafka command line flag is set.
  • Adds a stop_after argument to KafkaSourceStage when defined the source will stop once stop_after records have been read from the source.
  • Fixes bug in C++ impl preventing KafkaSourceStage from respecting max_batch_size ([BUG] KafkaSourceStage max_batch_size is ignored #309)

Note: Many of the kafka tests are also slower integration tests in effect requiring both the --run_kafka & --run_slow flags being passed to pytest

requires PR #290
fixes #258
fixes #296
fixes #309

@dagardner-nv
Simple copy test
5763a82
@dagardner-nv
Add instructions for partitioned test
2b47e46
@dagardner-nv
Add Kafka ABP validation instructions
bf16140
@dagardner-nv
Test data
81d4fd8
@dagardner-nv
Add a flag to optionally include or exclude the cudf ID column from C…
8b4554c 
…SV output, due to a known issue in cudf & pandas (rapidsai/cudf#11317 & pandas-dev/pandas#37600) this option has no effect on JSON output
@dagardner-nv
Replace usage of older MORPHEUS_HOME env var with MORPHEUS_ROOT
91636af
@dagardner-nv
Update copy test using Kafka's console producer and consumer
55f06c7
@dagardner-nv
Update partitioned testing:
abc4864
@dagardner-nv
Refactor ABP pipeline test
fafa8b7
@dagardner-nv
Compare with sorted keys
abb3343
@dagardner-nv
wip
ee9f2b1
@dagardner-nv
Skip python test, known bug
d5dfb91
@dagardner-nv
Move comparison logic from validation stage to be usable outside of a…
ee6c403 
… pipeline
@dagardner-nv
Rename morpheus.utils.logging to morpheus.utils.logger allowing other…
562e510 
… modules in morpheus.utils to import the std logger
@dagardner-nv
Make compare_df runnable from the command line
69d881b
@dagardner-nv
Fix bug in compare_df
ff913d1
@dagardner-nv
When rows don't match but are within the tolerances comparison.matche…
521cb87 
…s() will return non-None but diff_rows=0
@dagardner-nv
Add hammah user123 to test
d75b5d0
@dagardner-nv
Add phishing pipeline to kafka test
8a70db2
@dagardner-nv
Add sid validation to kafka testing
77a1f85
@dagardner-nv
Split bash commands
d914ddd
@dagardner-nv
Split bash commands
a79615a
@dagardner-nv
wip
d8d4027
@dagardner-nv
wip
f6fc77b
@dagardner-nv
wip
eab0051
@dagardner-nv
wip
ae5e089
@dagardner-nv
wip
eb407b7
@dagardner-nv
wip
ebcdda7
@dagardner-nv
Script no longer needed
f0c0fc8
@dagardner-nv
Fix includes
8f0da01
@dagardner-nv
Update docstring to match code
0f6b407
@dagardner-nv
Respect max_batch_size nv-morpheus#309
08a534d
@dagardner-nv
Revert configs for kafka, tell java not to try to use ip6
5f2b792
@dagardner-nv
Use a client_id for kafka producer, sleep after writing to ensure the…
112ec6b 
… reader is able to consume all messages in a single batch
@dagardner-nv
Set a poll interval of 1 second ensuring the kafka consumer has enoug…
363cc74 
…h time to read up to max_batch_size messages
@dagardner-nv
Remove unused import
2c62032
@dagardner-nv
Remove unused import
95a1eac
@dagardner-nv
Kafka source/sink now optionally provide a client_id
ad2b061 
Default group id for kafka source is now "morpheus" (was custreamz)
Removed dead branches in kafka_source_stage.py due to self._engine always being None.
Remove default value of "test_pcap" from kafka source
@dagardner-nv
Change casing for stop after exception to conform to clang-tidy
f5595fc
@dagardner-nv
Set client_id in source/sink tests
5e97836
@dagardner-nv
Merge branch 'david-test-kafka' into david-kafka-unittest
25267a8
@dagardner-nv
Merge branch 'branch-22.08' into david-test-kafka
f81e3a3
@dagardner-nv
Merge branch 'david-test-kafka' into david-kafka-unittest
60f847e
@dagardner-nv
Copy link
Contributor Author

rerun tests

@dagardner-nv dagardner-nv added 3 - Ready for Review and removed DO NOT MERGE PR should not be merged; see PR for details 2 - In Progress labels Aug 8, 2022
@dagardner-nv dagardner-nv changed the title Draft: kafka integration tests kafka integration tests Aug 16, 2022
@dagardner-nv dagardner-nv added breaking Breaking change and removed non-breaking Non-breaking change labels Aug 16, 2022
@dagardner-nv dagardner-nv changed the base branch from branch-22.08 to branch-22.09 August 22, 2022 16:49
mdemoret-nv
mdemoret-nv approved these changes Sep 6, 2022
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@mdemoret-nv
Copy link
Contributor

@gpucibot merge

@ghost ghost merged commit 5c92142 into nv-morpheus:branch-22.09 Sep 7, 2022
@dagardner-nv dagardner-nv deleted the david-kafka-unittest branch February 12, 2024 23:21
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees
No one assigned
Labels
breaking Breaking change improvement Improvement to existing functionality
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
2 participants
@dagardner-nv @mdemoret-nv