Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate NVTabular into Morpheus Core and replace existing column_info based workflows. #938

Merged
merged 75 commits into from
Jul 11, 2023
Merged

Integrate NVTabular into Morpheus Core and replace existing column_info based workflows. #938

merged 75 commits into from
Jul 11, 2023

Conversation

drobison00
Copy link
Contributor

Description

Closes #862
Closes #863

Integrates NVT into Morpheus core

  • Appropriate conda packages added
  • Existing process_dataframe calls in column_info.py are converted to nvt.Workflows
  • Backwards compatibility is maintained for anyone using DataFrameInputSchema objects

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@drobison00
First commit -- update environments to pull nvtabular
b541e53
@drobison00
Testing
e18642e
@drobison00
More testing code, developing a work around for being able to add dyn…
cbad33b 
…amic columns in an operator
@drobison00
More experimental code, converging on 'MutateOp' implementation
b00afb1
@drobison00
Land operators and transforms in their proper physical location and a…
249d801 
…dd unit tests
@drobison00
Merge 23.07 and fix protobuf incompatibility problem
c111b10
@drobison00
Column converters are in place, working through unit tests and proble…
d72fd33 
…ms with combining workflows
@drobison00
Checkpoint -- Finally have operators all working correctly
f345f5c
@drobison00
Functional graph composition on top of legacy Schema declaration
ff5680c
@drobison00
Working!
46c118e
@drobison00
Resolved remaining unittest issues
be42d6a
@drobison00
Add more unit tests
ae7b088
@drobison00
Switch to BFS approach, much cleaner and easier to remove duplicates
554db56
@drobison00
More unit tests, modularize BFS code
383d001
@drobison00
More unit tests
9b27206
@drobison00
More unit tests, coverage for post filtering and column preservation …
a377682 
…is done
@drobison00
NVT Tests passing, still working on data mappings
233c80b
@drobison00
Implement work arounds for Merlin StructDType handling, all unit test…
9622a02 
…s passing
@drobison00
Updates
806da3c
@drobison00
Add a few more unit tests, add process_workflow args
a9b167d
@drobison00
Merge latest 23.07
6e51004
@drobison00 drobison00 added enhancement Additional functionality added to an existing feature non-breaking Non-breaking change DO NOT MERGE PR should not be merged; see PR for details 2 - In Progress labels May 3, 2023
@drobison00 drobison00 self-assigned this May 3, 2023
@drobison00 drobison00 requested review from a team as code owners May 3, 2023 23:49
drobison00 and others added 24 commits July 10, 2023 13:19
@drobison00
Merge pull request #12 from dagardner-nv/devin_issue_862_dg
6fa9da7 
Misc cleanups
@dagardner-nv
Merge branch 'devin_issue_862' of github.com:drobison00/devin-morpheu…
9bfc33f 
…s-public into devin_issue_862
@dagardner-nv
Fix import sorting
64f24bc
@drobison00 @mdemoret-nv
Update morpheus/utils/nvt/schema_converters.py
ce6f9d1 
Co-authored-by: Michael Demoret <42954918+mdemoret-nv@users.noreply.github.com>
@dagardner-nv
pylint fixes for morpheus/utils/column_info.py
2411732
@drobison00 @mdemoret-nv
Update morpheus/utils/schema_transforms.py
90b4f8f 
Co-authored-by: Michael Demoret <42954918+mdemoret-nv@users.noreply.github.com>
@drobison00
PR feedback update
4e6c9b1
@dagardner-nv
pylint fixes for morpheus/utils/nvt/mutate.py
463cbd9
@drobison00
Merge pull request #13 from dagardner-nv/devin_issue_862_dg_lint
b7191ea 
WIP: Lint fixes
@drobison00
More PR feedback updates - docstrings, generalized sync decorator, ot…
fee9cf8 
…hers..
@drobison00
PR feedback updates
c5c665e
@drobison00
Update tritonclient install, docs updates
f840705
@drobison00
PR Feedback updates. Adds additional explicit type hints
1f581e9
@drobison00
Additional PR feedback updates
5aa6d1e
@drobison00
Formatting fixes
61dec6c
@drobison00
Merge branch-23.07
1f5acb2
@drobison00
Docs fixes, formatting
227509e
@drobison00
Formatting fixes
3e41f93
@drobison00
Formatting fixes
a2bfc14
@drobison00
Formatting fixes
d01eda0
@drobison00
Formatting fixes
a739181
@drobison00
Formatting fixes
85e7d68
@drobison00
PR feedback + test numba channel removal
37388e8
@mdemoret-nv
Removing the data class descriptor since pylint doesnt like it.
bb75522
mdemoret-nv
mdemoret-nv approved these changes Jul 11, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now. Thanks for all the hard work on this @drobison00

@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit c3d2fc7 into nv-morpheus:branch-23.07 Jul 11, 2023
4 checks passed
@mdemoret-nv mdemoret-nv mentioned this pull request Aug 21, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@drobison00 drobison00

Labels
enhancement Additional functionality added to an existing feature improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
3 participants
@drobison00 @mdemoret-nv @dagardner-nv