Merge 23.11 #3
Merged
Merge 23.11 #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…morpheus#1111) This PR changes the stub generation to be committed into the repo instead of as part of the build. This will allow CI steps that need the PYI files to not require a GPU as well. This works by always copying the PYI files back to the source tree so they will show up as diffs. Any diffs during the test phase will be reported as an error. Closes nv-morpheus#766 Authors: - Michael Demoret (https://github.com/mdemoret-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1111
* Add a `--viz_direction` flag to CLI * Cherry-pick'd [9532bfe](nv-morpheus@9532bfe) from @mdemoret-nv fixes nv-morpheus#643 Authors: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1119
* Publish artifact urls in job summaries fixes nv-morpheus#1105 * Remove redundant apt installs for packages that exist in the conda env fixes nv-morpheus#1106 * Checks now performs a build first. This allows pylint to accurately warn for undefined members. fixes nv-morpheus#956 * Custom pylint checker to validate correct importing of `morpheus._lib` fixes nv-morpheus#727 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1112
…heus#1088) The `gitutils.py` script often fails when you are working in a PR, or if there has been a commit to your target branch since you created your current branch. This improves the utility by using the Github CLI to query the PR information when possible and get the true target branch for the current PR. If the Github CLI is not available, or the branch isnt in a PR, it falls back to the previous functionality. Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1088
…morpheus#1113) * Call the `manual_seed` method from within the subprocess, this ensures the subprocess runs deterministically. * Add a sleep to a busy-loop in `morpheus/models/dfencoder/multiprocessing.py` * Misc pylint fixes fixes nv-morpheus#1021 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1113
…hen S3 is not in use (nv-morpheus#1120) * `DFPFileToDataFrameStage` is no longer an S3 specific stage, but some of the log messages and the `unique-name` still refer to S3. Fixes nv-morpheus#451 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1120
The test reorganization resulted in a couple of problems. First, not all tests were being run due to a mistake I made in handling cmake function arguments. Second, the `test_cuda` test was producing a segfault inside of triton client while initializing curl, but only in debug builds. The fix was to the segfault was not obvious, but apparently only occured when the test utilities were compiled as a separate library. Interestingly, the other tests were not affected. Perhaps there is some global state that needs to be managed that is not managed in the simple `test_cuda` test, which only calls a Matx function (also important, the Matx function result is not inspected, so the test itself is not comprehensive, as Matx does not produce an error if the kernel is not available for the given architecture). Depends on nv-morpheus#1110 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1118
Update the devcontainer base to 23.10 (this has no relation to cudf versioning).
While there are other improvements, the main reason I'm upgrading is due to a newer version of git, which allows inline public key signing of commits - something we need to be able to commit changes while inside the container without having to manage separate container-specific signing keys.
Example of signing key usage (from host, inherited by devcontainer):
```
ssh-keygen -t ed25519 -C "example@nvidia.com"
git config --global user.signingkey "key::$(cat ~/.ssh/id_ed25519.pub)"
git config --global commit.gpgsign true
git config --global gpg.format ssh
mkdir -p ~/.config/git
git config --global gpg.ssh.allowedSignersFile ~/.config/git/allowed_signers
awk 'BEGIN {FS=" "; OFS=" "} {print $3, $1, $2}' <(ssh-add -L) > ~/.config/git/allowed_signers
```
Authors:
- Christopher Harris (https://github.com/cwharris)
Approvers:
- Michael Demoret (https://github.com/mdemoret-nv)
URL: nv-morpheus#1116
…rpheus#1137) Closes nv-morpheus#1132 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1137
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
tzemicheal
pushed a commit
that referenced
this pull request
Jan 6, 2024
I have a few changes here. Some of them might not make it depending on how others feel.
- Move test utilities in to a library to share across tests.
- Move tests in to individual executable to enforce individual tests to not sharing code.
- Use `gtest_discover_tests` to let `ctest` run and show all tests individually.
- Use `gtest::gtest_main` and remove explicit main entry file.
- ~Build and run tests by default.~
- Ability to filter tests by `ctest --test-dir build -R "TestCuda.*"`
```
+ echo 'Running CTest...'
Running CTest...
+ ctest --test-dir build
Internal ctest changing into directory: /workspaces/morpheus/build
Test project /workspaces/morpheus/build
Start 1: TestCuda.LargeShape
1/35 Test #1: TestCuda.LargeShape ....................................... Passed 0.17 sec
Start 2: TestDataLoader.DataLoaderInitializationTest
2/35 Test #2: TestDataLoader.DataLoaderInitializationTest ............... Passed 0.07 sec
Start 3: TestDataLoader.DataLoaderRegisterLoaderTest
3/35 Test #3: TestDataLoader.DataLoaderRegisterLoaderTest ............... Passed 0.07 sec
Start 4: TestDataLoader.DataLoaderRemoveLoaderTest
4/35 Test #4: TestDataLoader.DataLoaderRemoveLoaderTest ................. Passed 0.07 sec
Start 5: TestDataLoader.PayloadLoaderTest
5/35 Test #5: TestDataLoader.PayloadLoaderTest .......................... Passed 1.42 sec
Start 6: TestDataLoader.FileLoaderTest
6/35 Test #6: TestDataLoader.FileLoaderTest ............................. Passed 2.65 sec
Start 7: TestControlMessage.InitializationTest
7/35 Test #7: TestControlMessage.InitializationTest ..................... Passed 0.07 sec
Start 8: TestControlMessage.SetMessageTest
8/35 Test #8: TestControlMessage.SetMessageTest ......................... Passed 0.07 sec
Start 9: TestControlMessage.TaskTest
9/35 Test #9: TestControlMessage.TaskTest ............................... Passed 0.07 sec
Start 10: TestControlMessage.PayloadTest
10/35 Test #10: TestControlMessage.PayloadTest ............................ Passed 1.45 sec
Start 11: TestDataLoaderModule.DataLoaderModuleInitializationTest
11/35 Test #11: TestDataLoaderModule.DataLoaderModuleInitializationTest ... Passed 0.07 sec
Start 12: TestDataLoaderModule.EndToEndPayloadDataLoaderTest
12/35 Test #12: TestDataLoaderModule.EndToEndPayloadDataLoaderTest ........ Passed 0.12 sec
Start 13: TestDeserializers.GetIndexColCountNoIdxFromFile
13/35 Test #13: TestDeserializers.GetIndexColCountNoIdxFromFile ........... Passed 1.39 sec
Start 14: TestDeserializers.GetIndexColCountWithIdxFromFile
14/35 Test nv-morpheus#14: TestDeserializers.GetIndexColCountWithIdxFromFile ......... Passed 1.37 sec
Start 15: TestDeserializers.GetIndexColCountNoIdxSimilarName
15/35 Test nv-morpheus#15: TestDeserializers.GetIndexColCountNoIdxSimilarName ........ Passed 0.19 sec
Start 16: TestDeserializers.GetIndexColCountIdx
16/35 Test nv-morpheus#16: TestDeserializers.GetIndexColCountIdx ..................... Passed 0.18 sec
Start 17: TestDeserializers.GetIndexColCountValidNameInvalidType
17/35 Test nv-morpheus#17: TestDeserializers.GetIndexColCountValidNameInvalidType .... Passed 0.16 sec
Start 18: TestDevMemInfo.RmmBufferConstructor
18/35 Test nv-morpheus#18: TestDevMemInfo.RmmBufferConstructor ....................... Passed 0.16 sec
Start 19: TestDevMemInfo.VoidPtrConstructor
19/35 Test nv-morpheus#19: TestDevMemInfo.VoidPtrConstructor ......................... Passed 0.25 sec
Start 20: TestDevMemInfo.MakeNewBuffer
20/35 Test nv-morpheus#20: TestDevMemInfo.MakeNewBuffer .............................. Passed 0.25 sec
Start 21: TestFileInOut.RoundTripCSV
21/35 Test nv-morpheus#21: TestFileInOut.RoundTripCSV ................................ Passed 2.67 sec
Start 22: TestFileInOut.RoundTripJSON
22/35 Test nv-morpheus#22: TestFileInOut.RoundTripJSON ............................... Passed 2.69 sec
Start 23: TestMatxUtil.ReduceMax1d
23/35 Test nv-morpheus#23: TestMatxUtil.ReduceMax1d .................................. Passed 0.17 sec
Start 24: TestMatxUtil.ReduceMax2dRowMajor
24/35 Test nv-morpheus#24: TestMatxUtil.ReduceMax2dRowMajor .......................... Passed 0.17 sec
Start 25: TestMatxUtil.ReduceMax2dColMajor
25/35 Test nv-morpheus#25: TestMatxUtil.ReduceMax2dColMajor .......................... Passed 1.40 sec
Start 26: TestMatxUtil.Cast
26/35 Test nv-morpheus#26: TestMatxUtil.Cast ......................................... Passed 0.16 sec
Start 27: TestMatxUtil.Threshold
27/35 Test nv-morpheus#27: TestMatxUtil.Threshold .................................... Passed 0.16 sec
Start 28: TestMatxUtil.ThresholdByRow
28/35 Test nv-morpheus#28: TestMatxUtil.ThresholdByRow ............................... Passed 0.19 sec
Start 29: TestMultiSlices.Ranges
29/35 Test nv-morpheus#29: TestMultiSlices.Ranges .................................... Passed 1.37 sec
Start 30: TestTensor.UtilsShapeString
30/35 Test nv-morpheus#30: TestTensor.UtilsShapeString ............................... Passed 0.06 sec
Start 31: TestTensor.GetElementStride
31/35 Test nv-morpheus#31: TestTensor.GetElementStride ............................... Passed 0.06 sec
Start 32: TestTensor.AsType
32/35 Test nv-morpheus#32: TestTensor.AsType ......................................... Passed 0.16 sec
Start 33: TestTensor.Create
33/35 Test nv-morpheus#33: TestTensor.Create ......................................... Passed 0.16 sec
Start 34: TestTensor.UtilsValidateShapeAndStride
34/35 Test nv-morpheus#34: TestTensor.UtilsValidateShapeAndStride .................... Passed 0.16 sec
Start 35: TestTypeUtils.DTypeCopy
35/35 Test nv-morpheus#35: TestTypeUtils.DTypeCopy ................................... Passed 0.06 sec
100% tests passed, 0 tests failed out of 35
Total Test time (real) = 19.91 sec
```
Authors:
- Christopher Harris (https://github.com/cwharris)
Approvers:
- David Gardner (https://github.com/dagardner-nv)
URL: nv-morpheus#1095
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.