Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge 23.11 #3

Merged
merged 11 commits into from
Aug 22, 2023
Merged

Commits on Aug 4, 2023

  1. Updating tests to force .pyi files to be committed into the repo (nv-…

    …morpheus#1111)
    
    This PR changes the stub generation to be committed into the repo instead of as part of the build. This will allow CI steps that need the PYI files to not require a GPU as well.
    
    This works by always copying the PYI files back to the source tree so they will show up as diffs. Any diffs during the test phase will be reported as an error.
    
    Closes nv-morpheus#766
    
    Authors:
      - Michael Demoret (https://github.com/mdemoret-nv)
      - David Gardner (https://github.com/dagardner-nv)
    
    Approvers:
      - Christopher Harris (https://github.com/cwharris)
      - David Gardner (https://github.com/dagardner-nv)
    
    URL: nv-morpheus#1111
    mdemoret-nv authored Aug 4, 2023
    Configuration menu
    Copy the full SHA
    662cd40 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. Add a --viz_direction flag to CLI (nv-morpheus#1119)

    * Add a `--viz_direction` flag to CLI
    * Cherry-pick'd [9532bfe](nv-morpheus@9532bfe) from @mdemoret-nv 
    
    fixes nv-morpheus#643
    
    Authors:
      - David Gardner (https://github.com/dagardner-nv)
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    Approvers:
      - Christopher Harris (https://github.com/cwharris)
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    URL: nv-morpheus#1119
    dagardner-nv authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    43dfcc5 View commit details
    Browse the repository at this point in the history
  2. Slimmed down CI runners and published artifact urls (nv-morpheus#1112)

    * Publish artifact urls in job summaries fixes nv-morpheus#1105
    * Remove redundant apt installs for packages that exist in the conda env fixes nv-morpheus#1106
    * Checks now performs a build first. This allows pylint to accurately warn for undefined members. fixes nv-morpheus#956
    * Custom pylint checker to validate correct importing of `morpheus._lib` fixes nv-morpheus#727
    
    Authors:
      - David Gardner (https://github.com/dagardner-nv)
    
    Approvers:
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    URL: nv-morpheus#1112
    dagardner-nv authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    70cceef View commit details
    Browse the repository at this point in the history
  3. Improve gitutils.py by using the Github CLI when available (nv-morp…

    …heus#1088)
    
    The `gitutils.py` script often fails when you are working in a PR, or if there has been a commit to your target branch since you created your current branch.
    
    This improves the utility by using the Github CLI to query the PR information when possible and get the true target branch for the current PR.
    
    If the Github CLI is not available, or the branch isnt in a PR, it falls back to the previous functionality.
    
    Authors:
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    Approvers:
      - David Gardner (https://github.com/dagardner-nv)
    
    URL: nv-morpheus#1088
    mdemoret-nv authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    aba421e View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. Fix intermittent failures in test_dfencoder_distributed_e2e test (nv-…

    …morpheus#1113)
    
    * Call the `manual_seed` method from within the subprocess, this ensures the subprocess runs deterministically.
    * Add a sleep to a busy-loop in `morpheus/models/dfencoder/multiprocessing.py`
    * Misc pylint fixes
    
    fixes nv-morpheus#1021
    
    Authors:
      - David Gardner (https://github.com/dagardner-nv)
    
    Approvers:
      - Christopher Harris (https://github.com/cwharris)
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    URL: nv-morpheus#1113
    dagardner-nv authored Aug 22, 2023
    Configuration menu
    Copy the full SHA
    0853e82 View commit details
    Browse the repository at this point in the history
  2. Fix issue where DFPFileToDataFrameStage logs messages about S3 even w…

    …hen S3 is not in use (nv-morpheus#1120)
    
    * `DFPFileToDataFrameStage` is no longer an S3 specific stage, but some of the log messages and the `unique-name` still refer to S3.
    
    Fixes nv-morpheus#451
    
    Authors:
      - David Gardner (https://github.com/dagardner-nv)
    
    Approvers:
      - Christopher Harris (https://github.com/cwharris)
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    URL: nv-morpheus#1120
    dagardner-nv authored Aug 22, 2023
    Configuration menu
    Copy the full SHA
    c92e212 View commit details
    Browse the repository at this point in the history
  3. Fix Tests broken by Test Reorganization (nv-morpheus#1118)

    The test reorganization resulted in a couple of problems. First, not all tests were being run due to a mistake I made in handling cmake function arguments. Second, the `test_cuda` test was producing a segfault inside of triton client while initializing curl, but only in debug builds. The fix was to the segfault was not obvious, but apparently only occured when the test utilities were compiled as a separate library. Interestingly, the other tests were not affected. Perhaps there is some global state that needs to be managed that is not managed in the simple `test_cuda` test, which only calls a Matx function (also important, the Matx function result is not inspected, so the test itself is not comprehensive, as Matx does not produce an error if the kernel is not available for the given architecture).
    
    Depends on nv-morpheus#1110
    
    Authors:
      - Christopher Harris (https://github.com/cwharris)
    
    Approvers:
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    URL: nv-morpheus#1118
    cwharris authored Aug 22, 2023
    Configuration menu
    Copy the full SHA
    7cbbb7e View commit details
    Browse the repository at this point in the history
  4. update devcontainer base to 23.10 (nv-morpheus#1116)

    Update the devcontainer base to 23.10 (this has no relation to cudf versioning).
    
    While there are other improvements, the main reason I'm upgrading is due to a newer version of git, which allows inline public key signing of commits - something we need to be able to commit changes while inside the container without having to manage separate container-specific signing keys.
    
    Example of signing key usage (from host, inherited by devcontainer):
    ```
    ssh-keygen -t ed25519 -C "example@nvidia.com"
    git config --global user.signingkey "key::$(cat ~/.ssh/id_ed25519.pub)"
    git config --global commit.gpgsign true
    git config --global gpg.format ssh
    mkdir -p ~/.config/git
    git config --global gpg.ssh.allowedSignersFile ~/.config/git/allowed_signers
    awk 'BEGIN {FS=" "; OFS=" "} {print $3, $1, $2}' <(ssh-add -L) > ~/.config/git/allowed_signers
    ```
    
    Authors:
      - Christopher Harris (https://github.com/cwharris)
    
    Approvers:
      - Michael Demoret (https://github.com/mdemoret-nv)
    
    URL: nv-morpheus#1116
    cwharris authored Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5af1cdb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c285380 View commit details
    Browse the repository at this point in the history
  6. ???

    dagardner-nv committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5b193a8 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c447089 View commit details
    Browse the repository at this point in the history