[FEA]: Avoid running stubgen in CI

[dagardner-nv]

Is this a new feature, an improvement, or a change to existing functionality?

Change

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

Running stubgen in CI requires the Nvidia drivers to be installed in the CI image. This unfortunately locks us to a specific driver version, and ops would like to accelerate adopting new versions.

Describe your ideal solution

Include the stubs as part of the source code and add a CI check to make sure they are up to date

• This would shift generating the stubs to the developer (can still be part of the build. The dev just needs to commit them)
• The test in CI could happen in the test stage which already has a GPU
• The test would be as simple as running the stubgen code and seeing if there are any diffs. Similar to what we do for clang-format

Describe any alternatives you have considered

1. See if the cuda-toolkit driver stubs work the same as cudatoolkit
   If we can load the stubs but not run anything, that would work for us. The current stubs throw an error on load.
   This is a long shot but requires the least amount of work

2. Generate the stubs in a pre-commit hook

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[mdemoret-nv]

Steps:

• Remove .pyi files from .gitignore
• Always copy .pyi back to source tree even if inplace is off
• Change test stage in CI to perform a full build
• Add a step before tests to verify no git diffs (i.e. the stubs are up to date)
• Break up CI scripts into smaller components so they can be timed independently (i.e. step for conda env, build, run tests)