Holoscan Flow Benchmarking Logging Updates #555
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tbirdso
force-pushed
the
test-benchmarking-log-failures
branch
from
466e7ff
to
157d8ee
Compare
Changes: - Adopts standard `logging` module to enhance granular control and redirection of benchmark script logging - Updates missing log files from warning to error severity Signed-off-by: Tom Birdsong <tbirdsong@nvidia.com>
Refactors code and addresses issue where log file parent directory was not appended to path search for existence, resulting in only the local directory being checked for log files existence. NVBug 4933187 Signed-off-by: Tom Birdsong <tbirdsong@nvidia.com>
tbirdso
force-pushed
the
test-benchmarking-log-failures
branch
from
157d8ee
to
89dfefd
Compare
Changes: - Add `--level` option to control logging verbosity and use `logger` object throughout - Refactor logging so that each output filepath is printed exactly once. Available files are printed with INFO verbosity, missing files are printed with ERROR verbosity. - Exit with error if log files are missing Sample log printout: ``` 2024-11-08 05:23:47,323 INFO: Log file directory: /workspace/holohub/log_directory_20241108-052346 2024-11-08 05:23:47,324 INFO: Log files are available: 2024-11-08 05:23:47,324 ERROR: Log files are missing: /workspace/holohub/log_directory_20241108-052346/logger_greedy_1_1.log 2024-11-08 05:23:47,324 ERROR: Some log files are missing. Please check the log directory. ``` Signed-off-by: Tom Birdsong <tbirdsong@nvidia.com>
Updates `run` script to propagate errors from `run launch` as the `run` script exit code. Allows Holoscan Flow Benchmarking tools to detect when a HoloHub application process exits abnormally. Signed-off-by: Tom Birdsong <tbirdsong@nvidia.com>
Changes: - Adds logging handler to log detailed (DEBUG) output to a "benchmark.log" file in the given run log directory. Adds a stream handler to preserve control over CLI output verbosity with the "--level" CLI argument. - Logs stdout and stderr from an failing process to aid error investigation. Includes thread ID. Signed-off-by: Tom Birdsong <tbirdsong@nvidia.com>
tbirdso
force-pushed
the
test-benchmarking-log-failures
branch
from
89dfefd
to
06e815f
Compare
|
Added quality-of-life updates:
Failed run example (headless SSH): Corresponding Successful run example, with an additional run and GPU utilization enabled for purpose of demonstration: Corresponding Failed run with console logging limited to errors only: These updates aim to improve the |
jjomier
approved these changes
Nov 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes:
benchmark.pyto use standard Python logging moduleLog files may be missing if the underlying
./run launchcommand fails to run the application, such as in the event of a bad app configuration or run environment.