Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix parsing App IDs inside metrics directory in QualX #1167

Merged
merged 2 commits into from
Jul 5, 2024

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented Jul 3, 2024

Fixes #1163. QualX assumes App IDs start with app* and uses this regex pattern to fetch profiling metrics. However, App IDs can be any alphanumeric string.

This PR fixes the parsing to read all subdirectories and create the list. We are guaranteed that all sub directories within raw_metrics contain individual app metrics so we do not need a regex matching and can safely read all of them.

Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
@parthosa parthosa added bug Something isn't working user_tools Scope the wrapper module running CSP, QualX, and reports (python) labels Jul 3, 2024
@parthosa parthosa self-assigned this Jul 3, 2024
@parthosa parthosa changed the title Fix parsing App IDs inside metrics directory Fix parsing App IDs inside metrics directory in QualX Jul 3, 2024
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Copy link
Collaborator

@amahussein amahussein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Partho!
LGTME

Comment on lines +445 to +447
# search sub directories for App IDs
app_ids = [p.name for p in Path(metrics_dir).iterdir() if p.is_dir()]
if len(app_ids) == 0:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix should do it.
The trick part in this change that we have to rely on directory structure to pick the right folders.
For example, moving fwd, we can't introduce a subdirectory in raw-metrics because it will be mistakenly treated as an Application folder.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amahussein. Yes, If we change the raw metrics folder then all its usages should be updated as well.

@parthosa parthosa merged commit e1e03ff into NVIDIA:dev Jul 5, 2024
14 checks passed
@parthosa parthosa deleted the spark-rapids-tools-1163 branch July 5, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] QualX fails to detect profiling metrics for non-standard App IDs
2 participants