Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix key error and cross-join error during qualx evaluate #1298

Merged
merged 2 commits into from
Aug 19, 2024

Conversation

leewyang
Copy link
Collaborator

This PR fixes some bugs with the evaluate command in the internal CLI:

  • key error when all stages are unsupported.
  • unexpected cross-join in per-app results under specific conditions (appName and scaleFactor are all the same, but description is different).

I have confirmed that the evaluate results match the results from before (except they fix the cross-join case).

Changes

  1. set default columns in new/empty DataFrames to avoid key error during subsequent joins.
  2. use description field in all joins during evaluate.

Test

Following CMDs have been tested:

Internal Usage:

python qualx_main.py evaluate

Signed-off-by: Lee Yang <leey@nvidia.com>
@leewyang leewyang self-assigned this Aug 19, 2024
@leewyang leewyang added the user_tools Scope the wrapper module running CSP, QualX, and reports (python) label Aug 19, 2024
Signed-off-by: Lee Yang <leey@nvidia.com>
@parthosa parthosa added the bug Something isn't working label Aug 19, 2024
@leewyang leewyang changed the title Fix evaluate bugs Fix key error and cross-join error during qualx evaluate Aug 19, 2024
Copy link
Collaborator

@parthosa parthosa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @leewyang. LGTME.

results = pd.DataFrame()
summary = pd.DataFrame()
results = pd.DataFrame(
columns=[
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the python tool, we use a config file to store constants like these. It can be loaded using the class YAMLPropertiesContainer. In future, we could think of having a similar config file for QualX.

Example,

summaryReport:
fileName: rapids_4_spark_qualification_output.csv
columns:
- Vendor
- Driver Host
- Cluster Id
- Cluster Name
- App Name

@leewyang leewyang merged commit 52e3577 into NVIDIA:dev Aug 19, 2024
14 checks passed
@leewyang leewyang deleted the qualx_evaluate branch August 19, 2024 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working user_tools Scope the wrapper module running CSP, QualX, and reports (python)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants