Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Audit][FEA] [SPARK-36070][CORE] Log time cost info for writing rows out and committing the task #3959

Closed
nartal1 opened this issue Oct 28, 2021 · 4 comments · Fixed by #4437
Closed

[Audit][FEA] [SPARK-36070][CORE] Log time cost info for writing rows out and committing the task #3959

nartal1 opened this issue Oct 28, 2021 · 4 comments · Fixed by #4437
Assignees
@HaoYang670
Labels
audit_3.3.0 Audit related tasks for 3.3.0 P1 Nice to have for release

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Oct 28, 2021 •
edited
Loading

Is your feature request related to a problem? Please describe.
This issue is just to keep GpuFileFormatWriter.scala in sync with changes in Apache Spark FileFormatWriter.scala. The PR is used to add logging which would help to debug. It doesn't affect functionality in anyway.
PR- apache/spark@f5a63322de

Another trivial change - apache/spark@2fc9c0bfb5 (change logInfo to logWarning)
@nartal1 nartal1 added feature request New feature or request ? - Needs Triage Need team to review and classify audit_3.3.0 Audit related tasks for 3.3.0 labels Oct 28, 2021
@Salonijain27 Salonijain27 added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Nov 2, 2021
@HaoYang670 HaoYang670 self-assigned this Dec 21, 2021
@HaoYang670
Copy link
Collaborator

I'd like to try to fix it.

@HaoYang670
Copy link
Collaborator

Maybe the logging time has been added.
https://github.com/NVIDIA/spark-rapids/blob/branch-22.02/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala#L254

@HaoYang670
Copy link
Collaborator

Another related issue is #2123

@HaoYang670
Copy link
Collaborator

This is another PR that delect the code of apache/spark@f5a6332

apache/spark@3ff8c9f#diff-03e7ff19e93cb270d82aca907ac0a1f87463ba6eb5dce78a407bb169b840a6cbL305-L311

@HaoYang670 HaoYang670 mentioned this issue Dec 29, 2021
Merged
@sameerz sameerz removed the feature request New feature or request label Jan 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@HaoYang670 HaoYang670

Labels
audit_3.3.0 Audit related tasks for 3.3.0 P1 Nice to have for release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change logInfo to logWarning HaoYang670/spark-rapids
4 participants
@sameerz @Salonijain27 @nartal1 @HaoYang670