Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] CSV Timestamp parseing is broken for TS < 1902 and TS > 2038 #122

Closed
revans2 opened this issue Jun 8, 2020 · 0 comments · Fixed by #870
Closed

[BUG] CSV Timestamp parseing is broken for TS < 1902 and TS > 2038 #122

revans2 opened this issue Jun 8, 2020 · 0 comments · Fixed by #870
Labels
bug Something isn't working P2 Not required for release SQL part of the SQL/Dataframe plugin
Milestone
Sep 28 - Oct 9

Comments

@revans2
Copy link
Collaborator

revans2 commented Jun 8, 2020

Describe the bug

Timestamps are disabled by default for CSV because of the option to have time zone info in the time stamp which cudf does not support. If we enable them they only work for dates after 1902 and before the end of 2038. These look like overflow/underflow issues.

If you try to parse a timestamp column in CSV as a date it is even worse. It looks like it tries to read in the time information and parse it for days and just ends up producing really large numbers.

Steps/Code to reproduce bug
New tests are being pushed to the integration tests that reproduce this.
@revans2 revans2 added bug Something isn't working SQL part of the SQL/Dataframe plugin labels Jun 8, 2020
@sameerz sameerz added ? - Needs Triage Need team to review and classify P2 Not required for release and removed ? - Needs Triage Need team to review and classify labels Aug 18, 2020
This was referenced Sep 28, 2020
Closed
Merged
@revans2 revans2 added this to the Sep 28 - Oct 9 milestone Sep 28, 2020
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to 40baeb4 (NVIDIA#122)
13ab566 
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working P2 Not required for release SQL part of the SQL/Dataframe plugin
Projects
None yet
Milestone
Sep 28 - Oct 9
Development

Successfully merging a pull request may close this issue.

Cleaned up tests and documentation for csv timestamp parsing revans2/spark-rapids
2 participants
@revans2 @sameerz