Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Specify schema when calling readJSON in GpuJsonToStructs #10326

Closed
wants to merge 1 commit into from
Closed

WIP: Specify schema when calling readJSON in GpuJsonToStructs #10326

wants to merge 1 commit into from

Conversation

andygrove
Copy link
Contributor

@andygrove andygrove commented Jan 30, 2024 •
edited
Loading

Draft PR for discussion.

Related to #8558

Status: prototype code, exploring how we can specify the schema rather than have cuDF infer it in GpuJsontoStructs.

Benefits:

  • Performance improvement because we only read the columns we need
  • Makes code more consistent with GpuJsonScan and reads primitives as strings and then we cast to the expected type in the plugin for compatibility with Spark

@andygrove andygrove self-assigned this Jan 30, 2024
@andygrove andygrove mentioned this pull request Feb 9, 2024
Closed
@revans2 revans2 mentioned this pull request Feb 28, 2024
Merged
@andygrove andygrove closed this Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@andygrove andygrove

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@andygrove