Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA][JSON] Respect allowNumericLeadingZeros in from_json #9588

Closed
andygrove opened this issue Oct 31, 2023 · 1 comment · Fixed by #10542
Closed

[FEA][JSON] Respect allowNumericLeadingZeros in from_json #9588

andygrove opened this issue Oct 31, 2023 · 1 comment · Fixed by #10542
Labels
task Work required that improves the product but is not user facing

Comments

@andygrove
Copy link
Contributor

andygrove commented Oct 31, 2023

Is your feature request related to a problem? Please describe.
Our implementation of from_json supports leading zeros for numeric values, but Spark does not support this with the default setting of allowNumericLeadingZeros=false, so we are not compatible.

Describe the solution you'd like
We should respect this setting, or fall back to CPU if the schema asks for numeric values and allowNumericLeadingZeros=false.

We have a test test_from_json_struct that contains a reference to this issue and should be updated once this issue is resolved.

Describe alternatives you've considered
None

Additional context
None

@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify labels Oct 31, 2023
@revans2
Copy link
Collaborator

revans2 commented Oct 31, 2023

I thought we would treat this like other format parsing. We ask CUDF to return the data to us as a string with an indication if it was quoted or not. Then we can parse it ourselves with our own checks for leading zeros or not. Unless we can convince cudf that they want this to be a feature on their parser too.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Oct 31, 2023
@sameerz sameerz added task Work required that improves the product but is not user facing and removed feature request New feature or request labels Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants