Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] JsonToStructs does not match parsing invalid structs on Spark 3.4.1 and below #10588

Open
revans2 opened this issue Mar 13, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Mar 13, 2024

Describe the bug
I think that https://issues.apache.org/jira/browse/SPARK-44079 did this, but I am not 100% sure it could also be related to https://issues.apache.org/jira/browse/SPARK-44940

Essentially if we try to parse a struct that has integers that overflow 3.4.2 and beyond we get back nulls for each field, which matches the plugin.

{"data": {"B": 99999999999999999999, "A": -9999999999999999999}}

Parsed as data struct<a long, b long> results in

{null, null}

But before 3.4.2 we get a single null out for the entire struct

null

Ideally we would make it match whatever it is we do. Note that this is similar to #10573

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Mar 13, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants