Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA][Audit] - [SPARK-37750][SQL] ANSI mode: optionally return null result if element not exists in array/map #4668

Closed
nartal1 opened this issue Jan 31, 2022 · 1 comment · Fixed by #4839
Closed

[FEA][Audit] - [SPARK-37750][SQL] ANSI mode: optionally return null result if element not exists in array/map #4668

nartal1 opened this issue Jan 31, 2022 · 1 comment · Fixed by #4839
Assignees
@amahussein
Labels
audit_3.3.0 Audit related tasks for 3.3.0 bug Something isn't working P0 Must have for release

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Jan 31, 2022 •
edited
Loading

Is your feature request related to a problem? Please describe.
This PR adds a config which controls whether it throws exceptions or returns null results if array index is out of bound or map key does not exist for arrays and maps respectively. Have to check if we support ANSI mode for these types and if we want to match the same behavior as CPU.
PR: apache/spark@08fd5013ad
Follow on PR to change config name: apache/spark@0b9f120c99
@nartal1 nartal1 added feature request New feature or request ? - Needs Triage Need team to review and classify audit_3.3.0 Audit related tasks for 3.3.0 labels Jan 31, 2022
@sameerz sameerz added bug Something isn't working P0 Must have for release and removed feature request New feature or request ? - Needs Triage Need team to review and classify labels Feb 1, 2022
@amahussein
Copy link
Collaborator

I am working on #4564 . I thought of adding new config as part of my fix, but I decided to keep those two separate.
It will be great to merge #4564 before this one.

This was referenced Feb 4, 2022
Closed
Merged
@amahussein amahussein self-assigned this Feb 11, 2022
This was referenced Feb 17, 2022
Closed
Merged
@nartal1 nartal1 mentioned this issue Apr 5, 2022
Open
50 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amahussein amahussein

Labels
audit_3.3.0 Audit related tasks for 3.3.0 bug Something isn't working P0 Must have for release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Optionally return null if element not exists map/array amahussein/spark-rapids
3 participants
@sameerz @amahussein @nartal1