Re-enable support for GetJsonObject by default #916
Conversation
Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me> Contributes to NVIDIA#865 Re-enable the support for GetJsonObject as RAPIDS 24.04 is supporting it by default - Update the unit-test accordingly - Minor change in notes of `InMemoryTableScanExec`
amahussein
added
bug
| @@ -18,7 +18,7 @@ AQEShuffleReadExec,TNEW,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,S,NS,PS,PS,PS | |||
| HashAggregateExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,PS,NS,PS,PS,PS,NS,NS,NS | |||
| ObjectHashAggregateExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,PS,NS,PS,PS,PS,NS,NS,NS | |||
| SortAggregateExec,S,None,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,S,PS,NS,PS,PS,PS,NS,NS,NS | |||
| InMemoryTableScanExec,S,This is disabled by default because there could be complications when using it with AQE. For more details please check https://github.com/NVIDIA/spark-rapids/issues/10603,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,NS,NS,PS,PS,PS,NS,S,S | |||
| InMemoryTableScanExec,S,This is disabled by default because there could be complications when using it with AQE with Spark-3.5.0 and Spark-3.5.1. For more details please check https://github.com/NVIDIA/spark-rapids/issues/10603,Input/Output,S,S,S,S,S,S,S,S,PS,S,S,NS,NS,NS,PS,PS,PS,NS,S,S | |||
There was a problem hiding this comment.
Not related to this PR but if it is disabled by default, then second column should have been "NS" which we see for other Execs such as CollectLimitExec, WindowInPandasExec
There was a problem hiding this comment.
Thanks @nartal1 !
Yes, I don't like it either. @cindyyuanjiang Can you take a look into fixing this while you are making modifications to the script?
This is happening because the plugin disabled that exec in spark-3.5, and Spark-3.5.1
Since we give Precedence to S over NS, then we end up with S
It does not affect the behavior here, but at least we get rid of it showing up everytime the sync-script is executed until Cindy fixes it.
There was a problem hiding this comment.
From spark-rapids repo it looks like this is disabled only in "3.5" and supported in all prior versions and since the script to union all the Spark-versions uses the latest one, this is getting the notes from "3.5" but second column is not updated to "NS".
Signed-off-by: Ahmed Hussein (amahussein) a@ahussein.me
Contributes to #865
Re-enable the support for
GetJsonObjectas RAPIDS 24.04 is supporting it by defaultInMemoryTableScanExec