[WIP] Add tests to getJSONObject for queries involving single quotes
#10476
Conversation
Signed-off-by: Suraj Aralihalli <suraj.ara16@gmail.com>
SurajAralihalli
added
the
cudf_dependency
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuGetJsonObject.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Suraj Aralihalli <suraj.ara16@gmail.com>
Signed-off-by: Suraj Aralihalli <suraj.ara16@gmail.com>
Signed-off-by: Suraj Aralihalli <suraj.ara16@gmail.com>
|
How is this different from #10466? If we are fixing some things in CUDF, that is great, but I think the other PR has duplicated a lot of the work that would be needed here. |
Signed-off-by: Suraj Aralihalli <suraj.ara16@gmail.com>
SurajAralihalli
force-pushed
the
get_json_fix_invalid_queries
branch
from
2ba05ce to
da352e8
Compare
SurajAralihalli
changed the title
getJSONObject for queries involving single quotes and invalid syntaxgetJSONObject for queries involving single quotes
SurajAralihalli
changed the title
getJSONObject for queries involving single quotesgetJSONObject for queries involving single quotes
Thanks for pointing it out! I have limited the scope of this PR to only add tests for the valid queries that include single quotes and currently fail due to the bugs in cudf. I believe it is not covered in the scope of #10466 as it examines invalid queries. |
|
Sounds good thanks for adding these tests. |
| f.get_json_object('jsonStr',"$.A.B").alias('sub_d'), | ||
| f.get_json_object('jsonStr',"$.'A").alias('sub_e') | ||
| ), | ||
| conf={'spark.rapids.sql.expression.GetJsonObject': 'true'}) |
There was a problem hiding this comment.
nit: This config is now true by default so it's safe to remove it here.
thirtiseven
removed
the
cudf_dependency
Depends on PR 15082
Cudf PR 15082 fixes issues with
get_json_objectwhen single quotes are used in the JSONPath query. Additionally, it now throws astd::invalid_argumenterror for invalid queries. The JNI function catches this exception and produces a string column consisting of null values.This PR verifies the output of
get_json_objectbetween the cudf implementation and the standard Spark version.For example in
{ "A.B": 2, "'A": { "B'": 3 } }