[OPPRO-314] Correctly fall back scan operator with unsupported filters #559
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/oap-project/gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
| @@ -94,26 +94,26 @@ case class TransformPreOverrides() extends Rule[SparkPlan] { | |||
| case TransformHint.TRANSFORM_UNSUPPORTED => | |||
| logDebug(s"Columnar Processing for ${plan.getClass} is under row guard.") | |||
| plan match { | |||
| case plan: ShuffledHashJoinExec => | |||
| case shj: ShuffledHashJoinExec => | |||
There was a problem hiding this comment.
Yes, the changes for this part has no impact on functionality, just for making it more readable. Thanks!
PHILO-HE
changed the title
PHILO-HE
changed the title
| // With this setting, row based input will be used in the fallback case of data source v1. | ||
| // TODO: remove this setting if vanilla spark's vectorized input is supported. | ||
| conf.setConfString("spark.sql.parquet.enableVectorizedReader", "false") | ||
|
|
There was a problem hiding this comment.
what happens if we use columnar input? Will it add a columnarToRow, then RowTo ArrowColumnar?
There was a problem hiding this comment.
At least for velox,` we have no support to transform vanilla spark columnar input to velox columnar input. So if we do use vanilla columnar input, an exception will be thrown to interrupt the execution. cc @rui-mo.
BTW, we found setting the configuration here can cause some issues, maybe we have to tell user to set it manually.
This reverts commit 30657aa.
|
can fallback to Spark's columnar scan now? |
Will fix it in another PR. Thanks! |
For now, the config to disable Spark's columnar output is still needed. |
The code for handling filter push down should check whether scan is transformable, considering unsupported filter can be pushed down to scan.
Reproducible issue link: #560.