[VL] add support for reading ORC

[zuochunwei]

What changes were proposed in this pull request?

add support for reading ORC files encoded with RLE v1/v2

the PR oap-velox orc supprt should be merged before this PR

(Please fill in changes proposed in this fix)

(Fixes: #ISSUE-ID)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[zuochunwei]

@JkSelf

[JkSelf]

@zuochunwei Can you also add some orc test in gluten unit test scala side to check whether offload to native and the correctness？

[zuochunwei]

@zuochunwei Can you also add some orc test in gluten unit test scala side to check whether offload to native and the correctness？

I have added query_benchmark test for ORC RLE V1 and V2, it's ok, and another partner has helped me test reading ORC from Hive/HDFS.

what scala test do you want me to provide?

[JkSelf]

We can read an orc file to check whether the scan operator can be offloaded and then verify the correctness to compare with vanilla spark. We can also add the orc file format test in TPC-H/DS later.

[JkSelf]

@zuochunwei You may need to change the velox repo and branch here if you changed the velox code.

[zuochunwei]

@zuochunwei You may need to change the velox repo and branch here if you changed the velox code.

I proposed a corresponding PR for oap-project/velox, please refer to PR

[zuochunwei]

We can read an orc file to check whether the scan operator can be offloaded and then verify the correctness to compare with vanilla spark. We can also add the orc file format test in TPC-H/DS later.

We have tested read orc file through the TableScan operator, and it's OK

[zhejiangxiaomai]

Here is a example to use your local branch to test gluten CI. @zuochunwei
https://github.com/oap-project/gluten/pull/1503/files

[zuochunwei]

Here is a example to use your local branch to test gluten CI. @zuochunwei https://github.com/oap-project/gluten/pull/1503/files

OK, thank you!

[JkSelf]

@zuochunwei I tested orc scan in gluten with following unit test. It can work. Can you add this unit test in TestOperator.scala in this PR? And we can add the orc file format test in TPC-H/DS in the following PRs later.

test("orc scan") {
val df = spark.read.format("orc").load("./cpp/velox/benchmarks/data/bm_lineitem/orc/lineitem.orc")
df.createOrReplaceTempView("lineitem_orc")
runQueryAndCompare(
"select l_orderkey from lineitem_orc") { df => {
assert(getExecutedPlan(df).count(plan => {
plan.isInstanceOf[BatchScanExecTransformer]}) == 1)
}}
}

[zuochunwei]

@zuochunwei I tested orc scan in gluten with following unit test. It can work. Can you add this unit test in TestOperator.scala in this PR? And we can add the orc file format test in TPC-H/DS in the following PRs later.

test("orc scan") { val df = spark.read.format("orc").load("./cpp/velox/benchmarks/data/bm_lineitem/orc/lineitem.orc") df.createOrReplaceTempView("lineitem_orc") runQueryAndCompare( "select l_orderkey from lineitem_orc") { df => { assert(getExecutedPlan(df).count(plan => { plan.isInstanceOf[BatchScanExecTransformer]}) == 1) }} }

done

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[zhejiangxiaomai]

run mvn clean install -Pspark-3.2 -PskipTests -Pbackends-velox can see code style error.

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[zhejiangxiaomai]

Thanks.