Add support for reading orc column statistics #6588

wypb · 2023-09-15T09:08:55Z

Currently, the dwrf module only supports reading the column statistics of dwrf format, and does not support reading the column statistics of orc format. When I use velox to read the orc table, the following exception occurs:

presto:tpch_velox> create table region_orc with(format = 'ORC') as select * from tpch.sf10.region;
CREATE TABLE: 5 rows

Query 20230804_032253_00003_tubug, FINISHED, 1 node
Splits: 22 total, 22 done (100.00%)
0:35 [5 rows, 0B] [0 rows/s, 0B/s]

presto:tpch_velox> select * from region_orc;

Query 20230804_043524_00023_m8qw7, FAILED, 1 node
Splits: 2 total, 0 done (0.00%)
0:40 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20230804_043524_00023_m8qw7 failed:  Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: vector

with this pr, we support read orc column statistics, than we can read orc data through velox:

presto:tpch_velox> select * from region_orc;
 regionkey |    name     |                                                       comment
-----------+-------------+---------------------------------------------------------------------------------------------------------------------
         0 | AFRICA      | lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are according to
         1 | AMERICA     | hs use ironic, even requests. s
         2 | ASIA        | ges. thinly even pinto beans ca
         3 | EUROPE      | ly final courts cajole furiously final excuse
         4 | MIDDLE EAST | uickly special accounts cajole carefully blithely close requests. carefully final asymptotes haggle furiousl
(5 rows)

Query 20230804_044016_00026_m8qw7, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0:14 [5 rows, 603B] [0 rows/s, 44B/s]

presto:tpch_velox>

CC: @Yuhta

netlify · 2023-09-15T09:08:59Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`48f1ee0`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/666a560f236a4b00092a640b

Yuhta · 2023-12-15T19:23:46Z

velox/dwio/dwrf/reader/BinaryStreamReader.cpp

@@ -97,9 +97,9 @@ std::unique_ptr<BinaryStripeStreams> BinaryStreamReader::next() {
      stripeReaderBase_, columnSelector_, stripeIndex_++);
 }

-std::unordered_map<uint32_t, proto::ColumnStatistics>
+std::unordered_map<uint32_t, ColumnStatisticsWrapper>


We have some problem here, the wrapper is just a reference but here we are returning actual values. If ORC does not need it, can we keep the signature and make it DWRF only (with a check)?

wypb · 2023-12-18T07:39:48Z

velox/dwio/dwrf/reader/BinaryStreamReader.cpp

@@ -111,7 +111,8 @@ BinaryStreamReader::getStatistics() const {
        "Corrupted file detected, Footer stats are missing, but stripes are present");
    for (auto node = 0; node < typesSize; node++) {
      if (columnSelector_.shouldReadNode(node)) {
-        stats[node] = proto::ColumnStatistics();
+        const proto::ColumnStatistics cs = proto::ColumnStatistics();


@Yuhta Thank you for your review, It should be that temporary local variables are used here, I will modify it.

wypb · 2023-12-18T08:39:17Z

Hi @Yuhta, Thank you for your review, I have modified the problematic code and the code has been rebased. You can check this diff for the relevant modified code.

Yuhta · 2023-12-18T19:09:00Z

velox/dwio/dwrf/reader/BinaryStreamReader.cpp

+        auto cs =
+            google::protobuf::Arena::CreateMessage<proto::ColumnStatistics>(
+                stripeReaderBase_.getReader().arena());
+        stats.emplace(node, ColumnStatisticsWrapper(cs));


We still have the same problem, I think for this function you should keep it as it is, and create another one returning proto::orc::ColumnStatistics if it is required by ORC as well. Wrappers can only be created on call sites.

@Yuhta I looked at the code again, ORC does not use this method. I'll keep the previous code logic. Thank you for testing this.

Yuhta · 2023-12-19T16:46:58Z

velox/dwio/dwrf/common/FileMetadata.h

@@ -436,12 +766,18 @@ class FooterWrapper : public ProtoWrapperBase {
    return dwrfPtr()->statistics();
  }

-  const ::facebook::velox::dwrf::proto::ColumnStatistics& statistics(
+  const ::facebook::velox::dwrf::proto::ColumnStatistics& statisticsByIndex(


A better name is dwrfStatistics

Fixed, thanks.

wypb · 2024-01-25T13:49:10Z

Hi @Yuhta Do you have any other comment for this PR? thank you.

wypb · 2024-02-26T02:26:06Z

Hi @Yuhta PTAL, Thank you.

wypb · 2024-06-13T01:24:04Z

HI @kevinwilfong I moved the implementation of options() and context() to ColumnStatisticsBase.h file to reduce duplication of code. I also synchronized the latest code of the main branch to make CI green. please help me review it again when you have a chance. Thanks!

facebook-github-bot · 2024-06-13T18:07:51Z

@kevinwilfong has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-06-14T06:22:27Z

@kevinwilfong merged this pull request in 0c0a973.

conbench-facebook · 2024-06-14T06:46:59Z

Conbench analyzed the 1 benchmark run on commit 0c0a9737.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 15, 2023

wypb closed this Sep 15, 2023

wypb deleted the orc_read branch September 15, 2023 09:10

wypb restored the orc_read branch September 15, 2023 09:11

wypb reopened this Sep 15, 2023

wypb force-pushed the orc_read branch from 4b61daf to f6fcca6 Compare September 18, 2023 06:51

wypb force-pushed the orc_read branch 3 times, most recently from 8b689ff to 70fa5e9 Compare November 30, 2023 07:16

wypb force-pushed the orc_read branch from acd2d62 to 66e76b7 Compare December 12, 2023 07:09

Yuhta reviewed Dec 15, 2023

View reviewed changes

wypb force-pushed the orc_read branch from 1f6daed to 1668f08 Compare December 18, 2023 07:24

wypb commented Dec 18, 2023

View reviewed changes

wypb force-pushed the orc_read branch 2 times, most recently from 7dd28a8 to 9c15412 Compare December 18, 2023 08:28

Yuhta reviewed Dec 18, 2023

View reviewed changes

wypb force-pushed the orc_read branch 2 times, most recently from 2af7412 to c5c5986 Compare December 19, 2023 01:33

Yuhta reviewed Dec 19, 2023

View reviewed changes

wypb closed this Dec 20, 2023

wypb reopened this Dec 20, 2023

wypb force-pushed the orc_read branch 2 times, most recently from 0d21b06 to 4b3960e Compare December 20, 2023 03:19

wypb force-pushed the orc_read branch 2 times, most recently from 6b3e52d to 5deef28 Compare February 26, 2024 06:26

wypb mentioned this pull request May 23, 2024

[native] Fix protocol::FileFormat::ORC mapping to use dwio::common::FileFormat::ORC prestodb/presto#22726

Merged

6 tasks

wypb force-pushed the orc_read branch 5 times, most recently from 0fbf1b2 to d886334 Compare June 13, 2024 01:20

wypb force-pushed the orc_read branch 14 times, most recently from 5ec1137 to cfc7d0c Compare June 13, 2024 02:09

Add support for reading orc column statistics

48f1ee0

wypb force-pushed the orc_read branch from cfc7d0c to 48f1ee0 Compare June 13, 2024 02:14

facebook-github-bot closed this in 0c0a973 Jun 14, 2024

facebook-github-bot added the Merged label Jun 14, 2024

wypb deleted the orc_read branch June 14, 2024 10:52

wypb mentioned this pull request Jun 20, 2024

[native] Add support for ORC reader prestodb/presto#23037

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for reading orc column statistics #6588

Add support for reading orc column statistics #6588

wypb commented Sep 15, 2023 •

edited

Loading

netlify bot commented Sep 15, 2023 •

edited

Loading

Yuhta Dec 15, 2023

wypb Dec 18, 2023

wypb commented Dec 18, 2023

Yuhta Dec 18, 2023

wypb Dec 19, 2023

Yuhta Dec 19, 2023

wypb Dec 20, 2023

wypb commented Jan 25, 2024

wypb commented Feb 26, 2024

wypb commented Jun 13, 2024 •

edited

Loading

facebook-github-bot commented Jun 13, 2024

facebook-github-bot commented Jun 14, 2024

conbench-facebook bot commented Jun 14, 2024

Add support for reading orc column statistics #6588

Add support for reading orc column statistics #6588

Conversation

wypb commented Sep 15, 2023 • edited Loading

netlify bot commented Sep 15, 2023 • edited Loading

✅ Deploy Preview for meta-velox canceled.

Yuhta Dec 15, 2023

Choose a reason for hiding this comment

wypb Dec 18, 2023

Choose a reason for hiding this comment

wypb commented Dec 18, 2023

Yuhta Dec 18, 2023

Choose a reason for hiding this comment

wypb Dec 19, 2023

Choose a reason for hiding this comment

Yuhta Dec 19, 2023

Choose a reason for hiding this comment

wypb Dec 20, 2023

Choose a reason for hiding this comment

wypb commented Jan 25, 2024

wypb commented Feb 26, 2024

wypb commented Jun 13, 2024 • edited Loading

facebook-github-bot commented Jun 13, 2024

facebook-github-bot commented Jun 14, 2024

conbench-facebook bot commented Jun 14, 2024

wypb commented Sep 15, 2023 •

edited

Loading

netlify bot commented Sep 15, 2023 •

edited

Loading

wypb commented Jun 13, 2024 •

edited

Loading