Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-3425] [VL] Fix the issue of incorrectly creating an HDFS path when writing a Parquet file into the HDFS directory #3464

Merged
merged 1 commit into from
Oct 27, 2023

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Oct 20, 2023

What changes were proposed in this pull req

This pull request fix the issue of incorrectly creating an HDFS path when writing a Parquet file into the HDFS directory.

How was this patch tested?

Local jenkins test.

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf changed the title Revert "[GLUTEN-3425] Create not existing HDFS folder when writing HDFS file (#3428)" [GLUTEN-3425] Make the thrift serialized buffer size configuable in velox parquet writer Oct 24, 2023
@github-actions
Copy link

#3425

@JkSelf JkSelf changed the title [GLUTEN-3425] Make the thrift serialized buffer size configuable in velox parquet writer [GLUTEN-3425] [VL] Fix the issue of incorrectly creating an HDFS path when writing a Parquet file into the HDFS directory Oct 25, 2023
@JkSelf
Copy link
Contributor Author

JkSelf commented Oct 26, 2023

@zhouyuan @PHILO-HE Can you help to review again? The jenkins test is here.

Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks!

@zhouyuan zhouyuan merged commit db7f98b into apache:main Oct 27, 2023
15 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3464_time.csv log/native_master_10_26_2023_d3b91cda7_time.csv difference percentage
q1 35.22 36.45 1.234 103.50%
q2 24.97 25.07 0.094 100.38%
q3 39.98 40.23 0.247 100.62%
q4 37.78 36.93 -0.854 97.74%
q5 70.02 71.27 1.251 101.79%
q6 8.48 7.21 -1.277 84.95%
q7 86.45 87.36 0.906 101.05%
q8 86.02 87.57 1.556 101.81%
q9 121.21 120.10 -1.110 99.08%
q10 54.33 51.67 -2.658 95.11%
q11 19.59 20.48 0.888 104.53%
q12 25.40 28.70 3.292 112.96%
q13 49.76 48.75 -1.002 97.99%
q14 17.84 20.14 2.302 112.91%
q15 34.24 33.99 -0.252 99.26%
q16 16.44 16.44 -0.004 99.97%
q17 101.90 101.67 -0.227 99.78%
q18 147.75 148.54 0.785 100.53%
q19 16.94 16.98 0.044 100.26%
q20 31.32 31.88 0.556 101.78%
q21 223.51 223.59 0.078 100.03%
q22 13.30 13.21 -0.086 99.36%
total 1262.46 1268.23 5.764 100.46%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants