Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamp reader support for Parquet file format #4681

Open
rui-mo opened this issue Apr 20, 2023 · 4 comments
Open

Timestamp reader support for Parquet file format #4681

rui-mo opened this issue Apr 20, 2023 · 4 comments
Labels
enhancement New feature or request parquet

Comments

@rui-mo
Copy link
Collaborator

rui-mo commented Apr 20, 2023

Description

Hi,
We would like to add the support for timestamp in Velox Parquet reader. It was noticed there were two kinds of timestamp types used in Parquet files, the INT96 type and the INT64 type (for TIMESTAMP_MILLIS or TIMESTAMP_MICROS). Which one is Velox's plan on timestamp reader for Parquet?

Although INT96 is deprecated in Parquet (see PARQUET-323), Spark uses it by default for timestamp during Parquet writing. Therefore, we firstly drafted this PR(#4680) to support a timestamp reader to read from INT96-typed timestamps, and would like to hear your opinions and suggestions. Thanks!

@rui-mo rui-mo added the enhancement New feature or request label Apr 20, 2023
@rui-mo
Copy link
Collaborator Author

rui-mo commented Apr 20, 2023

@mbasmanova @majetideepak Would you like to take a look on this issue?

@mbasmanova
Copy link
Contributor

CC: @oerling @Yuhta @yingsu00

@Yuhta Yuhta assigned Yuhta and unassigned Yuhta Apr 20, 2023
@majetideepak
Copy link
Collaborator

@rui-mo we definitely need to support INT96 timestamp type for legacy Parquet writers and Parquet files. I will review the PR. Thanks for raising this issue.

@rui-mo
Copy link
Collaborator Author

rui-mo commented Apr 21, 2023

@majetideepak Thanks for the feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request parquet
Projects
None yet
Development

No branches or pull requests

4 participants