Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support to write metadata files in parquet #5105

Merged
merged 50 commits into from
Apr 2, 2024
Merged

Added support to write metadata files in parquet #5105

merged 50 commits into from
Apr 2, 2024

Conversation

malhotrashivam
Copy link
Contributor

@malhotrashivam malhotrashivam commented Feb 1, 2024 •
edited
Loading

Closes #5039, #958

Also,

  • Added support to do partitioned parquet writing.
  • Offset index information will be read from parquet files on demand and not while reading column chunks.

@malhotrashivam malhotrashivam self-assigned this Feb 1, 2024
@malhotrashivam malhotrashivam added feature request New feature or request parquet Related to the Parquet integration NoDocumentationNeeded NoReleaseNotesNeeded No release notes are needed. labels Feb 1, 2024
@malhotrashivam malhotrashivam mentioned this pull request Feb 1, 2024
Merged
@malhotrashivam malhotrashivam requested a review from rcaudy February 1, 2024 19:09
@malhotrashivam malhotrashivam changed the title [WIP] Added support to write metadata files in parquet Added support to write metadata files in parquet Feb 13, 2024
rcaudy
rcaudy reviewed Feb 14, 2024
View reviewed changes
Copy link
Member

@rcaudy rcaudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered some questions

extensions/parquet/base/src/main/java/io/deephaven/parquet/base/MetadataFileWriter.java Outdated Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java Outdated Show resolved Hide resolved
lbooker42
lbooker42 reviewed Feb 21, 2024
View reviewed changes
Copy link
Contributor

@lbooker42 lbooker42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, a few comments about Javadoc parameters but this is easy to follow.

...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java Outdated Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java Outdated Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java Show resolved Hide resolved
lbooker42
lbooker42 previously approved these changes Feb 22, 2024
View reviewed changes
Copy link
Contributor

@lbooker42 lbooker42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

chipkent
chipkent reviewed Mar 22, 2024
View reviewed changes
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
chipkent
chipkent reviewed Mar 22, 2024
View reviewed changes
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
chipkent
chipkent previously approved these changes Mar 22, 2024
View reviewed changes
Copy link
Member

@chipkent chipkent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python LGTM

@malhotrashivam malhotrashivam mentioned this pull request Mar 25, 2024
Closed
rcaudy
rcaudy reviewed Mar 26, 2024
View reviewed changes
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetInstructions.java Outdated Show resolved Hide resolved
...ns/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetMetadataFileWriterImpl.java Outdated Show resolved Hide resolved
...ns/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetMetadataFileWriterImpl.java Outdated Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTableWriter.java Outdated Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java Outdated Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java Show resolved Hide resolved
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetTools.java Outdated Show resolved Hide resolved
...nsions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java Show resolved Hide resolved
py/server/deephaven/parquet.py Show resolved Hide resolved
rcaudy
rcaudy previously approved these changes Mar 27, 2024
View reviewed changes
chipkent
chipkent reviewed Mar 27, 2024
View reviewed changes
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
py/server/deephaven/parquet.py Show resolved Hide resolved
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
py/server/deephaven/parquet.py Outdated Show resolved Hide resolved
chipkent
chipkent previously approved these changes Mar 28, 2024
View reviewed changes
@malhotrashivam malhotrashivam merged commit a43948f into deephaven:main Apr 2, 2024
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Apr 2, 2024
@deephaven-internal
Copy link
Contributor

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#180

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@lbooker42 lbooker42 lbooker42 left review comments

@rcaudy rcaudy rcaudy approved these changes

@chipkent chipkent chipkent left review comments

@jmao-denver jmao-denver Awaiting requested review from jmao-denver jmao-denver is a code owner

Assignees

@malhotrashivam malhotrashivam

Labels
DocumentationNeeded feature request New feature or request parquet Related to the Parquet integration ReleaseNotesNeeded Release notes are needed
Projects
None yet
Milestone
1. March 2024
Development

Successfully merging this pull request may close these issues.

Support writing Parquet _metadata and _common_metadata files
7 participants
@malhotrashivam @deephaven-internal @lbooker42 @jmao-denver @chipkent @rcaudy @pete-petey