ORC-705: Predicate evaluation should take into account writer calendar #588
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java
Outdated
Show resolved
Hide resolved
java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
Outdated
Show resolved
Hide resolved
java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
Outdated
Show resolved
Hide resolved
java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java
Outdated
Show resolved
Hide resolved
|
Thanks again for the review @dongjoon-hyun ! Latest OS upgrade wiped the IDE style settings for some reason which is quite frustrating -- should be back to normal now :) |
dongjoon-hyun
approved these changes
Dec 30, 2020
dongjoon-hyun
pushed a commit
that referenced
this pull request
Dec 30, 2020
#588) ### What changes were proposed in this pull request? RecordReaderImp should pass down the writer calendar info (writerUsedProlepticGregorian) when evaluating predicates to make sure column stats are properly deserialized (affects TimestampStatistics) ### Why are the changes needed? Correct evaluation of predicates with Timestamps ### How was this patch tested? TestRecordReaderImpl.testPredEvalTimestampStatsDiffWriter (cherry picked from commit 60b03ef) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…iter calendar (apache#588) RecordReaderImp should pass down the writer calendar info (writerUsedProlepticGregorian) when evaluating predicates to make sure column stats are properly deserialized (affects TimestampStatistics) Correct evaluation of predicates with Timestamps TestRecordReaderImpl.testPredEvalTimestampStatsDiffWriter (cherry picked from commit 60b03ef) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> Change-Id: I2cb88599149ed05a9bbb91e48b9900e403efce78
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…count writer calendar (apache#588)" This reverts commit 317037e. Change-Id: I4bf42769c1a34ec300ca1af526dbb00f7ef72b10
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…iter calendar (apache#588) RecordReaderImp should pass down the writer calendar info (writerUsedProlepticGregorian) when evaluating predicates to make sure column stats are properly deserialized (affects TimestampStatistics) Correct evaluation of predicates with Timestamps TestRecordReaderImpl.testPredEvalTimestampStatsDiffWriter (cherry picked from commit 60b03ef) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> Change-Id: If28c18265e1439dc90a26af984d6e17adcfff59d
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…into cdw-master * changes: CDPD-45931: ORC-661: Use ChronoLocalDate and day of epoch instead of java's Date for column statistics. CDPD-45931: ORC-398 col stats list map (apache#395) CDPD-45931: ORC-705: Predicate evaluation should take into account writer calendar (apache#588) CDPD-45931: ORC-706: Put back DataReaderProperties default maxDiskRangeChunkLimit CDPD-45931: ORC-685: Add `ReaderImpl.extractFileTail` back (apache#570) CDPD-45931: ORC-676. Add getRawDataSizeFromColIndices back to ReaderImpl (apache#555) CDPD-45931: ORC-671: Add OrcTail.getStripeStatistics back for backward compatiblility CDPD-45931: ORC-669: Reduce breaking changes in ReaderImpl.java (apache#547) Change-Id: If1e18dadec1cb9ad9f08b20d4f031a9b18afed77 CDPD-45931: ORC-621: Fix reader for empty positions list in first row index entry. CDPD-45931: ORC-548: Fix Java InStream to work with BufferChunks that don't start at the stream. CDPD-45931: ORC-520: Fix file merging for column encryption. CDPD-45931: ORC-529: Allow configuration and table properties to control encryption. CDPD-45931: ORC-487: Refactor stripe planning to a new class StripePlanner. CDPD-45931: ORC-523: Update ReaderImpl to work with column encryption. CDPD-45931: ORC-516: Update InStream for column compression. CDPD-45931: ORC-486: Add column entryption to the TreeWriters. CDPD-45931: ORC-484: Update PhysicalWriter to support encryption. CDPD-45931: ORC-509: Fix the KeyProvider to match the current spec. CDPD-45931: ORC-510: Cleanup API for StreamOptions and CompressionCodec.Options CDPD-45931: ORC-485. Add API to write encrypted files. CDPD-45931: ORC-483 Update protobuf for column encryption. CDPD-45931: ORC-421: Separate compression options from the CompressionCodec. CDPD-45931: ORC-251: Extend InStream and OutStream to support encryption.
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…count writer calendar (apache#588)" This reverts commit cfde11e. Change-Id: I8ebe363c7629daaecc0ad68b27e91ded0ad73468
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…iter calendar (apache#588) RecordReaderImp should pass down the writer calendar info (writerUsedProlepticGregorian) when evaluating predicates to make sure column stats are properly deserialized (affects TimestampStatistics) Correct evaluation of predicates with Timestamps TestRecordReaderImpl.testPredEvalTimestampStatsDiffWriter (cherry picked from commit 60b03ef) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> Change-Id: If28c18265e1439dc90a26af984d6e17adcfff59d
zratkai
pushed a commit
to zratkai/orc
that referenced
this pull request
Jun 20, 2024
…iter calendar (apache#588) RecordReaderImp should pass down the writer calendar info (writerUsedProlepticGregorian) when evaluating predicates to make sure column stats are properly deserialized (affects TimestampStatistics) Correct evaluation of predicates with Timestamps TestRecordReaderImpl.testPredEvalTimestampStatsDiffWriter (cherry picked from commit 60b03ef) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> Change-Id: I135d33679fd4f454c2e2294404a9706562e6cdf2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
RecordReaderImp should pass down the writer calendar info (writerUsedProlepticGregorian) when evaluating predicates to make sure column stats are properly deserialized (affects TimestampStatistics)
Why are the changes needed?
Correct evaluation of predicates with Timestamps
How was this patch tested?
TestRecordReaderImpl.testPredEvalTimestampStatsDiffWriter