ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr… #442

lcspinter · 2019-10-24T08:15:42Z

…eady acid

omalley

You really need to add a test case as well.

omalley · 2019-10-29T23:14:54Z

java/core/src/java/org/apache/orc/impl/SchemaEvolution.java

@@ -87,7 +87,8 @@ public SchemaEvolution(TypeDescription fileSchema,
    this.fileSchema = fileSchema;
    this.isAcid = checkAcidSchema(fileSchema);
    this.includeAcidColumns = options.getIncludeAcidColumns();
-    this.readerColumnOffset = isAcid ? acidEventFieldNames.size() : 0;
+    this.readerColumnOffset = (isAcid && readerSchema != null && checkAcidSchema(readerSchema)) || !isAcid


I'd suggest keeping the checkAcidSchema(readerSchema) rather than calling it multiple times. This expression needs to be simplified.

omalley · 2019-10-29T23:15:47Z

java/core/src/java/org/apache/orc/impl/SchemaEvolution.java

@@ -87,7 +87,8 @@ public SchemaEvolution(TypeDescription fileSchema,
    this.fileSchema = fileSchema;
    this.isAcid = checkAcidSchema(fileSchema);
    this.includeAcidColumns = options.getIncludeAcidColumns();
-    this.readerColumnOffset = isAcid ? acidEventFieldNames.size() : 0;
+    this.readerColumnOffset = (isAcid && readerSchema != null && checkAcidSchema(readerSchema)) || !isAcid
+        ? 0 : acidEventFieldNames.size();
    if (readerSchema != null) {
      if (isAcid) {


You probably want to cut out here with if (isAcid && !isReaderAcid) rather than change addEventSchema.

…eady acid

lcspinter · 2019-10-30T12:32:19Z

@omalley I changed the implementation and added new test.

omalley

Thank you!

omalley

Thanks!

…eady acid Fixes #442 Signed-off-by: Owen O'Malley <omalley@apache.org>

omalley · 2019-11-05T18:52:13Z

I've committed this.

…eady acid Fixes apache#442 Signed-off-by: Owen O'Malley <omalley@apache.org>

…hema is already acid Fixes apache#442 Signed-off-by: Owen O'Malley <omalley@apache.org> (cherry picked from commit 2db32a8) Change-Id: Icfc548d19b0d0cc62b3e0e90ea164c791ece0a94

…eady acid Fixes apache#442 Signed-off-by: Owen O'Malley <omalley@apache.org>

Bumps [org.apache.commons:commons-csv](https://github.com/apache/commons-csv) from 1.11.0 to 1.12.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/apache/commons-csv/blob/master/RELEASE-NOTES.txt">org.apache.commons:commons-csv's changelog</a>.</em></p> <blockquote> <p>Apache Commons CSV Version 1.12.0 Release Notes</p> <p>This document contains the release notes for the 1.12.0 version of Apache Commons CSV. Commons CSV reads and writes files in variations of the Comma Separated Value (CSV) format.</p> <p>Commons CSV requires at least Java 8.</p> <p>The Apache Commons CSV library provides a simple interface for reading and writing CSV files of various types.</p> <p>Feature and bug fix release (Java 8 or above)</p> <p>Changes in this version include:</p> <h2>New Features</h2> <ul> <li>CSV-270: Add CSVException that extends IOException thrown on invalid input instead of IOException. Thanks to Thomas Kamps, Gary Gregory.</li> </ul> <h2>Fixed Bugs</h2> <ul> <li> <pre><code> Fix PMD issues for port to PMD 7.1.0. Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Fix some Javadoc links [#442](apache/commons-csv#442). Thanks to Dávid Szigecsán, Gary Gregory. </code></pre> </li> <li> <pre><code> Extract duplicated code into a method [#444](apache/commons-csv#444). Thanks to Dávid Szigecsán. </code></pre> </li> <li> <pre><code> Migrate CSVFormat#print(File, Charset) to NIO [#445](apache/commons-csv#445). Thanks to Dávid Szigecsán. </code></pre> </li> <li> <pre><code> Fix documentation for CSVFormat private constructor [#466](apache/commons-csv#466). Thanks to Dávid Szigecsán. </code></pre> </li> <li>CSV-294: CSVFormat does not support explicit " as escape char. Thanks to Joern Huxhorn, Gary Gregory.</li> <li>CSV-150: Escaping is not disableable. Thanks to dota17, Gary Gregory, Jörn Huxhorn.</li> <li> <pre><code> Fix Javadoc warnings on Java 23. Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Improve parser performance by up to 20%, YMMV. Thanks to Gary Gregory. </code></pre> </li> </ul> <h2>Changes</h2> <ul> <li> <pre><code> Bump commons-codec:commons-codec from 1.16.1 to 1.17.1 [#422](apache/commons-csv#422), [#449](apache/commons-csv#449). Thanks to Dependabot. </code></pre> </li> <li> <pre><code> Bump org.apache.commons:commons-parent from 69 to 75 [#435](apache/commons-csv#435), [#452](apache/commons-csv#452), [#465](apache/commons-csv#465), [#468](apache/commons-csv#468), [#475](apache/commons-csv#475). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump org.codehaus.mojo:taglist-maven-plugin from 3.0.0 to 3.1.0 [#441](apache/commons-csv#441). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump org.apache.commons:commons-lang3 from 3.14.0 to 3.17.0 [#450](apache/commons-csv#450), [#459](apache/commons-csv#459), [#470](apache/commons-csv#470). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump org.hamcrest:hamcrest from 2.2 to 3.0 [#455](apache/commons-csv#455). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump commons-io:commons-io from 2.16.1 to 2.17.0 [#476](apache/commons-csv#476). Thanks to Gary Gregory, Dependabot. </code></pre> </li> </ul> <p>Historical list of changes: <a href="https://commons.apache.org/proper/commons-csv/changes-report.html">https://commons.apache.org/proper/commons-csv/changes-report.html</a></p> <p>For complete information on Apache Commons CSV, including instructions on how to submit bug reports, patches, or suggestions for improvement, see the Apache Commons CSV website:</p> <p><a href="https://commons.apache.org/proper/commons-csv/">https://commons.apache.org/proper/commons-csv/</a></p> <p>Download page: <a href="https://commons.apache.org/proper/commons-csv/download_csv.cgi">https://commons.apache.org/proper/commons-csv/download_csv.cgi</a></p> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/apache/commons-csv/commit/67f0d6b30465d817a341b2e9cd31660a646e980c"><code>67f0d6b</code></a> Prepare for the next release candidate</li> <li><a href="https://github.com/apache/commons-csv/commit/89eacd90ef235444a79d16d695fec3ff9eb008d4"><code>89eacd9</code></a> Prepare for the next release candidate</li> <li><a href="https://github.com/apache/commons-csv/commit/f4e3269626140684b58901db9b9cd233ac960c9f"><code>f4e3269</code></a> Merge branch 'master' of <a href="https://gitbox.apache.org/repos/asf/commons-csv">https://gitbox.apache.org/repos/asf/commons-csv</a></li> <li><a href="https://github.com/apache/commons-csv/commit/342547b911dfe919787d9f53fb330f7d926ec6c3"><code>342547b</code></a> Prepare for the next release candidate</li> <li><a href="https://github.com/apache/commons-csv/commit/113147f1e7fbd84fe3948c52fda0ee2f6c6a8ea2"><code>113147f</code></a> Add dependency-review.yml to GitHub CI</li> <li><a href="https://github.com/apache/commons-csv/commit/86ce50b8078761e22cd947be247d5b66dda0d6b9"><code>86ce50b</code></a> Merge pull request <a href="https://redirect.github.com/apache/commons-csv/issues/478">#478</a> from apache/dependabot/github_actions/github/codeql-a...</li> <li><a href="https://github.com/apache/commons-csv/commit/5b2c26eedf942fc70080010af375b35ff3ddbbde"><code>5b2c26e</code></a> Bump github/codeql-action from 3.26.6 to 3.26.8</li> <li><a href="https://github.com/apache/commons-csv/commit/4f4b9cf2516762cd766368759c2c122f19f0caa5"><code>4f4b9cf</code></a> Sort members</li> <li><a href="https://github.com/apache/commons-csv/commit/6a11b896aa8601deceb119c3ee1fd32e2efe276c"><code>6a11b89</code></a> Enable Checkstyle for test sources and fix issues</li> <li><a href="https://github.com/apache/commons-csv/commit/fce94ea666f09f4c162cd1b67b86bd6e271a9558"><code>fce94ea</code></a> Fix header for Checkstyle</li> <li>Additional commits viewable in <a href="https://github.com/apache/commons-csv/compare/rel/commons-csv-1.11.0...rel/commons-csv-1.12.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.commons:commons-csv&package-manager=maven&previous-version=1.11.0&new-version=1.12.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Closes #2043 from dependabot[bot]/dependabot/maven/java/org.apache.commons-commons-csv-1.12.0. Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Bumps [org.apache.commons:commons-csv](https://github.com/apache/commons-csv) from 1.11.0 to 1.12.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/apache/commons-csv/blob/master/RELEASE-NOTES.txt">org.apache.commons:commons-csv's changelog</a>.</em></p> <blockquote> <p>Apache Commons CSV Version 1.12.0 Release Notes</p> <p>This document contains the release notes for the 1.12.0 version of Apache Commons CSV. Commons CSV reads and writes files in variations of the Comma Separated Value (CSV) format.</p> <p>Commons CSV requires at least Java 8.</p> <p>The Apache Commons CSV library provides a simple interface for reading and writing CSV files of various types.</p> <p>Feature and bug fix release (Java 8 or above)</p> <p>Changes in this version include:</p> <h2>New Features</h2> <ul> <li>CSV-270: Add CSVException that extends IOException thrown on invalid input instead of IOException. Thanks to Thomas Kamps, Gary Gregory.</li> </ul> <h2>Fixed Bugs</h2> <ul> <li> <pre><code> Fix PMD issues for port to PMD 7.1.0. Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Fix some Javadoc links [#442](apache/commons-csv#442). Thanks to Dávid Szigecsán, Gary Gregory. </code></pre> </li> <li> <pre><code> Extract duplicated code into a method [#444](apache/commons-csv#444). Thanks to Dávid Szigecsán. </code></pre> </li> <li> <pre><code> Migrate CSVFormat#print(File, Charset) to NIO [#445](apache/commons-csv#445). Thanks to Dávid Szigecsán. </code></pre> </li> <li> <pre><code> Fix documentation for CSVFormat private constructor [#466](apache/commons-csv#466). Thanks to Dávid Szigecsán. </code></pre> </li> <li>CSV-294: CSVFormat does not support explicit " as escape char. Thanks to Joern Huxhorn, Gary Gregory.</li> <li>CSV-150: Escaping is not disableable. Thanks to dota17, Gary Gregory, Jörn Huxhorn.</li> <li> <pre><code> Fix Javadoc warnings on Java 23. Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Improve parser performance by up to 20%, YMMV. Thanks to Gary Gregory. </code></pre> </li> </ul> <h2>Changes</h2> <ul> <li> <pre><code> Bump commons-codec:commons-codec from 1.16.1 to 1.17.1 [#422](apache/commons-csv#422), [#449](apache/commons-csv#449). Thanks to Dependabot. </code></pre> </li> <li> <pre><code> Bump org.apache.commons:commons-parent from 69 to 75 [#435](apache/commons-csv#435), [#452](apache/commons-csv#452), [#465](apache/commons-csv#465), [#468](apache/commons-csv#468), [#475](apache/commons-csv#475). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump org.codehaus.mojo:taglist-maven-plugin from 3.0.0 to 3.1.0 [#441](apache/commons-csv#441). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump org.apache.commons:commons-lang3 from 3.14.0 to 3.17.0 [#450](apache/commons-csv#450), [#459](apache/commons-csv#459), [#470](apache/commons-csv#470). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump org.hamcrest:hamcrest from 2.2 to 3.0 [#455](apache/commons-csv#455). Thanks to Gary Gregory. </code></pre> </li> <li> <pre><code> Bump commons-io:commons-io from 2.16.1 to 2.17.0 [#476](apache/commons-csv#476). Thanks to Gary Gregory, Dependabot. </code></pre> </li> </ul> <p>Historical list of changes: <a href="https://commons.apache.org/proper/commons-csv/changes-report.html">https://commons.apache.org/proper/commons-csv/changes-report.html</a></p> <p>For complete information on Apache Commons CSV, including instructions on how to submit bug reports, patches, or suggestions for improvement, see the Apache Commons CSV website:</p> <p><a href="https://commons.apache.org/proper/commons-csv/">https://commons.apache.org/proper/commons-csv/</a></p> <p>Download page: <a href="https://commons.apache.org/proper/commons-csv/download_csv.cgi">https://commons.apache.org/proper/commons-csv/download_csv.cgi</a></p> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/apache/commons-csv/commit/67f0d6b30465d817a341b2e9cd31660a646e980c"><code>67f0d6b</code></a> Prepare for the next release candidate</li> <li><a href="https://github.com/apache/commons-csv/commit/89eacd90ef235444a79d16d695fec3ff9eb008d4"><code>89eacd9</code></a> Prepare for the next release candidate</li> <li><a href="https://github.com/apache/commons-csv/commit/f4e3269626140684b58901db9b9cd233ac960c9f"><code>f4e3269</code></a> Merge branch 'master' of <a href="https://gitbox.apache.org/repos/asf/commons-csv">https://gitbox.apache.org/repos/asf/commons-csv</a></li> <li><a href="https://github.com/apache/commons-csv/commit/342547b911dfe919787d9f53fb330f7d926ec6c3"><code>342547b</code></a> Prepare for the next release candidate</li> <li><a href="https://github.com/apache/commons-csv/commit/113147f1e7fbd84fe3948c52fda0ee2f6c6a8ea2"><code>113147f</code></a> Add dependency-review.yml to GitHub CI</li> <li><a href="https://github.com/apache/commons-csv/commit/86ce50b8078761e22cd947be247d5b66dda0d6b9"><code>86ce50b</code></a> Merge pull request <a href="https://redirect.github.com/apache/commons-csv/issues/478">#478</a> from apache/dependabot/github_actions/github/codeql-a...</li> <li><a href="https://github.com/apache/commons-csv/commit/5b2c26eedf942fc70080010af375b35ff3ddbbde"><code>5b2c26e</code></a> Bump github/codeql-action from 3.26.6 to 3.26.8</li> <li><a href="https://github.com/apache/commons-csv/commit/4f4b9cf2516762cd766368759c2c122f19f0caa5"><code>4f4b9cf</code></a> Sort members</li> <li><a href="https://github.com/apache/commons-csv/commit/6a11b896aa8601deceb119c3ee1fd32e2efe276c"><code>6a11b89</code></a> Enable Checkstyle for test sources and fix issues</li> <li><a href="https://github.com/apache/commons-csv/commit/fce94ea666f09f4c162cd1b67b86bd6e271a9558"><code>fce94ea</code></a> Fix header for Checkstyle</li> <li>Additional commits viewable in <a href="https://github.com/apache/commons-csv/compare/rel/commons-csv-1.11.0...rel/commons-csv-1.12.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.commons:commons-csv&package-manager=maven&previous-version=1.11.0&new-version=1.12.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Closes #2043 from dependabot[bot]/dependabot/maven/java/org.apache.commons-commons-csv-1.12.0. Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 0cf506b) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

omalley requested changes Oct 29, 2019

View reviewed changes

lcspinter force-pushed the ORC-562 branch from 2d0f52a to 76afd23 Compare October 30, 2019 09:57

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr…

d294db3

…eady acid

lcspinter force-pushed the ORC-562 branch from 76afd23 to d294db3 Compare October 30, 2019 11:47

omalley approved these changes Nov 5, 2019

View reviewed changes

omalley pushed a commit that referenced this pull request Nov 5, 2019

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr…

39674f9

…eady acid Fixes #442 Signed-off-by: Owen O'Malley <omalley@apache.org>

omalley closed this Nov 5, 2019

abstractdog pushed a commit to abstractdog/orc that referenced this pull request Nov 6, 2019

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr…

2db32a8

…eady acid Fixes apache#442 Signed-off-by: Owen O'Malley <omalley@apache.org>

stiga-huang pushed a commit to stiga-huang/orc that referenced this pull request Jan 18, 2020

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr…

854ce20

…eady acid Fixes apache#442 Signed-off-by: Owen O'Malley <omalley@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr… #442

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr… #442

lcspinter commented Oct 24, 2019

omalley left a comment

omalley Oct 29, 2019

omalley Oct 29, 2019

lcspinter commented Oct 30, 2019

omalley left a comment

omalley left a comment

omalley commented Nov 5, 2019

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr… #442

ORC-562:Don't wrap readerSchema in acidSchema, if readerSchema is alr… #442

Conversation

lcspinter commented Oct 24, 2019

omalley left a comment

Choose a reason for hiding this comment

omalley Oct 29, 2019

Choose a reason for hiding this comment

omalley Oct 29, 2019

Choose a reason for hiding this comment

lcspinter commented Oct 30, 2019

omalley left a comment

Choose a reason for hiding this comment

omalley left a comment

Choose a reason for hiding this comment

omalley commented Nov 5, 2019