Improve verification for map columns #14054

brendan-driscoll · 2020-02-03T21:44:47Z

== RELEASE NOTES ==

Verifier Changes
* Add checks for keys, values, and cardinality sum when validating a map column.

linux-foundation-easycla · 2020-02-03T21:44:50Z

The committers are authorized under a signed CLA.

✅ brendan-driscoll (a80129143eb365ef0d5c8a8111dd625d629927a0)

mbasmanova

@brendan-driscoll Thanks for working on this. Looks good % a question.

Let's remove "Initial commit:" from the commit message and PR title.

@caithagoras Would you take a look?

presto-verifier/src/main/java/com/facebook/presto/verifier/checksum/MapColumnValidator.java

caithagoras · 2020-02-05T02:10:17Z

Apologize for the delay. I'll get it reviewed by tomorrow morning.

caithagoras

Take a look at QueryUtil.functionCall to simplify all callers for new FunctionCall

presto-verifier/src/main/java/com/facebook/presto/verifier/checksum/MapColumnValidator.java

caithagoras · 2020-02-05T20:16:04Z

Potentially, if key/values types are of other non-simple types, we'll be able to apply the respective column validator on the keys and values (with some restrictions).
Depends on #14036

caithagoras · 2020-02-05T20:16:29Z

@brendan-driscoll Please squash the 2 commits.

caithagoras · 2020-02-05T20:19:27Z

The PR is missing tests. Please take a look at TestChecksumValidator

caithagoras · 2020-02-05T22:31:38Z

presto-verifier/src/test/java/com/facebook/presto/verifier/checksum/TestChecksumValidator.java

@@ -297,9 +304,9 @@ public void testArray()
        ChecksumResult controlChecksum = new ChecksumResult(
                5,
                ImmutableMap.<String, Object>builder()
-                        .put("int_array_checksum", new SqlVarbinary(new byte[] {0xa}))
+                        .put("int_array_checksum", new SqlVarbinary(new byte[]{0xa}))


Your formatting is off. Please follow 7e and 7f in https://our.internmc.facebook.com/intern/wiki/DataInfra/Presto/Source/
to install Airlift code style, and then format the file with (option + cmd + L).

caithagoras · 2020-02-05T22:35:54Z

presto-verifier/src/test/java/com/facebook/presto/verifier/checksum/TestChecksumValidator.java

+                        .put(MAP_COLUMN, new ColumnMatchResult(false, "control(checksum: 0a, keys_checksum: 0b, values_checksum: 0c, cardinality_sum: 3) " +
+                                "test(checksum: 1a, keys_checksum: 0b, values_checksum: 0c, cardinality_sum: 3)"))


nit: all arguments on the same line, or one argument per line. Same below.

caithagoras · 2020-02-05T22:41:17Z

Looks good % nit.
As mentioned earlier, please squash the 3 commits into 1 using interactive rebase: https://thoughtbot.com/blog/git-interactive-rebase-squash-amend-rewriting-history
Commit title can be improved: "Add additional checks when validating map columns".
Add a release notes section similar to Add sum of cardinality for array column verifier results #13916 (comment)
- "Add checks for keys, values, and cardinality sum when validating a map column."

brendan-driscoll · 2020-02-06T16:06:19Z

== RELEASE NOTES ==

Verifier Changes
* Add checks for keys, values, and cardinality sum when validating a map column.

mbasmanova · 2020-02-06T16:18:35Z

@brendan-driscoll Would you please squash commits?

viczhang861 · 2020-02-06T22:26:02Z

@brendan-driscoll Why is this a fix for #3809 ?

mbasmanova · 2020-02-06T22:37:02Z

@brendan-driscoll Thanks for squashing commits. Please, update the commit to match the guidelines at https://chris.beams.io/posts/git-commit/

For example,

Improve verification for map columns

Add checksums for map key, values and map sizes.

caithagoras · 2020-02-06T22:49:32Z

Please update the commit title and the PR title.

brendan-driscoll · 2020-02-07T15:55:05Z

@brendan-driscoll Why is this a fix for #3809 ?

@viczhang861 That was a typo, should have been #13809

mbasmanova · 2020-02-07T19:42:33Z

@brendan-driscoll There are some merge conflicts. Would you rebase on top of latest master and update the PR?

mbasmanova · 2020-02-11T21:31:21Z

@brendan-driscoll Test failures seem to be related to the changes. Would you take a look?

caithagoras

lgtm % nit
please also fix Travis test

presto-verifier/src/test/java/com/facebook/presto/verifier/checksum/TestChecksumValidator.java

presto-verifier/src/main/java/com/facebook/presto/verifier/checksum/MapColumnValidator.java

Add checksums for map key, values and map sizes.

mbasmanova assigned caithagoras Feb 3, 2020

mbasmanova requested review from caithagoras, mbasmanova and yingsu00 February 3, 2020 21:49

mbasmanova added the verifier label Feb 3, 2020

mbasmanova reviewed Feb 3, 2020

View reviewed changes

presto-verifier/src/main/java/com/facebook/presto/verifier/checksum/MapColumnValidator.java Outdated Show resolved Hide resolved

presto-verifier/src/main/java/com/facebook/presto/verifier/checksum/MapColumnValidator.java Outdated Show resolved Hide resolved

brendan-driscoll changed the title ~~Initial commit: add verification mismatch info for map columns~~ Add verification mismatch info for map columns Feb 4, 2020

caithagoras reviewed Feb 5, 2020

View reviewed changes

caithagoras assigned brendan-driscoll Feb 5, 2020

caithagoras reviewed Feb 5, 2020

View reviewed changes

mbasmanova unassigned brendan-driscoll Feb 6, 2020

brendan-driscoll force-pushed the verify-map-columns branch from 41ab163 to cb58b59 Compare February 6, 2020 16:30

caithagoras approved these changes Feb 6, 2020

View reviewed changes

brendan-driscoll changed the title ~~Add verification mismatch info for map columns~~ Improve verification for map columns Feb 7, 2020

brendan-driscoll force-pushed the verify-map-columns branch from cb58b59 to c364084 Compare February 7, 2020 16:01

brendan-driscoll force-pushed the verify-map-columns branch 2 times, most recently from 2cbe269 to 1022df1 Compare February 10, 2020 16:24

brendan-driscoll force-pushed the verify-map-columns branch from f673412 to a8a9447 Compare February 12, 2020 16:55

caithagoras approved these changes Feb 13, 2020

View reviewed changes

presto-verifier/src/test/java/com/facebook/presto/verifier/checksum/TestChecksumValidator.java Outdated Show resolved Hide resolved

presto-verifier/src/main/java/com/facebook/presto/verifier/checksum/MapColumnValidator.java Outdated Show resolved Hide resolved

Improve verification for map columns

3b6846f

Add checksums for map key, values and map sizes.

brendan-driscoll force-pushed the verify-map-columns branch from 006c1f3 to 3b6846f Compare February 13, 2020 18:39

caithagoras merged commit 6e24976 into prestodb:master Feb 18, 2020

caithagoras mentioned this pull request Mar 5, 2020

Add release notes for 0.233 #14215

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve verification for map columns #14054

Improve verification for map columns #14054

brendan-driscoll commented Feb 3, 2020 •

edited by mbasmanova

Loading

linux-foundation-easycla bot commented Feb 3, 2020 •

edited

Loading

mbasmanova left a comment

caithagoras commented Feb 5, 2020

caithagoras left a comment

caithagoras commented Feb 5, 2020

caithagoras commented Feb 5, 2020

caithagoras commented Feb 5, 2020

caithagoras Feb 5, 2020

caithagoras Feb 5, 2020

caithagoras commented Feb 5, 2020 •

edited

Loading

brendan-driscoll commented Feb 6, 2020 •

edited by mbasmanova

Loading

mbasmanova commented Feb 6, 2020

viczhang861 commented Feb 6, 2020

mbasmanova commented Feb 6, 2020

caithagoras commented Feb 6, 2020

brendan-driscoll commented Feb 7, 2020 •

edited

Loading

mbasmanova commented Feb 7, 2020

mbasmanova commented Feb 11, 2020

caithagoras left a comment

		.put(MAP_COLUMN, new ColumnMatchResult(false, "control(checksum: 0a, keys_checksum: 0b, values_checksum: 0c, cardinality_sum: 3) " +
		"test(checksum: 1a, keys_checksum: 0b, values_checksum: 0c, cardinality_sum: 3)"))

Improve verification for map columns #14054

Improve verification for map columns #14054

Conversation

brendan-driscoll commented Feb 3, 2020 • edited by mbasmanova Loading

linux-foundation-easycla bot commented Feb 3, 2020 • edited Loading

mbasmanova left a comment

Choose a reason for hiding this comment

caithagoras commented Feb 5, 2020

caithagoras left a comment

Choose a reason for hiding this comment

caithagoras commented Feb 5, 2020

caithagoras commented Feb 5, 2020

caithagoras commented Feb 5, 2020

caithagoras Feb 5, 2020

Choose a reason for hiding this comment

caithagoras Feb 5, 2020

Choose a reason for hiding this comment

caithagoras commented Feb 5, 2020 • edited Loading

brendan-driscoll commented Feb 6, 2020 • edited by mbasmanova Loading

mbasmanova commented Feb 6, 2020

viczhang861 commented Feb 6, 2020

mbasmanova commented Feb 6, 2020

caithagoras commented Feb 6, 2020

brendan-driscoll commented Feb 7, 2020 • edited Loading

mbasmanova commented Feb 7, 2020

mbasmanova commented Feb 11, 2020

caithagoras left a comment

Choose a reason for hiding this comment

brendan-driscoll commented Feb 3, 2020 •

edited by mbasmanova

Loading

linux-foundation-easycla bot commented Feb 3, 2020 •

edited

Loading

caithagoras commented Feb 5, 2020 •

edited

Loading

brendan-driscoll commented Feb 6, 2020 •

edited by mbasmanova

Loading

brendan-driscoll commented Feb 7, 2020 •

edited

Loading