From 209a843b7ae46c0f397328dc7a633ec4ea46a0d2 Mon Sep 17 00:00:00 2001 From: Alan Cai Date: Mon, 15 Aug 2022 17:21:25 -0700 Subject: [PATCH 1/2] Add EvalMode to evaluation test assertion --- docs/partiql-tests-schema-proposal.md | 95 ++++++++++++++++--- .../fail/eval/query/select/select.ion | 3 +- partiql-tests-data/partiql-tests-schema.isl | 23 ++++- .../success/eval-equiv/path.ion | 2 + .../success/eval/query/select/select.ion | 2 + .../validator/PartiQLTestDataValidator.kt | 39 +++++++- 6 files changed, 144 insertions(+), 20 deletions(-) diff --git a/docs/partiql-tests-schema-proposal.md b/docs/partiql-tests-schema-proposal.md index d8c9e13..4447eb3 100644 --- a/docs/partiql-tests-schema-proposal.md +++ b/docs/partiql-tests-schema-proposal.md @@ -14,7 +14,7 @@ As of this proposal, we want to test these categories: --- The following is an abstraction to describe current and future tests we will have in the `partiql-tests` suite -``` +```ion { name: , statement: , @@ -36,7 +36,7 @@ Tests whether a given PartiQL statement is syntactically valid. For now, compose - PartiQL statement (string) - assert (struct or list of structs) with a syntax assertion -``` +```ion // syntax 'success' test with one assertion { name: , @@ -58,7 +58,7 @@ Tests whether a given PartiQL statement is syntactically valid. For now, compose The `assert` field could also be a list of structs if more test assertions are added in the future. -``` +```ion // syntax 'success' test with multiple assertions { ... @@ -100,7 +100,7 @@ between parsing and evaluation. It's up to the implementation to decide at what For now, composed of the same properties as the `syntax` `fail` tests. The only difference is the `assert`'s error (i.e. `StaticAnalysisFail`). -``` +```ion { name: , statement: , @@ -121,22 +121,23 @@ Tests whether a given PartiQL statement evaluates to the expected result. For no - [optional] input evaluation environment, `env` - struct - defaults to using environments specified in file (i.e. `envs`). If `envs` is unspecified, defaults to an empty environment with no bindings -- [optional] evaluation `options` other than the defaults - struct - - e.g. typing mode - strict vs permissive (permissive may be most useful) - `assert` - struct or list of structs - `result` key maps to symbol - `EvaluatorSuccess` if evaluation succeeds for statement - `EvaluatorFail` if evaluation fails for statement + - `evalMode` - evaluation mode to run the tests (symbol or list of symbols) + - `EvalModeCoerce` - dynamic type mismatch returns `MISSING` + - `EvalModeError` - dynamic type mismatch errors - expected `output` of evaluation (only for `EvaluatorSuccess` `result`s) -``` +```ion // eval 'success' test { name: , statement: , env: , // optional - options: , // optional assert: { + evalMode: | >, result: EvaluationSuccess, output: }, @@ -147,8 +148,8 @@ Tests whether a given PartiQL statement evaluates to the expected result. For no name: , statement: , env: , // optional - options: , // optional assert: { + evalMode: | >, result: EvaluationFail } } @@ -158,7 +159,7 @@ For ease of writing evaluation tests, it’s necessary to provide a way to speci outside a given test. Specifying environments available for a given file: -``` +```ion envs::{ 'table1': [{a:1}, {a:2}, {a:3}], 'table2': ... @@ -189,7 +190,7 @@ few approaches we could take to model PartiQL data: Of the above options, we've decided to go with the annotation approach. Values of these additional types will be denoted using a `$` annotation. This will be used for the output result and environments. -``` +```ion // bag -- list annotated with $bag $bag::[1, 2, 3] @@ -205,12 +206,76 @@ $time::'02:30:59' --- +#### PartiQL Evaluation Modes +The PartiQL specification defines two dynamic type mismatching evaluation modes (i.e. when a PartiQL statement is run +without schema). As defined in the PartiQL specification, these modes are: +- Permissive mode -- dynamic typing mismatches are neglected and PartiQL returns `MISSING` +- Type checking mode -- dynamic type mismatches result in evaluation errors + +The naming of these modes can be somewhat confusing especially "type checking mode", which is sometimes referred to as +`STRICT` mode in the specification and Kotlin reference implementation. For the purposes of this document and the +conformance tests, we will refer to permissive mode as `EvalModeCoerce` and type checking mode as `EvalModeError`. +These names can be changed in the future once we improve the terminology in the specification. + +```ion +// Test case using `EvalModeCoerce` +{ + name: "coerce eval mode tuple navigation missing attribute dot notation", + statement: "{'a':1, 'b':2}.noSuchAttribute", + assert: { + evalMode: EvalModeCoerce, + result: EvaluationSuccess, + output: $missing::null + } +} + +// Test case using `EvalModeError` +{ + name: "error eval mode tuple navigation missing attribute dot notation", + statement: "{'a':1, 'b':2}.noSuchAttribute", + assert: { + evalMode: EvalModeError, + result: EvaluationFail + } +} + +// Test case using both eval modes in assertions +{ + name: "tuple navigation missing attribute dot notation", + statement: "{'a':1, 'b':2}.noSuchAttribute", + assert: [ + { + evalMode: EvalModeError, + result: EvaluationFail + }, + { + evalMode: EvalModeCoerce, + result: EvaluationSuccess, + output: $missing::null + }, + ] +} + +// Test case using both eval modes in assertions with same result +{ + name: "tuple navigation for attribute dot notation", + statement: "{'a':1, 'b':2}.a", + assert: { + evalMode: [EvalModeError, EvalModeCoerce], + result: EvaluationSuccess, + output: 1 + } +} +``` + +--- + ### Equivalence The PartiQL specification mentions some PartiQL statements that could be rewritten using a different PartiQL syntax (e.g. wildcard expressions). A common use case could be to assert that such PartiQL statements evaluate to the same result or have the same plan. Users can specify an equivalence class as follows. -``` +```ion equiv_class::{ id: , // identifier that can be referred to in tests (e.g. evaluation) statements: > // list of equivalent PartiQL statements as strings @@ -220,7 +285,7 @@ equiv_class::{ Evaluation tests can check that an equivalence class defined in the file/namespace have statements that evaluate to the same result by referencing the equivalence class' symbol identifier in the `statement` field. -``` +```ion // evaluation equivalence test { name: , @@ -234,7 +299,7 @@ same result by referencing the equivalence class' symbol identifier in the `stat ``` As a simple example, the following would be how to write an evaluation equivalence test: -``` +```ion // equivalence class definition equiv_class::{ id: ten, @@ -263,7 +328,7 @@ The concept of namespacing tests can help categorize groups of tests and can red be used by the test runner to prepend additional text to a test name. E.g. namespace of "literals" can be prepended to test names of "int" and "null" to get "literals - int" and "literals - null" -``` +```ion // namespacing/grouping (using a symbol annotation) ::[ { diff --git a/partiql-tests-data/fail/eval/query/select/select.ion b/partiql-tests-data/fail/eval/query/select/select.ion index e6c7174..2026c8b 100644 --- a/partiql-tests-data/fail/eval/query/select/select.ion +++ b/partiql-tests-data/fail/eval/query/select/select.ion @@ -2,6 +2,7 @@ name: "select star from non existent binding", statement: "SELECT * FROM non_existent_binding", assert: { - result: EvaluationFail // example fails in type-checking mode TBD on default evaluation mode (https://github.com/partiql/partiql-tests/issues/25) + evalMode: EvalModeError, + result: EvaluationFail } } diff --git a/partiql-tests-data/partiql-tests-schema.isl b/partiql-tests-data/partiql-tests-schema.isl index 97d2207..83e7ed9 100644 --- a/partiql-tests-data/partiql-tests-schema.isl +++ b/partiql-tests-data/partiql-tests-schema.isl @@ -48,7 +48,6 @@ type::{ occurs: required }, env: { type: struct, occurs: optional }, - options: { type: struct, occurs: optional }, assert: { type: { one_of: [ @@ -75,6 +74,22 @@ type::{ } } +type::{ + name: EvaluationMode, + type: symbol, + valid_values: [EvalModeError, EvalModeCoerce] +} + +type::{ + name: EvaluationModeSymbolOrList, + type: { + one_of: [ + EvaluationMode, + { type: list, element: EvaluationMode } + ] + } +} + type::{ name: SyntaxSuccessAssertion, type: struct, @@ -107,7 +122,8 @@ type::{ type: struct, fields: { result: { type: symbol, valid_values: [EvaluationSuccess], occurs: required }, - output: { occurs: required } + output: { occurs: required }, + evalMode: { type: EvaluationModeSymbolOrList, occurs: required } }, content: closed } @@ -116,7 +132,8 @@ type::{ name: EvaluationFailAssertion, type: struct, fields: { - result: { type: symbol, valid_values: [EvaluationFail], occurs: required } + result: { type: symbol, valid_values: [EvaluationFail], occurs: required }, + evalMode: { type: EvaluationModeSymbolOrList, occurs: required } }, content: closed } diff --git a/partiql-tests-data/success/eval-equiv/path.ion b/partiql-tests-data/success/eval-equiv/path.ion index 5bd2e7f..68b011c 100644 --- a/partiql-tests-data/success/eval-equiv/path.ion +++ b/partiql-tests-data/success/eval-equiv/path.ion @@ -26,6 +26,7 @@ equiv_class::{ name: "equiv wildcard steps collection", statement: wildcard_steps_collection, assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess, output: $bag::[1, 2, 3] } @@ -35,6 +36,7 @@ equiv_class::{ name: "equiv wildcard steps struct", statement: wildcard_steps_struct, assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess, output: $bag::[1, 2] } diff --git a/partiql-tests-data/success/eval/query/select/select.ion b/partiql-tests-data/success/eval/query/select/select.ion index 599e6ea..643734a 100644 --- a/partiql-tests-data/success/eval/query/select/select.ion +++ b/partiql-tests-data/success/eval/query/select/select.ion @@ -9,6 +9,7 @@ envs::{ name: "SELECT path", statement: "SELECT r.v FROM sensors AS s, s.readings AS r", assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess, output: $bag::[ { @@ -39,6 +40,7 @@ envs::{ ] }, assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess, output: $bag::[ { diff --git a/partiql-tests-validator/src/test/kotlin/org/partiql/tests/validator/PartiQLTestDataValidator.kt b/partiql-tests-validator/src/test/kotlin/org/partiql/tests/validator/PartiQLTestDataValidator.kt index 52fa6f9..2743f49 100644 --- a/partiql-tests-validator/src/test/kotlin/org/partiql/tests/validator/PartiQLTestDataValidator.kt +++ b/partiql-tests-validator/src/test/kotlin/org/partiql/tests/validator/PartiQLTestDataValidator.kt @@ -306,6 +306,7 @@ class PartiQLTestDataValidator { name: "some name", statement: "some statement", assert: { + evalMode: EvalModeCoerce, result: EvaluationSuccess, output: some_output } @@ -323,6 +324,7 @@ class PartiQLTestDataValidator { name: "some name", statement: "some statement", assert: { + evalMode: EvalModeError, result: EvaluationFail } } @@ -356,6 +358,7 @@ class PartiQLTestDataValidator { name: "some name", statement: some_equivalence_class, assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess, output: some_output, } @@ -373,8 +376,8 @@ class PartiQLTestDataValidator { name: "some name", statement: "some statement", env: { some_key: some_value }, - options: { some_option_key: some_option_value }, assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess, output: some_output } @@ -589,6 +592,7 @@ class PartiQLTestDataValidator { name: "some name", statement: "some statement", assert: { + evalMode: [EvalModeCoerce, EvalModeError], result: EvaluationSuccess } } @@ -597,6 +601,39 @@ class PartiQLTestDataValidator { assertViolationsOccurred(dataInIon) } + @Test + fun testNoEvalModeInEvaluationSuccessSchema() { + val testData = + """ + { + name: "some name", + statement: "some statement", + assert: { + result: EvaluationSuccess, + output: 'some_output' + } + } + """ + val dataInIon = ion.loader.load(testData) + assertViolationsOccurred(dataInIon) + } + + @Test + fun testNoEvalModeInEvaluationFailSchema() { + val testData = + """ + { + name: "some name", + statement: "some statement", + assert: { + result: EvaluationFail, + } + } + """ + val dataInIon = ion.loader.load(testData) + assertViolationsOccurred(dataInIon) + } + @Test fun testNoOutputInEvaluationSuccessSchemaWithEquivalenceClass() { val testData = From ac367fb1a8fbdb5ae319327b6b5041fc8ef95af5 Mon Sep 17 00:00:00 2001 From: Alan Cai Date: Tue, 16 Aug 2022 17:26:51 -0700 Subject: [PATCH 2/2] Add tracking issue for updating spec terminology --- docs/partiql-tests-schema-proposal.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/partiql-tests-schema-proposal.md b/docs/partiql-tests-schema-proposal.md index 4447eb3..d4ee0d1 100644 --- a/docs/partiql-tests-schema-proposal.md +++ b/docs/partiql-tests-schema-proposal.md @@ -215,7 +215,8 @@ without schema). As defined in the PartiQL specification, these modes are: The naming of these modes can be somewhat confusing especially "type checking mode", which is sometimes referred to as `STRICT` mode in the specification and Kotlin reference implementation. For the purposes of this document and the conformance tests, we will refer to permissive mode as `EvalModeCoerce` and type checking mode as `EvalModeError`. -These names can be changed in the future once we improve the terminology in the specification. +These names can be changed in the future once we improve the terminology in the specification (see +[partiql-docs#24](https://github.com/partiql/partiql-docs/issues/24)). ```ion // Test case using `EvalModeCoerce`