Revamp failure testing with query assertions #20509

findepi · 2024-01-30T16:16:04Z

Before the introduction of query assertions (the QueryAssert class),
assertQueryFails was used to verify that a query fails with expected
error message AND the failure exception type is TrinoException.

With query assertions, the typical pattern for testing query failures
became:

assertThatThrownBy(() -> query(...))
    ....

This unfortunately does not verify that the exception type is the
TrinoException. The existing alternative:

assertTrinoExceptionThrownBy(() -> query(...))
    ...

is rarely used, perhaps because it's longer and the
established shorter pattern is followed.

Both above methods share the disadvantage of basing on the QueryAssert
construction side-effect of evaluating the query eagerly.

This commit revamps QueryAssert to make failures first class citizen
from QueryAssert perspective, on par with query results. Expected
failures are verified to be TrinoException type. The result is more
fluent API, as both success and failures assertions are available on the
same query assert object, and thus can be auto-completed by an IDE.

From code updating perspective, most of the code updates fall into two
categories:

failure testing. This functionality is available from query assert via
.failure() accessor. Code was updated with regular expression
- search for: (assertThatThrownBy|assertTrinoExceptionThrownBy)$\($ -> (assertions\.)?query\(
- replace with: assertThat($2query(
- add .failure() to access assertions on the failure exception.
  (use nonTrinoExceptionFailure() for cases where query incorrectly
  fails without a proper TrinoException)
detailed inspection of MaterializedResult objects. This lower-level
functionality is available from query assert via .result()
accessor.

findepi · 2024-01-30T17:46:53Z

cc @trinodb/maintainers

findepi · 2024-01-30T17:47:02Z

~~currently based on #20506~~

wendigo · 2024-01-30T19:40:51Z

core/trino-main/src/test/java/io/trino/sql/query/TestAggregation.java

@@ -38,22 +37,22 @@ public void teardown()
    @Test
    public void testQuantifiedComparison()
    {
-        assertThatThrownBy(() -> assertions.query("SELECT v > ALL (VALUES 1) FROM (VALUES (1, 1), (1, 2)) t(k, v) GROUP BY k"))
-                .hasMessageContaining("must be an aggregate expression or appear in GROUP BY clause");
+        assertThat(assertions.query("SELECT v > ALL (VALUES 1) FROM (VALUES (1, 1), (1, 2)) t(k, v) GROUP BY k"))


I like the new API :)

wendigo

@findepi Can we extract changes to the QueryAssertions in the last commit to the separate commit as you are mixing mechanical (regex) changes and manual ones in a single commit?

wendigo · 2024-01-30T19:44:32Z

There are related failures

findepi · 2024-01-30T21:34:26Z

Can we extract changes to the QueryAssertions in the last commit to the separate commit as you are mixing mechanical (regex) changes and manual ones in a single commit?

The changes are tightly related, I don't see a way to split them.

`ensureOrdering` was always false, so effectively unused.

Previously, the `QueryAssert.query` could represent original SQL query, or a description (in case of column projections). This commit separates those duties. Original query is not available in case of projections because it would not produce the current (projected) result.

Explained

wendigo

LGTM % failures

findepi · 2024-01-31T08:05:09Z

The new API requires that failures are a TrinoException (and has an scape hatch for cases where they are not).
When migrating from to the new API, i opted to expected TrinoException everywhere and this is the cause of all the test failures noticed so far. In so many tests we were testing for a failure, but the actual exception is wrong.

No plan to fix these in this PR, but they will now be easy to spot (any usages of nonTrinoExceptionFailure()).

Update places that depended on `query(...)` query assert constructor executing the query. These should use `execute`, `assertUpdate`, etc.

losipiuk

looks nice

wendigo · 2024-01-31T11:33:25Z

BigQuery related:

TestBigQueryAvroConnectorTest>BaseBigQueryConnectorTest.testDateYearOfEraPredicate:607 Expected TrinoException or wrapper, but got: io.trino.testing.QueryFailedException io.trino.testing.QueryFailedException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: request failed: Query error: Could not cast literal "-1996-09-14" to type DATE at [1:372]
Error:    TestBigQueryAvroConnectorTest>BaseBigQueryConnectorTest.testMissingWildcardTable:799 Expected TrinoException or wrapper, but got: io.trino.testing.QueryFailedException io.trino.testing.QueryFailedException: sep-bq-cicd:test.test_missing_wildcard_table_* does not match any table.
Error:    TestBigQueryAvroConnectorTest>BaseBigQueryConnectorTest.testNativeQuerySelectUnsupportedType:938 Expected TrinoException or wrapper, but got: io.trino.testing.QueryFailedException io.trino.testing.QueryFailedException: Unsupported type: BIGNUMERIC

Before the introduction of query assertions (the `QueryAssert` class), `assertQueryFails` was used to verify that a query fails with expected error message AND the failure exception type is `TrinoException`. With query assertions, the typical pattern for testing query failures became: assertThatThrownBy(() -> query(...)) .... This unfortunately does not verify that the exception type is the `TrinoException`. The existing alternative: assertTrinoExceptionThrownBy(() -> query(...)) ... is rarely used, perhaps because it's longer and the established shorter pattern is followed. Both above methods share the disadvantage of basing on the `QueryAssert` construction side-effect of evaluating the query eagerly. This commit revamps `QueryAssert` to make failures first class citizen from `QueryAssert` perspective, on par with query results. Expected failures are verified to be `TrinoException` type. The result is more fluent API, as both success and failures assertions are available on the same query assert object, and thus can be auto-completed by an IDE. From code updating perspective, most of the code updates fall into two categories: * failure testing. This functionality is available from query assert via `.failure()` accessor. Code was updated with regular expression - search for: `(assertThatThrownBy|assertTrinoExceptionThrownBy)$\($ -> (assertions\.)?query\(` - replace with: `assertThat($2query(` - add `.failure()` to access assertions on the failure exception (use `nonTrinoExceptionFailure()` for cases where query incorrectly fails without a proper `TrinoException`) * detailed inspection of `MaterializedResult` objects. This lower-level functionality is available from query assert via `.result()` accessor.

findepi added test no-release-notes This pull request does not require release notes entry labels Jan 30, 2024

findepi requested review from martint, wendigo, losipiuk, kokosing and hashhar January 30, 2024 16:16

cla-bot bot added the cla-signed label Jan 30, 2024

findepi marked this pull request as draft January 30, 2024 16:16

findepi marked this pull request as ready for review January 30, 2024 16:16

findepi force-pushed the findepi/assert-failure branch from aab4474 to 888f4d8 Compare January 30, 2024 17:45

github-actions bot added tests:hive iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector bigquery BigQuery connector mongodb MongoDB connector labels Jan 30, 2024

findepi force-pushed the findepi/assert-failure branch from 888f4d8 to b57cc19 Compare January 30, 2024 17:53

wendigo reviewed Jan 30, 2024

View reviewed changes

wendigo previously requested changes Jan 30, 2024

View reviewed changes

findepi added 2 commits January 30, 2024 22:35

Use getOnlyElement to get the only list element

126ff15

Remove unused parameter

eeddc8b

`ensureOrdering` was always false, so effectively unused.

findepi force-pushed the findepi/assert-failure branch from b57cc19 to c57ac24 Compare January 30, 2024 21:35

findepi requested a review from wendigo January 30, 2024 21:35

findepi added 2 commits January 30, 2024 22:43

Inline util class next to only usage

c865eba

findepi force-pushed the findepi/assert-failure branch 3 times, most recently from b09982c to 86afd6b Compare January 30, 2024 21:56

wendigo approved these changes Jan 31, 2024

View reviewed changes

Avoid positively depending on query assert constructor side effect

fe0d836

Update places that depended on `query(...)` query assert constructor executing the query. These should use `execute`, `assertUpdate`, etc.

findepi force-pushed the findepi/assert-failure branch from 86afd6b to 457cef5 Compare January 31, 2024 08:57

losipiuk reviewed Jan 31, 2024

View reviewed changes

findepi mentioned this pull request Jan 31, 2024

More assertion related cleanups #20522

Merged

findepi force-pushed the findepi/assert-failure branch from 457cef5 to 947fc87 Compare January 31, 2024 12:00

findepi merged commit 1da9822 into master Jan 31, 2024
100 checks passed

findepi deleted the findepi/assert-failure branch January 31, 2024 13:29

github-actions bot added this to the 438 milestone Jan 31, 2024

colebow mentioned this pull request Jan 31, 2024

Add Trino 438 release notes #20518

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamp failure testing with query assertions #20509

Revamp failure testing with query assertions #20509

findepi commented Jan 30, 2024 •

edited

Loading

findepi commented Jan 30, 2024

findepi commented Jan 30, 2024 •

edited

Loading

wendigo Jan 30, 2024

findepi Jan 30, 2024

wendigo left a comment

wendigo commented Jan 30, 2024

findepi commented Jan 30, 2024

wendigo left a comment

findepi commented Jan 31, 2024

losipiuk left a comment

wendigo commented Jan 31, 2024

Revamp failure testing with query assertions #20509

Revamp failure testing with query assertions #20509

Conversation

findepi commented Jan 30, 2024 • edited Loading

findepi commented Jan 30, 2024

findepi commented Jan 30, 2024 • edited Loading

wendigo Jan 30, 2024

Choose a reason for hiding this comment

findepi Jan 30, 2024

Choose a reason for hiding this comment

wendigo left a comment

Choose a reason for hiding this comment

wendigo commented Jan 30, 2024

findepi commented Jan 30, 2024

wendigo left a comment

Choose a reason for hiding this comment

findepi commented Jan 31, 2024

losipiuk left a comment

Choose a reason for hiding this comment

wendigo commented Jan 31, 2024

findepi commented Jan 30, 2024 •

edited

Loading

findepi commented Jan 30, 2024 •

edited

Loading