Cosmos: Review tests throwing client-evaluation exceptions #17246
Milestone
Comments
AndriySvyryd
changed the title
ajcvickers
added a commit
that referenced
this issue
Oct 17, 2021
Part of #26088 and #17235 Ideas for better catching behavior changes in the product code. Specifically: - Detect when a negative case stops failing - Detect when a negative case starts failing in a different way Fundamental approach: don't skip tests. In NorthwindAggregateOperatorsQueryTests, we had: - Negative cases that were no longer failing - Negative cases that were skipped for all providers, but worked on some. For example: - Failed on relational, but passed on in-memory - Failed on relational, but passed on Cosmos - Failed on SQL Server, but passed on SQLite - Negative cases that failed in different ways on different providers Specifics: - If a test throws, catch the exception - Were feasible, also validate the exception message or error number - Always call base where possible, rather than repeating the query in an overriden test - Add a standard comment where we have a bug or enhancement tracking the issue. For example: - `// Contains over subquery. Issue #17246.` - Always have an `AssertSql` call in Cosmos and SQL Server tests - Where we expect a provider-specific class to verify SQL, then add a test that checks all test methods are overriden.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Cosmos has ~1000 tests disabled referencing #14935 (now referencing this issue). Filing this issue to handle these and either skip or check an appropriate exception message is being thrown.
The text was updated successfully, but these errors were encountered: