Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CTest (using catch_discover_tests) treats skipped tests as failed #2873

Closed
jonathanverner opened this issue May 20, 2024 · 2 comments
Closed

CTest (using catch_discover_tests) treats skipped tests as failed #2873

jonathanverner opened this issue May 20, 2024 · 2 comments
Labels
Extras Touches utility scripts outside of Catch2 proper, e.g. CMake integration.

Comments

@jonathanverner
Copy link

Describe the bug
When using CMake integration to auto-discover tests and run them with CTest, tests which are skipped are reported as failed.

Expected behavior
Skipped tests are reported as skipped.

Platform information:

  • OS: Linux (Neon 22.04, based on Ubuntu 22.04)
  • Compiler+version: 17.0.6
  • Catch version: v3.3.2

Additional context
I think the reason is that ctest runs the tests one-by-one and if a test is skipped, this actually means that no test was run so, since --allow-running-no-tests is not provided, the return code is non-zero.

Now, CTest allows marking skipped tests via a special return code by setting the SKIP_RETURN_CODE property on the test. So, it seems to me, this could be fixed by replacing the

set(properties ${TEST_PROPERTIES})

line in CatchAddTests.cmake with

set(properties "${TEST_PROPERTIES};SKIP_RETURN_CODE;4")

This works for me as does passing SKIP_RETURN_CODE 4 to catch_discover_tests PROPETIES argument, e.g. like so:

catch_discover_tests(middleware_tests
  PROPERTIES
    SKIP_RETURN_CODE 4
)
@RichLogan RichLogan mentioned this issue Jul 3, 2024
Merged
fredemmott added a commit to fredemmott/bindline that referenced this issue Aug 6, 2024
@fredemmott
Make CTest understand skipped catch2 tests
3c03066 
Fix from catchorg/Catch2#2873
@fredemmott
Copy link

This ties into #1146 : unfortunately using SKIP_RETURN_CODE 4 means that if your test has 4 failed CHECK() invocations, CTest will consider it to be skipped.

@fredemmott
Copy link

For now I'm using SKIP_REGULAR_EXPRESSION "[1-9][0-9]* skipped"

@horenmar horenmar added the Extras Touches utility scripts outside of Catch2 proper, e.g. CMake integration. label Sep 17, 2024
GerHobbelt added a commit to GerHobbelt/Catch2 that referenced this issue Sep 27, 2024
@GerHobbelt
Merge tag 'v3.7.1' into googletest
f318c99 
v3.7.1

=== Improvements ===
* Applied the JUnit reporter's optimization from last release to the SonarQube reporter
* Suppressed `-Wuseless-cast` in `CHECK_THROWS_MATCHES` (catchorg#2904)
* Standardize exit codes for various failures
  * Running no tests is now guaranteed to exit with 2 (without the `--allow-running-no-tests` flag)
  * All tests skipped is now always 4 (...)
  * Assertion failures are now always 42
  * and so on

=== Fixes ===
* Fixed out-of-bounds access when the arg parser encounters single `-` as an argument (catchorg#2905)

=== Miscellaneous ===
* Added `catch_config_prefix_messages.hpp` to meson build (catchorg#2903)
* `catch_discover_tests` now supports skipped tests (catchorg#2873)
  * You can get the old behaviour by calling `catch_discover_tests` with `SKIP_IS_FAILURE` option.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCgAdFiEE8QyLZSqAHw/oZQgX3kgwe4sNOBoFAmbpQnMACgkQ3kgwe4sN
# OBpa1Q//UbNsxKpLU7cKzCr3kEOOc2RsryS5foHYRs+mhUzOFYf4y1HkSdvzc3TV
# ZUI8sA5FA1OVO7JtCSpgzizF5vGiOunDFGgroSw9kIxsX0KtHugksNgQcU7QeFEf
# 6KqtKb4/tmRryuyoHZTdWMxBGpB4dtc3zy1Yq+ebe8k52neBmilb80Mz9KbuQ4vT
# QAXz7xWiZukHyxosmKZe5kxDLF3Ck5iVCvvUJu9iwiE6Ua8TIJgs+sOFApsQztmu
# 5hwrmajqZGLB65RQe4G6AXScNLGnzRTJwByR5fg+YFZQmUdjSE0oQ6rd2Ni9qk5c
# QUOPBAVA2A3rahnV3k9bY6SapFjeTZlO4/go9eSzswc2CU7LCX0jRME5d+oW6YJ4
# H0XgacVYM362Exd4hKsSplfuvwkRi1irlnGxHpTvBLBp1czJgUnQV1lwsncaiQFx
# IZfxiqot+V0fPEweVkgHOwf175o9SLLZka6546MFgSvWY8yEEMnl3nLrmeHhCFPq
# 70MdKcj/Qbzkzd5PTSuUoUhxIFZJMzsszj9b0pdGlCb5ObRMcN0UHKG9nVNXWTmn
# tWnukCdzGHsZzaHjH/J0IY8P9ANNjAb5xtROzuF5TmE6h+D5J6kwxGTqZoCyqmC/
# FyPPf1WYkrU2qj+ms7SbdyGlE7s1w8VF/7p3/EKwoE2fpZk/9WM=
# =lDEO
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Sep 17 10:48:51 2024 WEDT
# gpg:                using RSA key F10C8B652A801F0FE8650817DE48307B8B0D381A
# gpg: Can't check signature: No public key

# Conflicts:
#	README.md
#	conanfile.py
#	docs/release-notes.md
#	docs/tutorial.md
#	include/internal/catch_compiler_capabilities.h
#	include/internal/catch_debugger.h
#	include/internal/catch_evaluate.hpp
#	include/internal/catch_expression_lhs.hpp
#	include/internal/catch_fatal_condition.hpp
#	include/internal/catch_message.hpp
#	include/internal/catch_result_builder.hpp
#	include/internal/catch_run_context.hpp
#	include/internal/catch_test_case_registry_impl.hpp
#	include/reporters/catch_reporter_console.hpp
#	include/reporters/catch_reporter_teamcity.hpp
#	include/reporters/catch_reporter_xml.hpp
#	scripts/updateVcpkgPackage.py
#	single_include/catch.hpp
#	src/catch2/catch_version.cpp
#	test_package/conanfile.py
GlynLeine added a commit to Rythe-Interactive/Catch2 that referenced this issue Oct 15, 2024
@GlynLeine
Merge tag 'v3.7.1' into rythe
e6254f3 
v3.7.1

=== Improvements ===
* Applied the JUnit reporter's optimization from last release to the SonarQube reporter
* Suppressed `-Wuseless-cast` in `CHECK_THROWS_MATCHES` (catchorg#2904)
* Standardize exit codes for various failures
  * Running no tests is now guaranteed to exit with 2 (without the `--allow-running-no-tests` flag)
  * All tests skipped is now always 4 (...)
  * Assertion failures are now always 42
  * and so on

=== Fixes ===
* Fixed out-of-bounds access when the arg parser encounters single `-` as an argument (catchorg#2905)

=== Miscellaneous ===
* Added `catch_config_prefix_messages.hpp` to meson build (catchorg#2903)
* `catch_discover_tests` now supports skipped tests (catchorg#2873)
  * You can get the old behaviour by calling `catch_discover_tests` with `SKIP_IS_FAILURE` option.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCgAdFiEE8QyLZSqAHw/oZQgX3kgwe4sNOBoFAmbpQnMACgkQ3kgwe4sN
# OBpa1Q//UbNsxKpLU7cKzCr3kEOOc2RsryS5foHYRs+mhUzOFYf4y1HkSdvzc3TV
# ZUI8sA5FA1OVO7JtCSpgzizF5vGiOunDFGgroSw9kIxsX0KtHugksNgQcU7QeFEf
# 6KqtKb4/tmRryuyoHZTdWMxBGpB4dtc3zy1Yq+ebe8k52neBmilb80Mz9KbuQ4vT
# QAXz7xWiZukHyxosmKZe5kxDLF3Ck5iVCvvUJu9iwiE6Ua8TIJgs+sOFApsQztmu
# 5hwrmajqZGLB65RQe4G6AXScNLGnzRTJwByR5fg+YFZQmUdjSE0oQ6rd2Ni9qk5c
# QUOPBAVA2A3rahnV3k9bY6SapFjeTZlO4/go9eSzswc2CU7LCX0jRME5d+oW6YJ4
# H0XgacVYM362Exd4hKsSplfuvwkRi1irlnGxHpTvBLBp1czJgUnQV1lwsncaiQFx
# IZfxiqot+V0fPEweVkgHOwf175o9SLLZka6546MFgSvWY8yEEMnl3nLrmeHhCFPq
# 70MdKcj/Qbzkzd5PTSuUoUhxIFZJMzsszj9b0pdGlCb5ObRMcN0UHKG9nVNXWTmn
# tWnukCdzGHsZzaHjH/J0IY8P9ANNjAb5xtROzuF5TmE6h+D5J6kwxGTqZoCyqmC/
# FyPPf1WYkrU2qj+ms7SbdyGlE7s1w8VF/7p3/EKwoE2fpZk/9WM=
# =lDEO
# -----END PGP SIGNATURE-----
# gpg: directory '/c/Users/Ty Rymer/.gnupg' created
# gpg: Signature made Tue Sep 17 10:48:51 2024 WEST
# gpg:                using RSA key F10C8B652A801F0FE8650817DE48307B8B0D381A
# gpg: Can't check signature: No public key
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Extras Touches utility scripts outside of Catch2 proper, e.g. CMake integration.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jonathanverner @fredemmott @horenmar