`GTEST_SKIP() << {message}` doesn't output message when executed from SetUp() fixtures #2208

ngie-eign · 2019-03-31T18:11:36Z

gtest_skip_test's Fixture tests test GTEST_SKIP() << "Fixture.SkipsAnotherTest" as part of the fixture setup:

42 class Fixture : public Test {
43  protected:
44   void SetUp() override {
45     GTEST_SKIP() << "skipping all tests for this fixture";
46   }
47 };

While it definitely works (as a general feature), the diagnostic message isn't output, confusing the end-user:

$ /home/ngie/nfs/git/googletest/googlemock/gtest/gtest_skip_test
Running main() from /home/ngie/nfs/git/googletest/googletest/src/gtest_main.cc
[==========] Running 3 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 1 test from SkipTest
[ RUN      ] SkipTest.DoesSkip
[  SKIPPED ] SkipTest.DoesSkip (0 ms)
[----------] 1 test from SkipTest (0 ms total)

[----------] 2 tests from Fixture
[ RUN      ] Fixture.SkipsOneTest
[  SKIPPED ] Fixture.SkipsOneTest (0 ms)
[ RUN      ] Fixture.SkipsAnotherTest
[  SKIPPED ] Fixture.SkipsAnotherTest (0 ms)
[----------] 2 tests from Fixture (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 2 test suites ran. (0 ms total)
[  PASSED  ] 0 tests.
[  SKIPPED ] 3 tests, listed below:
[  SKIPPED ] SkipTest.DoesSkip
[  SKIPPED ] Fixture.SkipsOneTest
[  SKIPPED ] Fixture.SkipsAnotherTest

Fixture.SkipsOneTest and Fixture.SkipsAnotherTest should both be outputting "Fixture.SkipsAnotherTest".

Reported by @asomers as part of https://reviews.freebsd.org/D19765 .

The text was updated successfully, but these errors were encountered:

This change adds fine grained execution of Google Test test programs. First, the Google Test test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub [here](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). This support is a very rough cut to provide an initial working integration effort. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (will be available in version 1.9.0 and is available in FreeBSD base's version of Google Test), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with Google Test, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>

snarkmaster · 2019-06-12T18:42:28Z

This also happens in skips that are not in SetUp.

This also affects me. I think the fix is just:

--- i/googletest/googletest/src/gtest.cc
+++ w/googletest/googletest/src/gtest.cc
@@ -3150,9 +3150,7 @@ void PrettyUnitTestResultPrinter::OnTestStart(const TestInfo& test_info) {
 void PrettyUnitTestResultPrinter::OnTestPartResult(
     const TestPartResult& result) {
   switch (result.type()) {
-    // If the test part succeeded, or was skipped,
-    // we don't need to do anything.
-    case TestPartResult::kSkip:
+    // If the test part succeeded, we don't need to do anything.
     case TestPartResult::kSuccess:
       return;
     default:

I confirmed via a run on an EdenFS test that uses a skip:

[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Clean/RawOverlayTest
[ RUN      ] Clean/RawOverlayTest.inode_number_scan_logs_corrupt_directories_once/0
eden/fs/inodes/test/OverlayTest.cpp:480: SkippedTest skipped by client
scanForNextInodeNumber should not log about corrupt directories on clean restart.
[  SKIPPED ] Clean/RawOverlayTest.inode_number_scan_logs_corrupt_directories_once/0 (14 ms)
[----------] 1 test from Clean/RawOverlayTest (14 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (14 ms total)
[  PASSED  ] 0 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] Clean/RawOverlayTest.inode_number_scan_logs_corrupt_directories_once/0

Prior to the fix, the skip message would not be displayed.

gennadiycivil · 2019-06-12T19:34:57Z

Thank you very much for this report. The best way to approach this would be to create a proper PR and submit it for consideration.

ngie-eign · 2019-06-13T19:05:56Z

This also happens in skips that are not in SetUp.

This also affects me. I think the fix is just:

...

Prior to the fix, the skip message would not be displayed.

@snarkmaster: this proposed change looks good to me. Could you please post a PR with it so it can be run through CI, etc?

Closes google#2208 Previously, skip messages were invisible, so debugging skips was hard. Now we have this: ``` $ ./googletest/gtest_skip_test Running main() from /home/lesha/github/snarkmaster/googletest/googletest/src/gtest_main.cc [==========] Running 3 tests from 2 test suites. [----------] Global test environment set-up. [----------] 1 test from SkipTest [ RUN ] SkipTest.DoesSkip /home/lesha/github/snarkmaster/googletest/googletest/test/gtest_skip_test.cc:38: Skipped skipping single test [ SKIPPED ] SkipTest.DoesSkip (0 ms) [----------] 1 test from SkipTest (1 ms total) ... ```

snarkmaster · 2019-10-17T15:47:14Z

@ngie-eign, in the PR #2517 I made an additional tweak to the string to make the messages look a bit better, but otherwise it's the fix I proposed here (plus a unit test).

Closes google#2208 Previously, skip messages were invisible, so debugging skips was hard. Now we have this: ``` $ ./googletest/gtest_skip_test Running main() from /home/lesha/github/snarkmaster/googletest/googletest/src/gtest_main.cc [==========] Running 3 tests from 2 test suites. [----------] Global test environment set-up. [----------] 1 test from SkipTest [ RUN ] SkipTest.DoesSkip /home/lesha/github/snarkmaster/googletest/googletest/test/gtest_skip_test.cc:38: Skipped skipping single test [ SKIPPED ] SkipTest.DoesSkip (0 ms) [----------] 1 test from SkipTest (1 ms total) ... ```

This change adds fine grained execution of Google Test test programs. First, the Google Test test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub [here](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). This support provides initial working integration for the GoogleTest framework. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (available in GoogleTest 1.9.0+), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with Google Test, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>

This change adds fine grained execution of GoogleTest test programs. How it's done: First, the GoogleTest test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub [here](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). Implementation notes and rationale: This support provides initial working integration for the GoogleTest framework. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (available in GoogleTest 1.9.0+), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with GoogleTest, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>

This change adds fine grained execution of GoogleTest framework-based test programs. How it's done: First, the GoogleTest test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub [here](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). Implementation notes and rationale: This support provides initial working integration for the GoogleTest framework. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (available in GoogleTest 1.9.0+), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with GoogleTest, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>

This change adds fine grained execution of GoogleTest framework-based test programs. First, the GoogleTest test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. Finally, the output from each unique testcase is scraped and digested by Kyua in a format that Kyua supports. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub in my [scratch repo](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). This is initial framework integration. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (available in GoogleTest 1.9.0+), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with GoogleTest, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>

This change adds fine grained execution of GoogleTest framework-based test programs. First, the GoogleTest test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. Finally, the output from each unique testcase is scraped and digested by Kyua in a format that Kyua supports. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub in my [scratch repo](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). This is initial framework integration. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (available in GoogleTest 1.9.0+), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with GoogleTest, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <ngie@FreeBSD.org>

This change adds fine grained execution of Google Test test programs. First, the Google Test test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub [here](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). This support is a very rough cut to provide an initial working integration effort. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (will be available in version 1.9.0 and is available in FreeBSD base's version of Google Test), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with Google Test, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>

This change adds fine grained execution of GoogleTest framework-based test programs. First, the GoogleTest test is executed with `--gtest_list_tests`. Next, based on the output from `--gtest_list_tests`, the testcases are run individually. Finally, the output from each unique testcase is scraped and digested by Kyua in a format that Kyua supports. The output from each unique testcase is based on the standard output it provides, per the test output protocol defined in the GoogleTest docs on github [1], [2], and instrumented via various demo programs I created, which can be found on GitHub in my [scratch repo](https://github.com/ngie-eign/scratch/tree/master/programming/c%2B%2B/gtest). This is initial framework integration. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library. This test interface supports fine grained execution of test programs like the ATF test interface, but matches behavior of plain/TAP interfaces by instead supporting metadata passing via `$TEST_ENV_` prefixed environment variables. This support adds additional tests for verifying pass, fail, skip (available in GoogleTest 1.9.0+), and disabled result determination, using mock output and a mock test program (`engine/googletest_helpers`), for parity with other test formats (ATF, plain, TAP). The helper test program purposely avoids relying on `getopt_long*` for portability reasons, and the GoogleTest test infrastructure, in order to limit Kyua's dependencies. As part of this change, `store/read_transaction.cpp` needed to support optional reasons provided with skip results. While it's bad form to omit test results with tests, providing a reason is optional with GoogleTest, and unfortunately not all portions of the test framework output a reason when `GTEST_SKIP()` is called. See the issue in [3] for one such example issue when `GTEST_SKIP()` is called from SetUp test fixtures. 1. https://github.com/google/googletest/blob/master/googletest/docs/primer.md 2. https://github.com/google/googletest/blob/master/googletest/docs/advanced.md 3. google/googletest#2208 Signed-off-by: Enji Cooper <ngie@FreeBSD.org>

ngie-eign changed the title ~~GTEST_SKIP() < <message> doesn't output message when executed from SetUp() fixtures~~ GTEST_SKIP() << {message} doesn't output message when executed from SetUp() fixtures Mar 31, 2019

snarkmaster mentioned this issue Oct 17, 2019

[googletest] Output skip message #2517

Merged

gennadiycivil closed this as completed in cbf019d Oct 22, 2019

ngie-eign mentioned this issue Aug 20, 2023

Add GoogleTest support freebsd/kyua-fork#11

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`GTEST_SKIP() << {message}` doesn't output message when executed from SetUp() fixtures #2208

`GTEST_SKIP() << {message}` doesn't output message when executed from SetUp() fixtures #2208

ngie-eign commented Mar 31, 2019

snarkmaster commented Jun 12, 2019

gennadiycivil commented Jun 12, 2019

ngie-eign commented Jun 13, 2019

snarkmaster commented Oct 17, 2019

GTEST_SKIP() << {message} doesn't output message when executed from SetUp() fixtures #2208

GTEST_SKIP() << {message} doesn't output message when executed from SetUp() fixtures #2208

Comments

ngie-eign commented Mar 31, 2019

snarkmaster commented Jun 12, 2019

gennadiycivil commented Jun 12, 2019

ngie-eign commented Jun 13, 2019

snarkmaster commented Oct 17, 2019

`GTEST_SKIP() << {message}` doesn't output message when executed from SetUp() fixtures #2208

`GTEST_SKIP() << {message}` doesn't output message when executed from SetUp() fixtures #2208