Test failed using the JUnit reporter but not using the console or xml reporter

[Alzathar]

Describe the bug
When I run the following test, the last check CHECK( i == 3 ) fails when I use the JUnit reporter (passing the -r junit argument to the binary), but passes when I use the xml or console reporter.

    // Stupid test to understand the behaviour of the std::filesystem library
    auto it2 = std::filesystem::directory_iterator("C:/tests/sample_dir", ec);
    int i = 0;
    for ( ; it2 != std::filesystem::directory_iterator() ; it2.increment(ec)) {
      REQUIRE( !ec );
      CHECK( it2 != std::filesystem::directory_iterator() );
      ++i;
    }
    CHECK( i == 3 ); // <-- This fails using the JUnit reporter but not with the xml and console reporter

Expected behavior
The test should pass for all reporters

Reproduction steps

1. Create a project to generate a binary and add the code below.
2. Create on your computer a folder and add 3 items (files or folders). If you are under macOS, you may have to create only two items because the Finder application will create a .DS_Store file (or you can adapt the test in consequence)
3. Run the binary using the argument -r console (pass)
4. Run the binary using the argument -r xml (pass)
5. Run the binary using the argument -r junit (fail)

// Copy/paste from a larger CPP file. This should work as is.

#define CATCH_CONFIG_MAIN
#include <catch.hpp>
#include <filesystem>
#include <system_error>

TEST_CASE("std::filesystem::directory_iterator") {
  SECTION("using a directory") {
    std::error_code ec;
    auto it2 = std::filesystem::directory_iterator("C:/tests/sample_dir", ec);
    int i = 0;
    for ( ; it2 != std::filesystem::directory_iterator() ; it2.increment(ec)) {
      REQUIRE( !ec );
      CHECK( it2 != std::filesystem::directory_iterator() );
      ++i;
    }
    CHECK( i == 3 ); 
  }
}

Platform information:

• Windows
  • OS: Win10 (1809)
  • Compiler+version: MSVC 2019
  • Catch version: v2.10.2 (tested with v2.10.0 too)
  • Build type: Debug
• macOS
  • OS: macOS 10.14
  • Compiler+version: XCode 11
  • Catch version: v2.10.0
  • Build type: Debug
• Azure (image windows-2019)
  • OS: Windows Server 2019
  • Compiler+version: Visual Studio 16 2019
  • Catch version: v2.10.0
  • Build type: Debug
• Azure (image vs2017-win2016)
  • OS: Windows Server 2016
  • Compiler+version: Visual Studio 15 2017
  • Catch version: v2.10.0
  • Build type: Debug

Supplementary information:
I discovered this problem does not happen when I comment the first CHECK command.
The following test does not fail with the JUnit reporter

    std::error_code ec;
    auto it2 = std::filesystem::directory_iterator("C:/tests/sample_dir", ec);
    int i = 0;
    for ( ; it2 != std::filesystem::directory_iterator() ; it2.increment(ec)) {
      REQUIRE( !ec );
      // CHECK( it2 != std::filesystem::directory_iterator() ); // <-- Pass with the JUnit reporter if this line is commented!
      ++i;
    }
    CHECK( i == 3 ); 

[JohannGerellTobii]

We're seeing very similar results for some build configs for all reporters except console.

[Alzathar]

This issue still happens using Catch2 v2.13.2 and the Junit reporter but only under macOS 10.15 with Xcode 11.7. I have not anymore this issue under Windows and Visual Studio 2019.

I tried to narrowed the problem and I have this assertion failure:

Assertion failed: (m_sectionStack.size() == 0), function testCaseEnded, file catch2/single_include/catch2/catch.hpp, line 5913.

This assertion failure was also mentioned in issues #1967 and #1210

The workaround found to suppress the crash is to comment the checks with the (un)equal operator for the std::filesystem::directory_iteratorclass.

auto it1 = std::filesystem::directory_iterator("")
auto it2 = it1; // NOLINT (test the copy)
CHECK( it1 == it2 );  // Must be commented otherwise it crashes
CHECK( it2 != std::filesystem::directory_iterator() ); // Must be commented otherwise it crashes

[Petezah]

I just hit this as well, using a REQUIRE inside of a test listener. Removing the REQUIRE from the test listener fixes the issue (only on Mac though; does not happen on Windows/Visual Studio 2017). I'll try to make a min repro if I have time.

[Petezah]

I was able to repro this minimally with this:

#define CATCH_CONFIG_MAIN
#include <catch.hpp>

struct RequireReproListener :
    Catch::TestEventListenerBase {
    using TestEventListenerBase::TestEventListenerBase;
    void testCaseStarting(Catch::TestCaseInfo const &testInfo) override
    {
        REQUIRE(true);
    }

    void testCaseEnded(Catch::TestCaseStats const &testCaseStats) override
    {
        REQUIRE(true);
    }
};

CATCH_REGISTER_LISTENER(RequireReproListener)

TEST_CASE("Repro test")
{
    REQUIRE(true);
}

If you build this on Linux with GCC (I was using gcc version 9.3.0), and run with "-r junit", it will fail with:

<?xml version="1.0" encoding="UTF-8"?>
catch2_repro: /home/petezah/catch2_repro/./catch.hpp:5899: bool Catch::CumulativeReporterBase<DerivedT>::assertionEnded(const Catch::AssertionStats&) [with DerivedT = Catch::JunitReporter]: Assertion `!m_sectionStack.empty()' failed.
Aborted

If you run without the flags, you get:

===============================================================================
All tests passed (3 assertions in 1 test case)