Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SonarQube Generic Execution Test Data reporter #1738

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ coverage:
- "**/catch_reporter_tap.hpp"
- "**/catch_reporter_automake.hpp"
- "**/catch_reporter_teamcity.hpp"
- "**/catch_reporter_sonarqube.hpp"
- "**/external/clara.hpp"


Expand Down
6 changes: 5 additions & 1 deletion docs/ci-and-misc.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Build Systems may refer to low-level tools, like CMake, or larger systems that r

## Continuous Integration systems

Probably the most important aspect to using Catch with a build server is the use of different reporters. Catch comes bundled with three reporters that should cover the majority of build servers out there - although adding more for better integration with some is always a possibility (currently we also offer TeamCity, TAP and Automake reporters).
Probably the most important aspect to using Catch with a build server is the use of different reporters. Catch comes bundled with three reporters that should cover the majority of build servers out there - although adding more for better integration with some is always a possibility (currently we also offer TeamCity, TAP, Automake and SonarQube reporters).

Two of these reporters are built in (XML and JUnit) and the third (TeamCity) is included as a separate header. It's possible that the other two may be split out in the future too - as that would make the core of Catch smaller for those that don't need them.

Expand Down Expand Up @@ -65,6 +65,10 @@ The Automake Reporter writes out the [meta tags](https://www.gnu.org/software/au

Because of the incremental nature of Catch's test suites and ability to run specific tests, our implementation of TAP reporter writes out the number of tests in a suite last.

### SonarQube Reporter
```-r sonarqube```
[SonarQube Generic Test Data](https://docs.sonarqube.org/latest/analysis/generic-test/) XML format for tests metrics.

## Low-level tools

### Precompiled headers (PCHs)
Expand Down
4 changes: 2 additions & 2 deletions docs/release-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ Tag version and release title should be same as the new version,
description should contain the release notes for the current release.
Single header version of `catch.hpp` *needs* to be attached as a binary,
as that is where the official download link links to. Preferably
it should use linux line endings. All non-bundled reporters (Automake,
TAP, TeamCity) should also be attached as binaries, as they might be
it should use linux line endings. All non-bundled reporters (Automake, TAP,
TeamCity, SonarQube) should also be attached as binaries, as they might be
dependent on a specific version of the single-include header.

Since 2.5.0, the release tag and the "binaries" (headers) should be PGP
Expand Down
1 change: 1 addition & 0 deletions docs/reporters.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Do this in one source file - the same one you have `CATCH_CONFIG_MAIN` or `CATCH
Use this when building as part of a TeamCity build to see results as they happen ([code example](../examples/207-Rpt-TeamCityReporter.cpp)).
* `tap` writes in the TAP ([Test Anything Protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol)) format.
* `automake` writes in a format that correspond to [automake .trs](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html) files
* `sonarqube` writes the [SonarQube Generic Test Data](https://docs.sonarqube.org/latest/analysis/generic-test/) XML format.

You see what reporters are available from the command line by running with `--list-reporters`.

Expand Down
181 changes: 181 additions & 0 deletions include/reporters/catch_reporter_sonarqube.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
/*
* Created by Daniel Garcia on 2018-12-04.
* Copyright Social Point SL. All rights reserved.
*
* Distributed under the Boost Software License, Version 1.0. (See accompanying
* file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
*/
#ifndef CATCH_REPORTER_SONARQUBE_HPP_INCLUDED
#define CATCH_REPORTER_SONARQUBE_HPP_INCLUDED


// Don't #include any Catch headers here - we can assume they are already
// included before this header.
// This is not good practice in general but is necessary in this case so this
// file can be distributed as a single header that works with the main
// Catch single header.

#include <map>

namespace Catch {

struct SonarQubeReporter : CumulativeReporterBase<SonarQubeReporter> {

SonarQubeReporter(ReporterConfig const& config)
: CumulativeReporterBase(config)
, xml(config.stream()) {
m_reporterPrefs.shouldRedirectStdOut = true;
m_reporterPrefs.shouldReportAllAssertions = true;
}

~SonarQubeReporter() override;

static std::string getDescription() {
return "Reports test results in the Generic Test Data SonarQube XML format";
}

static std::set<Verbosity> getSupportedVerbosities() {
return { Verbosity::Normal };
}

void noMatchingTestCases(std::string const& /*spec*/) override {}

void testRunStarting(TestRunInfo const& testRunInfo) override {
CumulativeReporterBase::testRunStarting(testRunInfo);
xml.startElement("testExecutions");
xml.writeAttribute("version", "1");
}

void testGroupEnded(TestGroupStats const& testGroupStats) override {
CumulativeReporterBase::testGroupEnded(testGroupStats);
writeGroup(*m_testGroups.back());
}

void testRunEndedCumulative() override {
xml.endElement();
}

void writeGroup(TestGroupNode const& groupNode) {
std::map<std::string, TestGroupNode::ChildNodes> testsPerFile;
for(auto const& child : groupNode.children)
testsPerFile[child->value.testInfo.lineInfo.file].push_back(child);

for(auto const& kv : testsPerFile)
writeTestFile(kv.first.c_str(), kv.second);
}

void writeTestFile(const char* filename, TestGroupNode::ChildNodes const& testCaseNodes) {
XmlWriter::ScopedElement e = xml.scopedElement("file");
xml.writeAttribute("path", filename);

for(auto const& child : testCaseNodes)
writeTestCase(*child);
}

void writeTestCase(TestCaseNode const& testCaseNode) {
// All test cases have exactly one section - which represents the
// test case itself. That section may have 0-n nested sections
assert(testCaseNode.children.size() == 1);
SectionNode const& rootSection = *testCaseNode.children.front();
writeSection("", rootSection, testCaseNode.value.testInfo.okToFail());
}

void writeSection(std::string const& rootName, SectionNode const& sectionNode, bool okToFail) {
std::string name = trim(sectionNode.stats.sectionInfo.name);
if(!rootName.empty())
name = rootName + '/' + name;

if(!sectionNode.assertions.empty() || !sectionNode.stdOut.empty() || !sectionNode.stdErr.empty()) {
XmlWriter::ScopedElement e = xml.scopedElement("testCase");
xml.writeAttribute("name", name);
xml.writeAttribute("duration", static_cast<long>(sectionNode.stats.durationInSeconds * 1000));

writeAssertions(sectionNode, okToFail);
}

for(auto const& childNode : sectionNode.childSections)
writeSection(name, *childNode, okToFail);
}

void writeAssertions(SectionNode const& sectionNode, bool okToFail) {
for(auto const& assertion : sectionNode.assertions)
writeAssertion( assertion, okToFail);
}

void writeAssertion(AssertionStats const& stats, bool okToFail) {
AssertionResult const& result = stats.assertionResult;
if(!result.isOk()) {
std::string elementName;
if(okToFail) {
elementName = "skipped";
}
else {
switch(result.getResultType()) {
case ResultWas::ThrewException:
case ResultWas::FatalErrorCondition:
elementName = "error";
break;
case ResultWas::ExplicitFailure:
elementName = "failure";
break;
case ResultWas::ExpressionFailed:
elementName = "failure";
break;
case ResultWas::DidntThrowException:
elementName = "failure";
break;

// We should never see these here:
case ResultWas::Info:
case ResultWas::Warning:
case ResultWas::Ok:
case ResultWas::Unknown:
case ResultWas::FailureBit:
case ResultWas::Exception:
elementName = "internalError";
break;
}
}

XmlWriter::ScopedElement e = xml.scopedElement(elementName);

ReusableStringStream messageRss;
messageRss << result.getTestMacroName() << "(" << result.getExpression() << ")";
xml.writeAttribute("message", messageRss.str());

ReusableStringStream textRss;
if (stats.totals.assertions.total() > 0) {
textRss << "FAILED:\n";
if (result.hasExpression()) {
textRss << "\t" << result.getExpressionInMacro() << "\n";
}
if (result.hasExpandedExpression()) {
textRss << "with expansion:\n\t" << result.getExpandedExpression() << "\n";
}
}

if(!result.getMessage().empty())
textRss << result.getMessage() << "\n";

for(auto const& msg : stats.infoMessages)
if(msg.type == ResultWas::Info)
textRss << msg.message << "\n";

textRss << "at " << result.getSourceInfo();
xml.writeText(textRss.str(), false);
}
}

private:
XmlWriter xml;
};

#ifdef CATCH_IMPL
SonarQubeReporter::~SonarQubeReporter() {}
#endif

CATCH_REGISTER_REPORTER( "sonarqube", SonarQubeReporter )

} // end namespace Catch

#endif // CATCH_REPORTER_SONARQUBE_HPP_INCLUDED
1 change: 1 addition & 0 deletions projects/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,7 @@ set(REPORTER_HEADERS
${HEADER_DIR}/reporters/catch_reporter_tap.hpp
${HEADER_DIR}/reporters/catch_reporter_teamcity.hpp
${HEADER_DIR}/reporters/catch_reporter_xml.h
${HEADER_DIR}/reporters/catch_reporter_sonarqube.hpp
)
set(REPORTER_SOURCES
${HEADER_DIR}/reporters/catch_reporter_bases.cpp
Expand Down
Loading