Skip to content

Commit

Permalink
v0.10.0
Browse files Browse the repository at this point in the history
  • Loading branch information
@Paebbels
Paebbels authored Jun 3, 2024
2 parents c00951c + 42ae24a commit ae5bf3d
Show file tree
Hide file tree
Showing 20 changed files with 442 additions and 3,340 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/Pipeline.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ jobs:
- UnitTestingParams
- UnitTesting
with:
additional_merge_args: '--pytest "pytest.tests.unit"'
additional_merge_args: '"--pytest=reduce-depth:pytest.tests.unit"'
merged_junit_artifact: ${{ fromJson(needs.UnitTestingParams.outputs.artifact_names).unittesting_xml }}

IntermediateCleanUp:
Expand Down
6 changes: 3 additions & 3 deletions doc/Dependency.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ pyEDAA.Reports Package
+-----------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| **Package** | **Version** | **License** | **Dependencies** |
+=================================================================+=============+===========================================================================================+========================================================================================================================================================+
| `pyTooling <https://GitHub.com/pyTooling/pyTooling>`__ | ≥6.1.0 | `Apache License, 2.0 <https://GitHub.com/pyTooling/pyTooling/blob/main/LICENSE.txt>`__ | *None* |
| `pyTooling <https://GitHub.com/pyTooling/pyTooling>`__ | ≥6.3 | `Apache License, 2.0 <https://GitHub.com/pyTooling/pyTooling/blob/main/LICENSE.txt>`__ | *None* |
+-----------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| `ruamel.yaml <https://sourceforge.net/projects/ruamel-yaml/>`__ | ≥0.18 | `MIT <https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/LICENSE>`__ | *Not yet evaluated.* |
+-----------------------------------------------------------------+-------------+-------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Expand Down Expand Up @@ -97,7 +97,7 @@ the mandatory dependencies too.
+-------------------------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| **Package** | **Version** | **License** | **Dependencies** |
+=================================================================================================+==============+==========================================================================================================+======================================================================================================================================================+
| `pyTooling <https://GitHub.com/pyTooling/pyTooling>`__ | ≥6.1.0 | `Apache License, 2.0 <https://GitHub.com/pyTooling/pyTooling/blob/main/LICENSE.md>`__ | *None* |
| `pyTooling <https://GitHub.com/pyTooling/pyTooling>`__ | ≥6.3 | `Apache License, 2.0 <https://GitHub.com/pyTooling/pyTooling/blob/main/LICENSE.md>`__ | *None* |
+-------------------------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| `Sphinx <https://GitHub.com/sphinx-doc/sphinx>`__ | ≥7.3 | `BSD 3-Clause <https://GitHub.com/sphinx-doc/sphinx/blob/master/LICENSE>`__ | *Not yet evaluated.* |
+-------------------------------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
Expand Down Expand Up @@ -133,7 +133,7 @@ install the mandatory dependencies too.
+----------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| **Package** | **Version** | **License** | **Dependencies** |
+============================================================================+==============+==========================================================================================================+======================================================================================================================================================+
| `pyTooling <https://GitHub.com/pyTooling/pyTooling>`__ | ≥6.1.0 | `Apache License, 2.0 <https://GitHub.com/pyTooling/pyTooling/blob/main/LICENSE.md>`__ | *None* |
| `pyTooling <https://GitHub.com/pyTooling/pyTooling>`__ | ≥6.3 | `Apache License, 2.0 <https://GitHub.com/pyTooling/pyTooling/blob/main/LICENSE.md>`__ | *None* |
+----------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| `wheel <https://GitHub.com/pypa/wheel>`__ | ≥0.43 | `MIT <https://github.com/pypa/wheel/blob/main/LICENSE.txt>`__ | *Not yet evaluated.* |
+----------------------------------------------------------------------------+--------------+----------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
Expand Down
82 changes: 81 additions & 1 deletion doc/Unittesting/index.rst
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,84 @@
Unittesting
###########

.. todo:: Write Unittesting introduction.
*pyEDAA.Reports* provides a unified and generic unittest summary data model. The data model allows the description of
testcases grouped in testsuites. Testsuites can be nested in other testsuites. The data model's root element is a
special testsuite called testsuite summary. It contains only testsuites, but no testcases.

.. todo:: UNIT:: Add data model diagram

The data model can be filled from various sources like **Ant JUnit test reports** or **OSVVM testsuite summaries** (more
to be added). Many programming languages and/or unit testing frameworks support exporting results in the Ant JUnit
format. See below for supported formats and their variations.

.. attention::

The so called JUnit XML format is the weakest file format and standard ever seen. At first it was not created. At
first was not created by JUnit (version 4). It was added by the built system Ant, but it's not called Ant XML format
nor Ant JUnit XML format. The latest JUnit 5 uses a completely different format called open test reporting. As JUnit
is not the formats author, no file format documentation nor XML schema was provided. Also Ant isn't providing any
file format documentation or XML schema. Various Ant JUnit XML adopters have tried to reverse engineer a description
and XML schemas, but unfortunately many are not even compatible to each other.

.. todo::

https://github.com/ota4j-team/open-test-reporting

.. admonition:: default box

test text

.. rubric:: Supported Ant JUnit XMl file outputs

* pytest
* VUnit
* OSVVM (OSVVM's YAML format should be preferred due to more content and meta information)

.. rubric:: Supported proprietary file formats

* OSVVM (YAML files)


File Formats
************

Ant and JUnit 4 XML
===================


JUnit 5 XML
===========


OSVVM YAML
==========


Frameworks
**********

JUnit
=====


OSVVM
=====


pytest
======


VUnit
=====


Consumers
*********

Jenkins
=======


Dorney
======
14 changes: 14 additions & 0 deletions doc/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,20 @@ Proposal to define an abstract model for outputs from EDA tools and logging libr

<br>


.. rubric:: Supported Ant JUnit XMl file outputs

* pytest
* VUnit
* OSVVM (OSVVM's YAML format should be preferred due to more content and meta information)

.. rubric:: Supported proprietary file formats

* OSVVM (YAML files)




.. _contributors:

Contributors
Expand Down
2 changes: 1 addition & 1 deletion doc/requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
-r ../requirements.txt

pyTooling ~= 6.1
pyTooling ~= 6.3
colorama >= 0.4.6
ruamel.yaml ~= 0.18.6
setuptools ~= 70.0
Expand Down
69 changes: 43 additions & 26 deletions pyEDAA/Reports/CLI/Unittesting.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
from pyTooling.Attributes.ArgParse import CommandHandler
from pyTooling.Attributes.ArgParse.ValuedFlag import LongValuedFlag

from pyEDAA.Reports.Unittesting import TestsuiteKind, TestsuiteSummary, MergedTestsuiteSummary, UnittestException
from pyEDAA.Reports.Unittesting import UnittestException, TestsuiteKind, TestsuiteSummary, Testsuite, Testcase
from pyEDAA.Reports.Unittesting import MergedTestsuiteSummary
from pyEDAA.Reports.Unittesting.JUnit import JUnitReaderMode


Expand Down Expand Up @@ -45,11 +46,6 @@ def HandleUnittest(self, args: Namespace) -> None:
if args.pytest is not None:
self._processPyTest(result, args.pytest)

if args.output is not None:
outputs = (args.output, )
for output in outputs:
self._output(result, output)

if args.render is not None and self.Verbose:
self.WriteVerbose("*" * self.Width)

Expand All @@ -59,6 +55,11 @@ def HandleUnittest(self, args: Namespace) -> None:

self.WriteVerbose("*" * self.Width)

if args.output is not None:
outputs = (args.output, )
for output in outputs:
self._output(result, output)

self.ExitOnPreviousErrors()

def _merge(self, testsuiteSummary: MergedTestsuiteSummary, task: str) -> None:
Expand All @@ -68,26 +69,31 @@ def _merge(self, testsuiteSummary: MergedTestsuiteSummary, task: str) -> None:
elif l == 2:
dialect, format = (x.lower() for x in parts[0].split("-"))
globPattern = parts[1]

foundFiles = tuple(f for f in Path.cwd().glob(globPattern))
if len(foundFiles) == 0:
self.WriteWarning(f"Found no matching files for pattern '{Path.cwd()}/{globPattern}'")
return

if format == "junit":
if dialect == "ant":
self._mergeAntJUnit(testsuiteSummary, globPattern)
self._mergeAntJUnit(testsuiteSummary, foundFiles)
elif dialect == "ctest":
self._mergeCTestJUnit(testsuiteSummary, globPattern)
self._mergeCTestJUnit(testsuiteSummary, foundFiles)
elif dialect == "gtest":
self._mergeGoogleTestJUnit(testsuiteSummary, globPattern)
self._mergeGoogleTestJUnit(testsuiteSummary, foundFiles)
elif dialect == "pytest":
self._mergePyTestJUnit(testsuiteSummary, globPattern)
self._mergePyTestJUnit(testsuiteSummary, foundFiles)
else:
self.WriteError(f"Unsupported JUnit XML dialect for merging: '{format}'")
else:
self.WriteError(f"Unsupported unit testing report format for merging: '{format}'")
else:
self.WriteError(f"Syntax error: '{task}'")

def _mergeAntJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPattern: str) -> None:
from pyEDAA.Reports.Unittesting.JUnit.AntJUnit import Document, JUnitReaderMode
def _mergeAntJUnit(self, testsuiteSummary: MergedTestsuiteSummary, foundFiles: Tuple[Path]) -> None:
from pyEDAA.Reports.Unittesting.JUnit.AntJUnit import Document

foundFiles = [f for f in Path.cwd().glob(globPattern)]
self.WriteNormal(f"Reading {len(foundFiles)} Ant-JUnit unit test summary files ...")

junitDocuments: List[Document] = []
Expand All @@ -100,10 +106,9 @@ def _mergeAntJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPattern:
self.WriteVerbose(f" merging {summary.Path}")
testsuiteSummary.Merge(summary.ToTestsuiteSummary())

def _mergeCTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPattern: str) -> None:
from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document, JUnitReaderMode
def _mergeCTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, foundFiles: Tuple[Path]) -> None:
from pyEDAA.Reports.Unittesting.JUnit.CTestJUnit import Document

foundFiles = [f for f in Path.cwd().glob(globPattern)]
self.WriteNormal(f"Reading {len(foundFiles)} CTest-JUnit unit test summary files ...")

junitDocuments: List[Document] = []
Expand All @@ -116,10 +121,9 @@ def _mergeCTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPattern
self.WriteVerbose(f" merging {summary.Path}")
testsuiteSummary.Merge(summary.ToTestsuiteSummary())

def _mergeGoogleTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPattern: str) -> None:
from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document, JUnitReaderMode
def _mergeGoogleTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, foundFiles: Tuple[Path]) -> None:
from pyEDAA.Reports.Unittesting.JUnit.GoogleTestJUnit import Document

foundFiles = [f for f in Path.cwd().glob(globPattern)]
self.WriteNormal(f"Reading {len(foundFiles)} GoogleTest-JUnit unit test summary files ...")

junitDocuments: List[Document] = []
Expand All @@ -132,14 +136,9 @@ def _mergeGoogleTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPa
self.WriteVerbose(f" merging {summary.Path}")
testsuiteSummary.Merge(summary.ToTestsuiteSummary())

def _mergePyTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, globPattern: str) -> None:
def _mergePyTestJUnit(self, testsuiteSummary: MergedTestsuiteSummary, foundFiles: Tuple[Path]) -> None:
from pyEDAA.Reports.Unittesting.JUnit.PyTestJUnit import Document

foundFiles = [f for f in Path.cwd().glob(globPattern)]
if len(foundFiles) == 0:
self.WriteWarning(f"Found no matching files for pattern '{Path.cwd()}/{globPattern}'")
return

self.WriteNormal(f"Reading {len(foundFiles)} pytest-JUnit unit test summary files ...")

junitDocuments: List[Document] = []
Expand Down Expand Up @@ -176,7 +175,25 @@ def _processPyTest(self, testsuiteSummary: TestsuiteSummary, cleanups: str) -> N

def _processPyTest_RewiteDunderInit(self, testsuiteSummary: TestsuiteSummary):
self.WriteVerbose(f" Rewriting '__init__' in classnames to actual Python package names")
self.WriteError("Rewrite __init__ not yet supported!")

def processTestsuite(suite: Testsuite) -> None:
testsuites: Tuple[Testsuite, ...] = tuple(ts for ts in suite.Testsuites.values())
for testsuite in testsuites: # type: Testsuite
if testsuite.Name != "__init__":
processTestsuite(testsuite)
continue

for ts in testsuite.Testsuites.values(): # type: Testsuite
ts._parent = None
suite.AddTestsuite(ts)

for tc in testsuite.Testcases.values(): # type: Testcase
tc._parent = None
suite.AddTestcase(tc)

del suite._testsuites["__init__"]

processTestsuite(testsuiteSummary)

def _processPyTest_ReduceDepth(self, testsuiteSummary: TestsuiteSummary, path: str):
self.WriteVerbose(f" Reducing path depth of testsuite '{path}'")
Expand Down
Loading

0 comments on commit ae5bf3d

Please sign in to comment.