Event System for extending PHPUnit

[sebastianbergmann]

History and Background

The TestListener Interface

First, there was the PHPUnit\Framework\TestListener interface:

interface TestListener
{
    public function addError(Test $test, Throwable $t, float $time): void;

    public function addWarning(Test $test, Warning $e, float $time): void;

    public function addFailure(Test $test, AssertionFailedError $e, float $time): void;

    public function addIncompleteTest(Test $test, Throwable $t, float $time): void;

    public function addRiskyTest(Test $test, Throwable $t, float $time): void;

    public function addSkippedTest(Test $test, Throwable $t, float $time): void;

    public function startTestSuite(TestSuite $suite): void;

    public function endTestSuite(TestSuite $suite): void;

    public function startTest(Test $test): void;

    public function endTest(Test $test, float $time): void;
}

The TestListener interface violates the Interface Segregation Principle, the "I" in "SOLID". This means that it requires the implementation of many methods, even if the client is not interested in the events they represent.

But the TestListener interface also has a fundamental design flaw: it passes around the Test and TestSuite objects, allowing clients to manipulate the outcome of a test run, for instance. To make things worse, client implementations of TestListener exist that bypass the public API of Test and TestSuite to perform such manipulation. In short, TestListener implementations were use to do more than "just listen".

The TestListener interface is deprecated since PHPUnit 8, it will be removed in PHPUnit 10.

The Hook Interfaces

Back in 2018 and with PHPUnit 7.1, we attempted to make extending PHPUnit's test runner easier with the introduction of the PHPUnit\Runner\Hook interfaces. Here is an example of one of these interfaces:

interface AfterSuccessfulTestHook
{
    public function executeAfterSuccessfulTest(string $test, float $time): void;
}

As you can see, we learned from the painful experience we made with the TestListener interface. The PHPUnit\Runner\Hook interfaces follow the Interface Segregation Principle and clients no longer get access to the real test objects that are used by PHPUnit to run the tests.

However, we did not "think big enough" and the PHPUnit\Runner\Hook interfaces were too limited. Even we did not manage to migrate PHPUnit's own TestListener implementations to the PHPUnit\Runner\Hook interfaces. They were a step in the right direction, but they fell short of actually being useful.

The PHPUnit\Runner\Hook interfaces will be removed in PHPUnit 11.

The New Event System

At the EU-FOSSA 2 Hackathon in October 2019, Sebastian Bergmann, Ewout Pieter den Ouden, Andreas Möller, Arne Blankerts, and Stefan Priebsch got together and designed a new system for extending PHPUnit based on events.

---

Photo: Arne Blankerts, Andreas Möller, and Ewout Pieter den Ouden work on the new event system for PHPUnit

---

Since then, Arne Blankerts and Andreas Möller worked on implementing this new event-based system in a branch. This issue is created as we are getting close to be able to merge this branch.

[Naktibalda]

Hi Sebastian,

Factory methods for value objects of TestSuite and TestCase classes make TestCase and TestSuite non-extensible.

phpunit/src/Event/Value/TestSuite/TestSuite.php

Lines 55 to 145 in 1c56cf3

	public static function fromTestSuite(FrameworkTestSuite $testSuite): self
	{
	$groups = [];
	foreach ($testSuite->getGroupDetails() as $groupName => $tests) {
	if (!isset($groups[$groupName])) {
	$groups[$groupName] = [];
	}
	foreach ($tests as $test) {
	$groups[$groupName][] = get_class($test);
	}
	}
	$tests = [];
	foreach ($testSuite->tests() as $test) {
	if ($test instanceof TestCase || $test instanceof PhptTestCase) {
	$tests[] = $test->valueObjectForEvents();
	}
	}
	if ($testSuite instanceof DataProviderTestSuite) {
	[$className, $methodName] = explode('::', $testSuite->getName());
	try {
	$reflector = new ReflectionMethod($className, $methodName);
	return new TestSuiteForTestMethodWithDataProvider(
	$testSuite->getName(),
	$testSuite->count(),
	$groups,
	$testSuite->provides(),
	$testSuite->requires(),
	$testSuite->sortId(),
	TestCollection::fromArray($tests),
	$testSuite->warnings(),
	$className,
	$methodName,
	$reflector->getFileName(),
	$reflector->getStartLine(),
	);
	// @codeCoverageIgnoreStart
	} catch (ReflectionException $e) {
	throw new Exception(
	$e->getMessage(),
	(int) $e->getCode(),
	$e
	);
	}
	// @codeCoverageIgnoreEnd
	}
	if (class_exists($testSuite->getName())) {
	try {
	$reflector = new ReflectionClass($testSuite->getName());
	return new TestSuiteForTestClass(
	$testSuite->getName(),
	$testSuite->count(),
	$groups,
	$testSuite->provides(),
	$testSuite->requires(),
	$testSuite->sortId(),
	TestCollection::fromArray($tests),
	$testSuite->warnings(),
	$reflector->getFileName(),
	$reflector->getStartLine(),
	);
	// @codeCoverageIgnoreStart
	} catch (ReflectionException $e) {
	throw new Exception(
	$e->getMessage(),
	(int) $e->getCode(),
	$e
	);
	}
	// @codeCoverageIgnoreEnd
	}
	return new TestSuiteWithName(
	$testSuite->getName(),
	$testSuite->count(),
	$groups,
	$testSuite->provides(),
	$testSuite->requires(),
	$testSuite->sortId(),
	TestCollection::fromArray($tests),
	$testSuite->warnings(),
	);
	}

phpunit/src/Event/Value/Test/TestMethod.php

Lines 43 to 67 in 1c56cf3

	public static function fromTestCase(TestCase $testCase): self
	{
	$className = $testCase::class;
	$methodName = $testCase->getName(false);
	$testData = self::dataFor($testCase);
	if ($testCase instanceof ErrorTestCase ||
	$testCase instanceof IncompleteTestCase ||
	$testCase instanceof SkippedTestCase ||
	$testCase instanceof WarningTestCase) {
	$className = $testCase->className();
	$methodName = $testCase->methodName();
	}
	$location = self::sourceLocationFor($className, $methodName);
	return new self(
	$className,
	$methodName,
	$location['file'],
	$location['line'],
	self::metadataFor($className, $methodName),
	$testData,
	);
	}

Would it be possible to implement mechanism for extending class to provide factory method for value objects?

[sebastianbergmann]

Please explain why you want to extend these classes.

[top-master]

No matter if hook, or event.

I don't think that any of this can help for situations like #4960

And should not be used as an excuse to close feature-requests.

[Jean85]

@sebastianbergmann this closing means that the event system is now stable? I saw that you heavily refactored it, creating a distinction between outcomes and other events (good choice!), is that permanent now? Can we start building on top of that?

[sebastianbergmann]

I have closed this issue because the initial work has been completed by @theseer and @localheinz a long time ago and their work has been merged a long time ago. Since then, with the exception of #5040, everything in PHPUnit itself has been migrated from TestListener to the event system.

I am reluctant to call the current situation "stable", though, and cannot guarantee that there will be no major changes until we approach a release.

[BafS]

Shall we still use hooks in the meantime? I didn't find documentation regarding the new event system, if I understand correctly

• TestListener is deprecated and will be remote in PHPUnit 10
• Hook will be deprecated in PHPUnit 10 and removed in PHPUnit 11
• The new event system is not stable and not documented (?)

It's a bit annoying to build something new when we know it is deprecated but it seems that is the way to go

[sebastianbergmann]

The TestListener interface as well as the Hook interfaces have been removed in PHPUnit 10. The new system will be documented before PHPUnit 10 is released.

In the meantime, you can look at, for instance, these examples:

• default progress printer
• JUnit XML logfile writer
• TeamCity logfile writer

[BafS]

Okay, thank you for those information!

[arderyp]

@sebastianbergmann you mentioned that the new event system would be documented before 10 was released.

I see some documentation here: https://phpunit.readthedocs.io/en/10.0/extending-phpunit.html#phpunit-s-event-system

However, it seems to be missing the basic example of how to use these new events.

How does one refactor:

use PHPUnit\Runner\BeforeFirstTestHook;
use PHPUnit\Runner\AfterLastTestHook;

class MyCustomeExtension implements BeforeFirstTestHook, AfterLastTestHook
{

I'm guessing it has something to do with:

PHPUnit\Event\Test\AfterLastTestMethodCalled;
PHPUnit\Event\Test\BeforeFirstTestMethodCalled;

But the implementation is not clear to me. They don't look like attributes... so?

[sebastianbergmann]

The documentation has not been updated for PHPUnit 10 yet. The JUnit logger is a good starting point to see how events are used.

[arderyp]

okay, i'll have a look at JUnit logger

EDIT: this write-up is very helpful, for anyone else who sees this: https://localheinz.com/articles/2023/02/14/extending-phpunit-with-its-new-event-system/

[bilogic]

I think it is safe to say that this is a topic for 90% of us because of removed functionality that we relied upon. This will probably help too https://docs.phpunit.de/en/10.3/extending-phpunit.html#extending-the-test-runner

[oleg-andreyev]

As for me the biggest issue with New Event System, is that event is immutable and I cannot alter results.

Example:

• We have huge test suite
• I want to parallel it using amqp
• I want to send tests into queue
• I want to consume test from queue

and fortunately or unfortunately... it's impossible.
So need to override entire TestRunner...

[Jean85]

@oleg-andreyev what you probably want to try is Paraunit (mine) or Paratest.