List of criteria for checking extensions for best practices (focus on testing) / results

[sypets]

As talked about with @oliverklee I would like to share my results so far, analyzing top 20 extensions and looking for commonly used best practices and differences in test setup.

List of criteria

general

1. does the extension have any tests / checks?
2. which tools are used (e.g. php-cs-fixer, see require-dev in composer.json)
   • which tools, additionally:
   • typo3/testing-framework or nimut/testing-framework
   • uses typo3/coding-standards?
3. Which tests exist and are executed? (e.g. phpstan)
4. how are tests executed: composer scripts vs. runTests.sh, ...?
5. git* pipeline CI exists?
6. how to run scripts locally? Is this documented somewhere (e.g. CONTRITING.md)
7. files and directory paths
   • .xml (unit / functional)
   • where is phpstan.neon (in root dir | Build | Build/phpstan | etc.)
   • where is php-cs-fixer config?
     ...

additional details

• has .gitattributes
• web-dir, bin dir, vendor dir
• has .editorconfig, contents of .editorconfig

---

Selection of extensions

• top 20 most liked v12 extension from https://extensions.typo3.org (as also used in Unify web-dir, vendor-dir, bin-dir in composer.json across extensions, documentation, etc #802)
• next time I would probably only select 1 extension for one author, e.g. use only 1 of lux and luxletter, only 1 of nn_helpers and nnrestapi
• language_detection and hcaptcha not considered, apart from that all verified extensions for v12 are considered (7)

Results of analysis

Extensions:

	Has tests / checks	has CI file	has runTests.sh	has composer scripts (for testing)	has Tests folder
sum	12	11	3	4	11

extension	Has tests / checks	has CI file	runTests or composer scripts	has Tests folder
bw_captcha	yes	yes	composer scripts	no
container	yes	yes	runTests.sh (has composer scripts but not for running tests)	yes
content_defender	yes	yes travis.yml	none	yes
content_slug	no	-	-	-
cs_seo	yes	no	none?	yes
go_maps_ext	no	-	-	-
lux	yes	yes	composer scripts	yes
luxletter	yes	yes	composer scripts	yes
mask	yes	yes	runTests.sh	yes
news	yes	yes	runTests.sh	yes
nnhelpers	no	-	-	-
nnrestapi	no	-	-	-
schema	yes	yes	Makefile	yes
sf_event_mgt	yes	yes	none	yes
sourceopt	yes	yes	composer scripts	yes
t3ext-image_autoresize	no	-	-	-
translate_locallang	no	-	-	-
TYPO3-Console	yes	yes	none (has composer scripts, but not for executing tests)	yes
typo3-dp_cookieconsent	no	-	-	-
ws_scss	no	-	-	-

Which (testing, check) tools are installed (require-dev):

• includes tools which measure effectiveness of tests, e.g. https://packagist.org/packages/infection/infection, coveralls

pkg	number of extension which use pkg (require-dev)	classification	description
phpunit/phpunit	11	phpunit	count implicit (via (nimut or typo3)/testing-framework, typo3/coding-standards) + explicit requirements
php-cs-fixer	11	CGL check+fix
typo3/testing-framework	8	phpunit
phpstan	6	static analyzer	count implicit + explicit requirements
typo3/coding-standard	4	CGL
phpmd/phpmd	3	static analyzer
codeception/codeception	2
helmich/typo3-typoscript-lint	2	static analysis
squizlabs/php_codesniffer	2	static analysis
nimut/testing-framework	2	phpunit
webmozart/assert	1
brotkrueml/coding-standards	1	CGL?
php-coveralls/php-coveralls	1	measure (test coverage)
rector/rector	1	static analysis check+fix
php-parallel-lint/php-parallel-lint	1	static analysis
tomasvotruba/cognitive-complexity	1	measure	measure cognitive complexity of your classes and methods
infection/infection	1	measure

additional (helper packages) phpstan:

• bnf/phpstan-psr-container
• saschaegerer/phpstan-typo3: 3x
• friendsoftypo3/phpstan-typo3: 1x
• phpstan/phpstan-phpunit
• phpstan/extension-installer: 2x
• symplify/phpstan-rules: 1x
• jangregor/phpstan-prophecy

additional (helper packages) codeception

• codeception/module-asserts
• codeception/module-db
• codeception/module-webdriver
• ericmartel/codeception-email

additional (helper packages) phpunit / testing-framework:

• sbuerk/typo3-cmscomposerinstallers-testingframework-bridge: 4x
• phpspec/prophecy: 2x
• phpspec/prophecy-phpunit
• dg/bypass-finals

peculiarities / detected problems

• phpstan-typo3: 2 packages exist: friendsoftypo3 + saschaegerer (mask uses friendsoftypo3/phpstan-typo3, see issue for mask: mask uses friendsoftypo3/phpstan-typo3 which is for core only (based on README) Gernott/mask#544)
• not all extensions which use typo3/testing-framework use sbuerk/typo3-cmscomposerinstallers-testingframework-bridge
• cs_seo: dev requirements can not be installed locally with composer install (conflicts)

[oliverklee]

Concerning PHPStan:

• friendsoftypo3/phpstan-typo3 is for the core only: ""
• saschaegerer/phpstan-typo3 is for extensions

Citing from https://github.com/FriendsOfTYPO3/phpstan-typo3#faq

I found this extension and the one from Sascha (saschaegerer/phpstan-typo3). Why are there two extensions and which should I use?

Well, this package has one specific purpose. It's made to help making the TYPO3 core phpstan max level compatible. To achieve this, the core team needs to be able to have its own extension which can be quickly adjusted as soon as the core itself changes. If for example, a new core version is released, the core team can quickly raise the dependency constraints for typo3/cms-core and typo3/cms-extbase which cannot be done when working with Sascha's package.

Also, Sascha's package contains dynamic return type providers that are not needed (yet) to make the core more compatible with phpstan.

To sum it all up: There is no competition between both extensions and this extension should not be used by users but only by the TYPO3 core.

[sypets]

I double-checked some values above and corrected it. This concerns mostly:

• extensions which have CI file. I overlooked it for content_defender and bw_captcha
• phpunit: number of extensions as results of all which include testing-framework or typo3/coding-standard (also cross-checked with number of extensions which contain the executable after composer install: is ok)
• phpstan: cross-checked with number of extensions which contain executable (after composer install): is ok
• php-cs-fixer: check for number of extensions which contain executable after composer install: 10 (+cs_seo which was not installable)

[brotkrueml]

Small addition to my schema extension (just to be accurate):
I am using ecs (which uses php-cs-fixer under the hood along with php-codesniffer), and instead of composer scripts I am using a Makefile (I don't like the idea to have non-PHP scripts in a composer.json file).

[sypets]

@brotkrueml Interesting. I will take a closer look and update the list.

[sypets]

On a side note: I personally really like Makefiles. The goal here is however to find some common ground to make recommendations. Not everyone will agree and not everyone will do it the same way. If it is easy contribute, easy to execute tests locally, add tests etc, that is ok, if not it makes contribution more difficult. In any case, the currently discussed things are merely recommendations.

[brotkrueml]

Yes, I understand the why. I added it because so we have a sane overview above.