Generify collections using Psalm #177

nschoellhorn · 2019-03-13T15:15:52Z

I've talked to @Ocramius and we find that generics/template types would make a very good addition to this library. Since PHP doesn't come with that feature natively, I have decided to implement it using Psalm and Psalm's template annotations.

How is that useful?
A lot of people already use static analysis to reduce the risk of introducing bugs in your application. Psalm is one of the most popular tools out there to do that. By adding the annotations to Psalm and using the correct @psalm-var annotations when using the collections in projects, Psalm makes sure that you only put values of the right type in your collection automatically, since it also scans the vendor directory.
I have created a very minimal proof of concept project over at nschoellhorn/psalm-collections-test. You can clone that, install the dependencies and run ./vendor/bin/psalm to check it out.

Ocramius · 2019-03-13T15:19:21Z

Huge 👍 from me on this.

This is meta-code that provides immediate value for downstream consumers that use Psalm, without any runtime being affected. The only possible issues that might arise are with doctrine/annotations and people that are (erroneously) trying to parse them from vendor/

Ocramius · 2019-03-13T15:21:12Z

To be checked: scrutinizer-ci failure (can't run composer install)

nschoellhorn · 2019-03-13T16:38:12Z

Needed to bump the PHP version to allow the installation of Psalm to complete. Should be no issue though, since it went from 7.1.0 only up to 7.1.3

Ocramius · 2019-03-13T17:51:27Z

@muglug @weirdan can you glance over this, if you have some time?

weirdan · 2019-03-13T18:35:05Z

@Ocramius sure, will do.

weirdan

Overall, it might make sense to use two generic parameters for Collection: Collection<TKey,TVal>, so that people may narrow down the key type further:

/** @psalm-var Collection<string,string> $coll */
$coll = new ArrayCollection;
$coll->containsKey(42); // then Psalm will be able to flag this as invalid

lib/Doctrine/Common/Collections/Collection.php

psalm.xml.dist

lib/Doctrine/Common/Collections/Criteria.php

lib/Doctrine/Common/Collections/AbstractLazyCollection.php

lib/Doctrine/Common/Collections/Collection.php

muglug

Looks great!

lib/Doctrine/Common/Collections/Collection.php

muglug · 2019-03-13T20:35:28Z

lib/Doctrine/Common/Collections/Collection.php

@@ -170,6 +198,8 @@ public function next();
     * @param Closure $p The predicate.
     *
     * @return bool TRUE if the predicate is TRUE for at least one element, FALSE otherwise.
+     *
+     * @psalm-param Closure(int|string, T):bool $p


@psalm-param Closure(int|string=, T=):bool $p should have the same effect I think?

lib/Doctrine/Common/Collections/Collection.php

muglug · 2019-03-13T20:37:06Z

lib/Doctrine/Common/Collections/Collection.php

@@ -209,6 +248,9 @@ public function map(Closure $func);
     * @return Collection[] An array with two elements. The first element contains the collection
     *                      of elements where the predicate returned TRUE, the second element
     *                      contains the collection of elements where the predicate returned FALSE.
+     *
+     * @psalm-param Closure(int|string, T):bool $p
+     * @psalm-return Collection<T>[]


You can also use @psalm-return array{0: Collection<T>, 1: Collection<T>} for explicitness

lib/Doctrine/Common/Collections/Criteria.php

Accepted as a sign I agree with this. Some work still left to do though

nschoellhorn · 2019-03-13T21:04:47Z

Thank you for the review guys, especially @weirdan for all the suggestions. Going through them atm, but I think we should agree on whether to use array-key or an additional template parameter for the collection key? I am currently more in favor of array-key since it comes closer to Java's Collection API, which I am generally a fan of. But that's very subjective. What do you guys think?

Found while reviewing doctrine/collections#177 Also: - Limited key type to array-key - Added future tests, pending psalm implementation (see vimeo/psalm#1462)

Ocramius · 2019-03-13T21:12:44Z

or an additional template parameter for the collection key?

The key can be int|string, and should be a template parameter, because the possible values for it are int, string and int|string, as far as I know.

weirdan · 2019-03-13T21:29:05Z

@muglug if all those ArrayCollection docblocks were replaced with {@inheritDoc}, would Psalm inherit the signatures from Collection interface?

weirdan · 2019-03-13T23:38:46Z

@nschoellhorn You will need to adjust ArrayCollection signatures as well (things like first, next, etc).

alcaeus

👍 I'm in favour of this. We should try to allow our users to make full use of the static analysis tools they provide. Since psalm provides functionality around generics, this makes perfect sense. Once phpstan supports something like this, we should add that as well. Of course, I'd prefer it if @muglug and @ondrejmirtes could come up with an annotation that doesn't include a tool prefix, but that's for them to settle, not us.

Last but not least, checking our code with phpstan and psalm is also a plus as there are some things that are not detected in both tools, so this should improve the code quality as well.

ondrejmirtes · 2019-03-14T08:38:33Z

could come up with an annotation that doesn't include a tool prefix

Yeah, I'm usually more aggressive about annotation adoption, I'm not worried about using new syntax in at-param, at-var etc. Thanks to shoving intersection types there, they are currently supported by PhpStorm and other tools, which was my goal (driving innovation).

nschoellhorn · 2019-03-14T09:14:17Z

@muglug if all those ArrayCollection docblocks were replaced with {@inheritDoc}, would Psalm inherit the signatures from Collection interface?

Sadly, Psalm does not seem to catch that. I tried that with my test repository that I linked above and when running Psalm after removing the annotations from ArrayCollection::add(), it doesn't complain anymore. So I think we have to keep the annotations in there.

@nschoellhorn You will need to adjust ArrayCollection signatures as well (things like first, next, etc).

Yeah, still at it. Going for an additional template parameter TKey as array-key now, so people can either make a collection of type Collection<array-key, something> if they don't care about the exact type of their key or they can restrict to either int or string if they like.

lib/Doctrine/Common/Collections/ArrayCollection.php

ostrolucky · 2019-03-14T09:29:18Z

Sadly, Psalm does not seem to catch that. So I think we have to keep the annotations in there.

While I'm all in for adding psalm specific notations here (since there doesn't exist any standard for this so far), I'm not in favor of incorporating psalm bug specific workarounds. This is something that should be fixed by upstream (psalm), not us having to copy these to every subclass.

Ocramius · 2019-03-14T11:37:58Z

I'm not in favor of incorporating psalm bug specific workarounds.

Must back this too: doctrine/collections, being very central to the PHP ecosystem, should not have workarounds, as the types do indeed propagate statically, which will affect downstream consumers. Therefore, while this patch can indeed be polished and finished up, we'll need to address the upstream issue in psalm before merging.

nschoellhorn · 2019-03-14T12:01:37Z

I'd also prefer that. Maybe @muglug can check if there's something that I've done wrong, causing the inheritance not to work properly. If it's not my fault, I'd offer submitting an issue to Psalm.

Ocramius · 2019-03-14T14:34:48Z

Well, that seems to be fixed: https://github.com/vimeo/psalm/releases/tag/3.2

muglug · 2019-03-14T14:43:25Z

Yup, thanks for letting me know! Templated inheritance is non-trivial. I already had it working for templated return types, but had neglected templated param types.

weirdan

I think it'll be good to merge after this round of suggestions.

lib/Doctrine/Common/Collections/Collection.php

lib/Doctrine/Common/Collections/ArrayCollection.php

lib/Doctrine/Common/Collections/AbstractLazyCollection.php

nschoellhorn · 2019-03-14T18:33:28Z

After another round of improvements (thanks @weirdan), we are now only missing the discussion on the (possibly) optional parameters of the closures for map(), filter(), etc.

weirdan

Collection::partition() should probably be able to accept argless closures. Other than that this looks fine.

lib/Doctrine/Common/Collections/Collection.php

Signed-off-by: Niklas Schoellhorn <schoellhorn.niklas@gmail.com>

nschoellhorn · 2019-03-18T13:09:48Z

Thank you @muglug for the fast update once again. I've bumped the dependency of Psalm to the new version and rolled back the change regarding uasort(). No issues anymore.

Ocramius

@doctrine/doctrinecore can I have a vote on this, please?

SenseException

Looks neato.

If some psalm annotations change in the classes/interfaces, would this break psalm checks of other projects using Collections?

Ocramius · 2019-03-18T16:08:42Z

Yes, type checks propagate statically

SenseException · 2019-03-18T16:17:44Z

Should we have some kind of tests that prevent logical changes in the psalm annotations of doctrine/collections?

Ocramius · 2019-03-18T16:19:14Z

That would be like roave/backwards-compatibility-check: the ecosystem doesn't have that kind of stuff yet (even BC checker is way too new)

weirdan · 2019-03-18T20:01:19Z

Should we have some kind of tests that prevent logical changes in the psalm annotations of doctrine/collections?

You can take a look at https://github.com/weirdan/doctrine-psalm-plugin/blob/master/tests/acceptance/Collections.feature - most of those tests would apply here as well. Feel free to borrow them.

nschoellhorn · 2019-03-20T07:50:31Z

Looks good, I can add that if wanted.

Ocramius · 2019-03-20T07:54:44Z

Let's please defer that for now

Ocramius · 2019-03-20T07:55:11Z

Or better: separate patch (based on this one)

nschoellhorn · 2019-03-20T16:36:10Z

Sure, I will prepare one soon then.

Ocramius · 2019-03-24T17:33:32Z

Right, calling the merge here 🚢

Ocramius added the Enhancement label Mar 13, 2019

ostrolucky previously approved these changes Mar 13, 2019

View reviewed changes

weirdan suggested changes Mar 13, 2019

View reviewed changes

weirdan reviewed Mar 13, 2019

View reviewed changes

lib/Doctrine/Common/Collections/Collection.php Outdated Show resolved Hide resolved

muglug previously approved these changes Mar 13, 2019

View reviewed changes

nschoellhorn dismissed muglug’s stale review via 5ba82bc March 13, 2019 21:25

alcaeus reviewed Mar 14, 2019

View reviewed changes

lib/Doctrine/Common/Collections/ArrayCollection.php Show resolved Hide resolved

nschoellhorn changed the title ~~Generify collections using Psalm~~ [WIP] Generify collections using Psalm Mar 14, 2019

weirdan suggested changes Mar 14, 2019

View reviewed changes

lib/Doctrine/Common/Collections/Collection.php Outdated Show resolved Hide resolved

lib/Doctrine/Common/Collections/Collection.php Outdated Show resolved Hide resolved

Merge dependency changes

067a137

Signed-off-by: Niklas Schoellhorn <schoellhorn.niklas@gmail.com>

nschoellhorn force-pushed the master branch from dc5004b to 067a137 Compare March 18, 2019 12:28

ostrolucky approved these changes Mar 18, 2019

View reviewed changes

malarzm approved these changes Mar 18, 2019

View reviewed changes

Ocramius approved these changes Mar 18, 2019

View reviewed changes

SenseException approved these changes Mar 18, 2019

View reviewed changes

alcaeus approved these changes Mar 18, 2019

View reviewed changes

alcaeus requested a review from Majkl578 March 18, 2019 19:07

Ocramius merged commit d348a61 into doctrine:master Mar 24, 2019

Ocramius added this to the 1.6.0 milestone Mar 24, 2019

Ocramius self-assigned this Mar 24, 2019

Ocramius mentioned this pull request Mar 24, 2019

V1.6.0 throws errors in Symfony context #180

Closed

kdambekalns mentioned this pull request Mar 25, 2019

Latest Doctrine Collections break reflection (template / psalm annotation) neos/flow-development-collection#1529

Closed

Ocramius mentioned this pull request May 27, 2019

Built-in assertion and mock type definitions sebastianbergmann/phpunit#3708

Closed

10 tasks

Ocramius mentioned this pull request Jun 7, 2019

WIP: psalm-specific annotations for restricting post-assertion value types webmozarts/assert#102

Closed

Ocramius mentioned this pull request Aug 24, 2019

Allow custom collections doctrine/orm#6667

Open

alcaeus mentioned this pull request Dec 6, 2019

Add support for generics via static analysis doctrine/persistence#72

Closed

greg0ire mentioned this pull request Aug 24, 2022

Mark closure arguments as always-provided #323

Merged

weirdan mentioned this pull request Feb 15, 2023

doctrine/collections phpstubs outdated/obsolete psalm/psalm-plugin-doctrine#138

Open

Generify collections using Psalm #177

Generify collections using Psalm #177

Conversation

nschoellhorn commented Mar 13, 2019

Ocramius commented Mar 13, 2019

Ocramius commented Mar 13, 2019

nschoellhorn commented Mar 13, 2019

Ocramius commented Mar 13, 2019

weirdan commented Mar 13, 2019

weirdan left a comment

Choose a reason for hiding this comment

muglug left a comment

Choose a reason for hiding this comment

muglug Mar 13, 2019

Choose a reason for hiding this comment

muglug Mar 13, 2019

Choose a reason for hiding this comment

nschoellhorn commented Mar 13, 2019

Ocramius commented Mar 13, 2019

weirdan commented Mar 13, 2019

weirdan commented Mar 13, 2019

alcaeus left a comment

Choose a reason for hiding this comment

ondrejmirtes commented Mar 14, 2019

nschoellhorn commented Mar 14, 2019

ostrolucky commented Mar 14, 2019 • edited Loading

Ocramius commented Mar 14, 2019

nschoellhorn commented Mar 14, 2019

Ocramius commented Mar 14, 2019

muglug commented Mar 14, 2019

weirdan left a comment

Choose a reason for hiding this comment

nschoellhorn commented Mar 14, 2019

weirdan left a comment

Choose a reason for hiding this comment

nschoellhorn commented Mar 18, 2019

Ocramius left a comment

Choose a reason for hiding this comment

SenseException left a comment

Choose a reason for hiding this comment

Ocramius commented Mar 18, 2019

SenseException commented Mar 18, 2019

Ocramius commented Mar 18, 2019

weirdan commented Mar 18, 2019

nschoellhorn commented Mar 20, 2019

Ocramius commented Mar 20, 2019

Ocramius commented Mar 20, 2019

nschoellhorn commented Mar 20, 2019

Ocramius commented Mar 24, 2019

ostrolucky commented Mar 14, 2019 •

edited

Loading