Improvements and tweaks to the package score

[daveverwer]

Enhancing package score comes up every now and then, most recently in this thread where @Sherlouk talks about the license being an indicator:

For instance, I personally think too much weight is given to an "App Store Compatible" license which for many reasons may not be picked up, and indeed something I do value (cross-platform compatibility) is not currently taken into account.

I'd love to gather more thoughts on this, and a thread here seems like a good place to do that to keep the forums thread focused as much as possible on the Packages page.

Also, @cyndichin is working in this area over the next few weeks as part of the Swift Mentorship Programme, and this discussion will be helpful to that project too.

[Sherlouk]

Link to score calculation: https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/blob/main/Sources/App/Core/Score.swift

I do want to preface this entire post by saying I don't think that the scoring system is fundamentally flawed, and I am not strong in these feelings, but the above abstract from the thread emphasises that there is disparity on what each person would describe as a "good package" and thus what SPI ranks highly currently and what I might rank highly can sometimes be different.

Dependencies are inherently a personalised decision, the scoring needs to act as the most balanced decision for the average user and project.

And also, this is just my opinion.

License Compatibility

We currently give 10 points to any package which has a recognised license, and one which we deem suitable for the app store. We give 3 points to a recognised license but one that is incompatible with the app store, 0 points if you use a custom or altered license, and 0 points if you don't have a license at all.

Personally I don't think this balance is right.

Simply look at the results of this search (https://swiftpackageindex.com/search?query=license%3Aother) and see just how many popular and well-used packages have no license and are being kneecapped in our rankings.

License is important, and I think most people need a lesson in that, but I think the scoring on this right now lacks nuance. Not everybody is releasing to the App Store. Not every missing license is a complete red flag (sometimes just needs an extra check).

Personally, I'd remove the license check.

Platform Compatibility

Apple continues to encourage products that support multiple platforms across it's ecosystem. Finding dependencies that allow this can be a challenge, and I think SPI could benefit from providing a boost to packages which support more than just one platform.

For me when I'm searching for Logging packages and see that it supports every Apple platform except watchOS, I begin to question what weird APIs are they using that it prevents a simple logging capability? To me, for that particular search, it's a sign of smell. In fact I regularly use Linux as a method of identifying packages without complex dependencies (usually going to cause me less headaches).

I don't think this is an easy one to solve, but I would love to see bonus points for supporting all Apple platforms. Support for Linux/Android/Windows is a very specific use case and so (while valuable to me) doesn't fit into the "balanced user" I prefaced above.

Other Tidbits

We currently add 10 points if you have more than 5 releases, 0 points if you don't hit that threshold. I think there's a big difference between a package with 0 releases (which actually makes it harder to add to your project) vs a package with only one or two releases. I think we should -3 points if you haven't marked a release.

We currently dock points for having more than 5 dependencies, but this includes test dependencies. I understand it's hard to tell the difference right now, but should I be penalised in the rankings before I use SnapshotTesting or Quick/Nimble in my unit tests?

I love the open PR which looks to give extra points for packages with tests, this shows an effort towards maintaining a level of quality. Though as with many other checks it doesn't always pass the sniff test - just because I have a test target doesn't actually mean I have any tests. The same is true for documentation which I love and value, but is it good documentation? Is it complete documentation? How can we prove this? Is it even possible?

Is there a way to automatically review/evaluate the quality of the README? Does it include helpful information to make an informed decision, and to ease the installation process? Could we penalise packages that only have the SwiftPM template README? or packages with no README at all?

---

This isn't something we get right overnight, but I hope this brain dump does potentially add some value to the conversation. Feel free to add counterpoints and discuss below! Again this is only an opinion.

[maxxfrazer]

I think this is a great step for the swift package index to make this score more visible, and agree with most of @Sherlouk's comments above.

Drawing parallels with other ecosystems, I find the scoring system of pub.dev to be pretty good. They encompass several key checks, some of which could serve as inspiration here:

• Absence of a README
• OSI-approved license
• % of public APIs that have doc comments (20+% yields full points)
• Platform support (no deductions for missing linux)
• Static analysis that looks out for errors, warnings, and issues related to linting or formatting
• Up-to-date dependencies

https://pub.dev/packages/routemaster/score

Not all of these would be feasible for SPI of course.

Adding lint/formatting checkers would be great idea. A choice between SwiftLint and SwiftFormat though .spi.yml may be necessary to offer flexibility. However, since linters aren't widely adopted among developers in the current ecosystem (though I believe they should be), they might carry a relatively small weight in terms of points.

[timsneath]

Having been part of the Dart/Flutter team that worked on scoring with pub.dev, we learned some useful things over time that drove what you currently see here.

I think a lot comes down to whether the measures feel arbitrary or not. A few learnings from pub.dev:

1. We should never penalize packages (no negative points): positive reinforcement feels better for everyone
2. We should focus on awarding points for best practices, focusing exclusively on the package rather than its popularity or development process. For example, is it well documented? Does it support the latest releases? It doesn't need to be popular to get a perfect score -- since a great package may serve a specialist niche
3. Stable is good -- a package doesn't have to ship every month or quarter to be well-maintained. So long as it works well with the latest builds, it doesn't have to churn for the sake of it.
4. It's OK to add unscored items to the checklist. When we first introduced null safety in mainline builds, we gave a checkmark for null-safe packages but didn't add points. I believe that later we started to add scores for it, as it became more clearly a requirement of a well-maintained package.
5. For the top ~100 packages, we introduced a concept of "Flutter Favorites" for packages that had additionally been through an editorial review that focused on qualitative measures. Such packages get an extra "seal of approval".

Hope this helps!

[dempseyatgithub]

I just added a package to the index for the first time, so these are my thoughts in encountering the packing scoring system for the first time:

For each category:

• Is there a maximum number of points for that category?
• Does my package have the maximum number of points in that category?
• What can I do to increase my score in that category?

For the package as a whole:

• Is there a maximum score?
• How does my score fit in compared to the range of package scores?
• How does my score fit in as a percentile of other packages?

And then some questions about a couple of the categories:

For the Contributors category, it doesn't seem that a package that has many contributors is inherently better than one with a small number of contributors.

I think a single person or two could create an excellent, focused package where they stay on top of bug fixes and enhancements, but it ends up with a lower score because it really doesn't need other contributors.

It's also not clear know how the Dependencies category works.

Does the score go up or down when the number of dependencies goes up?
(Personally, I prefer packages with the least number of dependencies. )

[daveverwer]

I want to do more with package scoring, including better representing how we display score information, so your comments are valuable and I’ll bear all of them in mind as I do that design.

That said, I’m sure you’d also like to know the answers to some of the questions 😂 so here goes!

• Is there a maximum number of points for that category?

Yes. It’s different per category, too, based on how important we think that category is. We also adjust these as we provide more information behind the score.

For example, the total number of points available for having a small number of dependencies is 5 points total because there are some limitations with how we gather that data right now. As we get better data and can exclude test-only dependencies, etc… we will expand the total points for this catefgory.

• Is there a maximum score?

There is, but again it’s arbitrary and increases over time. We would likely normalise it if we ever display this more publicly than it is right now.

For the Contributors category, it doesn't seem that a package that has many contributors is inherently better than one with a small number of contributors.

I think a single person or two could create an excellent, focused package where they stay on top of bug fixes and enhancements, but it ends up with a lower score because it really doesn't need other contributors.

I’d push back on this a little and argue that having multiple contributors is a sign of a couple of things:

1. That the package gained some level of popularity that people became aware of it. This takes care of the inevitable “fixed typo” PRs that popular packages get!
2. That once people became aware of it, that it was useful enough to prompt changes based on user needs.

However, in a similar vein to what I mentioned with dependencies, the number of points in this category represents its importance, and the maximum score for this category is 5 points less than a package gets for having a README file.

Does the score go up or down when the number of dependencies goes up?
(Personally, I prefer packages with the least number of dependencies. )

You’re correct, this is exactly how it works.