Improve experience around scores

[LeaVerou]

A recurring point of feedback for State of HTML is that respondents get lower scores than they expected. Part of the reason is that it skews more cutting edge. But a bigger part is that mini-features only ask about usage, and there is no way to specify usage, so features you know about but haven't needed are not counted towards your score. Then at the end, participants get a score like "You have heard of or used X% of features", which is a little disingenuous if they have no way of specifying they know some of them!

Some ways to improve this experience are:

• Change score display, e.g. display separate awareness and usage scores, do not count usage-only questions towards awareness. Final score would be a weighed sum of the two (or simply the average).
  • Edit: Done. We switched to a points score rather than a percentage (each used feature counts 10 pts, each known feature counts 5)
• Show respondent rank, to establish that low scores don't mean low knoweldge.

[LeaVerou]

Prototype for 1:

Do note that the difference of this algorithm and the current one is quite staggering (and I think it will be even more so in the actual results).

I actually have a functional local implementation I could push, but I'm worried that while fair, it's also confusing ("Why are there two feature counts?"). I think this can be fixed with a wording change, but I can't think of how right now. Ideas welcome!

[LeaVerou]

It's ok if people miss it, it's an additional easter egg, not essential information. I'd rather not clutter the UI even more with a pie chart, and draw more attention to it than is warranted.

There's also the red=error association, so we should probably avoid that color?

I agree on the red=error association, but then we shouldn't be using that as an accent color in the first place? I only used the colors that were already used in the theme...

Maybe a pie chart showing used/known/unaware would work well here? I'm sure you can even do one with pure CSS ;)

The point of this visualization was to show the breakdown between known and used, e.g. whether the respondent is more of a "doer" (e.g. devs in the trenches) vs "knower" (e.g. web standards people).

I also think the tooltip is over-complicating things a bit. Don't forget any extra instructions we add means more cognitive load for users, more work for translators, etc.

I don't think a tooltip adds cognitive load, since for users to trigger it, they need to be actively wondering, so it's reducing cognitive load.

I think we can just do a simple (9/131 + 5/131)*100 calculation here.

(I assume you mean (9/131 + 5/42)*100, otherwise we end up with exactly the same score as today). You mean have this calculation on the tooltip? I think that would be even more confusing: where did the 42 come from?
If you just mean calculate the score using that formula, that's what's already happening (after this PR).
If the (9/131 + 5/131)*100 wasn't in error, then that would change nothing; the whole point of this change is to make the score more fair.

[SachaG]

I get that you want to make the score more "fair" and I appreciate the effort, but I think with this specific method it comes at the cost of making it a less straightforward metric, which I don't think is the right direction.

Another idea: what if instead we replaced the percentage by a points score? We could have something like known features = 10 points, heard about features = 5 points. And then in addition to that we show the pie chart with the percentages so we don't lose the current information.

And as long as we're rethinking this screen, it would be cool to incorporate somehow a way to actually show the features in each category (as we discussed somewhere I think), since that would give people one last chance to add more of them to their reading list.

And then optionally we can also leave some space for the rank somewhere, assuming we can make it work with a cronjob.

Not saying we have to do all of it right away, but if we're going to change things we might as well think about it in the larger context of how to make that final screen more useful to respondents.

[LeaVerou]

I like the points idea. No need to artificially inflate it though, the lowest increment should be 1 point. Perhaps 1 point for heard of, and 3 points for used? Or 1:2?

I think any changes to scoring need to be done ASAP or not at all. We can deliberate what other visualizations are optimal later, but if we change the score (and I think we should), we can't wait too long, so perhaps I can tweak the algorithm and we can merge the visualization that is already implemented in the PR?

[SachaG]

Personally I'm ok with the current scoring, I don't care that much if people get low scores. It's literally one of the least important priorities with the survey right now when compared with a11y, low contrast, outreach, and a dozen other concerns. So if you're ok postponing it until later that's definitely what I would do personally.

[LeaVerou]

Given we have a PR that's ready to merge with minor changes, I think it would be a waste not to, especially since it's very clear there's a problem. People getting low scores is discouraging, and unfair if they're penalized for not having heard of features they cannot say they've heard. Yes, other things are also high priority, and we're working on them too.

[SachaG]

Another thing to consider is how the new numerical score will affect this chart. To give people additional context we should probably calculate the maximum possible score and display it as well?

[LeaVerou]

It’s easy to calculate the maximum score: 131 * 10 = 1310

[SachaG]

Thanks for figuring that out for me, I don't think my MacBook is powerful enough to run this kind of processor-intensive calculations.