Learning more about Web Components needs #13
Comments
I assume this would run on JS & API pages?
This is a 2 questions survey :)
Would a responder still get question 2 if they pick this answer for question 1?
These 2 are kind of orthogonal to the rest - I'm not sure the survey infrastructure allows to deal with them differently (e.g. if the order of options is random, these 2 ideally would always be first or last of the list). Regarding the list of actual options - I've done some limited Web components development, and most of the names of the features feel obscure to me; if that's the case for most respondents, we risk detecting recognition more than need. Maybe this should be instead a list of X features, each of which comes with "I need this", "I don't need this", "I don't know what this is" ratings? |
That would match the short survey, yeah. But maybe it'd make sense to target it even narrower to just things that are related to web components, which would also include some HTML pages. That might be tricky to do because it's a lot of URLs though.
Fixed.
No, this question can end the survey.
Good point about randomization. This was drafted as a 1-question survey but now that we added a first filtering question, maybe we don't need this one. @mfreed7 WDYT? |
Yeah, I think with the new 2-question format (which I like!) I think we can probably remove both of these. As to the comment that this will be a recognition survey for some folks, that's probably correct. But maybe that's also useful - a problem people have heard of is likely worse than one they've never heard of. Having said that, I think the suggestion to change to "I need", " I don't need", and "don't know" for each item is interesting. I have two concerns: 1) it's a much longer survey that way, and we might get fewer responses, and 2) I would assume that would give us less of a ranking signal than the choose-5 approach. WDYT? Also, ping @Westbrook - would you and the WC CG like to take a look at the survey questions here and see if anything is missing or should be tweaked? The items were shamelessly stolen from the last WC-CG report. |
|
Another option here would be to do something totally exploratory: for people who have experience using web components give them a free-form text field to describe the biggest pain points that they're experiencing, rather than trying to create a long list of features that they might not be able to map to their experiences. |
|
I’m biased through lots of use of web components and lots of “I don’t need that {insert API name}, until I do need {insert API capability}” conversations with people who have “heard that web components are no good”. Is there a productive way to separate capability from API name? Or to maybe do a named pass and a feature pass? Not sure how to support getting quality signal to noise ratios on this part of the web API. I’ve shared this issue back to the WCCG and will take a closer look at the proposed questions, soon! |
While I like the idea of this suggestion, I'm afraid of two things: 1) much less participation, and 2) less actionable feedback. But perhaps I'm wrong on both counts.
I 100% agree that trying to separate the functionality from the name is a good idea. That's another thing I'm afraid of if we go the freeform-text route: comments like "web components are awful" without context or rationale. One other point: I think this is mainly an issue for the name "Web Components" overall, and perhaps less so once you step into a specific list of things (e.g. "Children changed callback", "Composed Selection", etc.). Would you agree?
Awesome, thanks! |
This might be tricky, reducing scope to just the Web Components area and HTML will mean that relatively few people will see it. In that case we might want to compensate by either running it much longer or showing it to a higher percentage of visitors. |
|
@Westbrook any further thoughts?
Right, I agree. So it sounds like the plan is to scope to JS and API pages, but not further scoped than that. One question: can we add links to the survey? I.e. we have a big list of things, and people might think "oh, I think I know what that is, but let me check". It'd be nice to have a link there for them. But then it might be an overwhelming list of links? So at this point we're here, right? Survey prompt before expanding: Browser vendors are working together to improve support across browsers for Web Components. Shape the future of Web Components by taking this 2-question survey! Question 1: What's your experience with Web Components (custom elements, Shadow DOM, etc.)?
Question 2: For your needs as a developer, which Web Components features should be improved across browsers in the coming year? This means enabling support or fixing bugs. (This is not a complete list, other features may also be planned.) Each of these have these three options: "I need this", " I don't need this", or "don't care / don't know" (the default)
|
|
@Westbrook just wanted to check whether there was any feedback from the CG? |
|
@Westbrook gentle ping. Otherwise, any objections to the final wording of the question in this comment? I'm happy with it. What's next, in order to get this published/started? |
|
Thank you @mfreed7. I've posted this to the short survey repo for consideration by the MDN team. If there are any strong objections to the survey as defined above, please note it here, otherwise we'll go ahead. |
Great, thanks! |
|
I wonder if we should also include HTML in the targeting? I haven't checked the visitor counts, but these pages under HTML are related to Web Components: The same argument can be made for CSS with https://developer.mozilla.org/en-US/docs/Web/CSS/::part, but notably nothing under JS is directly related to Web Components. @mfreed7 what's your hunch, what would be the best targeting? |
Certainly the pages you listed would be directly related to web components, but there are many more. E.g. it’s not under JS, but this would be a good one: https://developer.mozilla.org/en-US/docs/Web/API/Element/attachShadow I’m beginning to wonder if it might be better to display the survey more broadly (like any page under “/Web/“) and deal with a larger percent of people bailing at question 1? |
|
Adding it additionally on only the listed pages is barely worth the effort. With the low visitor count of the pages and a 5% sampling, expect a handful of additional survey takers over the course of a a week. I'd suggest just adding it to /web in that case. Will suggest to the MDN team. |
|
Agreed with running it under /web - it'd be difficult to pick out components specific pages as a feature which crosses MDNs taxonomy... something else feature sets could solve @atopal :) |
|
Sounds like maybe rough consensus to run on all of /Web/? |
|
From the conversation here, it sounds like we're in agreement on the content of this survey. What's next? @Westbrook @justinfagnani @kevinpschaaf any items to add to the list? |
|
@mfreed7 that looks like a great list. I would add:
I would also maybe want to clarify what some things mean there, since there often is some confusion even among current web components users. For instance, it's still often thought that Declarative Shadow DOM is somehow related to Declarative Custom Elements, when they're not. Misunderstandings like that could skew the survey. Plus, items like "DOM Parts" aren't necessarily well-defined as they have overlap with proposals like Template Instantiation. Also, Declarative Custom Elements imply things that aren't on the list yet like a template system (which is why I think it should maybe be added). I don't know how the survey is presented. Maybe we could work up a one or two sentence description for each item that could be shows on an info-tooltip? |
+1 let's add that to the list.
For this, I'd advocate for simply replacing the existing "DOM Parts" line item with the more general "Template Instantiation" (minus "syntax") and have it apply to the entire concept.
Agree that many of these are subject to confusion or ambiguity. I think the general thrust of this survey is to identify "interesting areas" for future more-in-depth surveys. So while I don't necessarily think it's a bad idea to add links or tooltips for each item, I also think it might not actually change the results or next steps too much. |
|
So based on the comment above, here would be the final survey content. I changed the two items listed above, and fixed up a few small things. LMK if I missed anything. NOTE: I also added two more items to the first question, which I think might help us understand which votes are coming from particularly passionate folks. Feel free to strike these back out if you disagree. Survey prompt before expanding: Browser vendors are working together to improve support across browsers for Web Components. Shape the future of Web Components by taking this 2-question survey! Question 1: What's your experience with Web Components (custom elements, shadow DOM, etc.)?
Question 2: For your needs as a developer, which Web Components features should be improved across browsers in the coming year? This means enabling support or fixing bugs. (This is not a complete list, other features may also be planned.) {Each of these have these three options: "I need this", " I don't need this", or "don't care / don't know" (the default)}
|
|
@mfreed7 , for question 1, adding those two last two items makes the options ambiguous. For example, people who love them, might know about them, but might not have used them. What's more important to know about the audience? We could also ask sentiment as a follow up question, and only ask people who have used them. |
|
Fair point! I’m ok leaving those two back off. They were spur of the moment additions. I think the original three options are good. |
|
Any updates on this? What do I need to do to get this survey moving? (Sorry, this will be my first such survey, so I don't know the process.) This set of questions feels like it is ready to go. |
|
@mfreed7 We're just getting it set up - I'm just about to send an email out 👍 |
|
Sorry that I am very late to this party :(. I do have a few comments..
Where did this particular initial list come from? |
I'm fine with this - it would just mean replacing this item:
with this:
I see this as the first survey of a series, with this one providing guidance to the interesting/non-interesting/confusing areas. We can then follow up with other surveys to try to tease that out. As-is, it's a huge list, and I'm afraid that we'll get lower quality results if we say "please answer this quick 2 question survey" and then whammo, hit them with a wall of text for question 2. Mind if we try to crisp up the data after this first survey?
We brainstormed it internally and here on this issue. I think a lot of it came from the Web Components Community Group report. |
Yes, that is exactly what I was suggesting. As to the other point - I totally agree, but I think we are kind of just saying the same thing from two ends. If you turn this into full sentences that looks really overwhleming. Sure. Conversely though, a long list of probably unfamiliar terms which aren't exactly self-explanatory is just hiding that, right? It's still overwhelming. To really answer those in an actually helpful way, people are gonna have to do some research, probably. Maybe the list is just too long to start with. |
I don't disagree that the list is long. But my hope would be that roughly half of the items get no response or mostly "don't care/don't know", and we can mostly ignore those for subsequent surveys. If the items sound confusing or unclear, but nobody's willing to spend 2 minutes searching for definitions, then they truly "don't care", and it should be safe for us to also not spend more time on those items. Conversely, if we proactively remove half the items to make the list shorter for this first survey, then we won't have a way to know if we got that truncation right. |
|
FWIW I agree with @bkardell here. My guess is that the most common reaction to seeing this long list is going to be "I don't understand enough to help with this survey" and so we end up with high drop off, and the only usable responses are from people who are so deep into web components that they're unlikely to be representative of the wider developer community. So of course we can run the study and see what happens, but I'd be nervous the results are going to be misleading. |
|
Perhaps there's a way to use the first question to try to capture or understand that problem? The current three options are:
What if we add one more option:
or something similar? |
|
I think that helps a little, although I definitely think that "I'm an expert" is too subjective a criteria. I think what we care about is more like "I use them regularly and keep up to date with future developments in this space", although there's probably a shorter rephrasing that captures the same idea. To be clear I'm not opposed to running the survey. But I have two concerns:
The concerns are mitigated a bit by sending out the survey to a small fraction of the MDN audience (as we do for all short surveys), but I think if we see fewer responses to this survey or high dropoff after the first question, we should be more wary of running similar surveys in the future. |
Thanks for the feedback. I do think the suggestion is a bit too long, but perhaps something like "I'm a regular, active user" or similar could address your concern?
Thanks also for this feedback. I'd be concerned about truncating the list or splitting it into multiple surveys, since the entire point (or at least my entire point) is to try to suss out relative priorities among all of these projects. My hope would be that a) the list isn't so long that we tire out too many people, and b) the 5% sampling limits the potential "damage" considerably. Just to summarize where I think we are now, the question 1 options should be: Question 1: What's your experience with Web Components (custom elements, shadow DOM, etc.)?
|
|
I think there's a distinction between "I regularly use web components" and "I actively keep up to date with the technical underpinnings of web components". Indeed there's a distinction between using components and authoring them. At risk of making the survey longer, you could have two questions: Q2. How happy are you with web components
That doesn't really help with concern that the main question about different possible priorities depends on too much knowledge of ongoing standards work to be answered by the majority of MDN users. But it would at least mean we would probably get some useful data about overall attitudes towards components from the first two questions. |
|
I suppose my first reaction would be that you were worried about the length of the survey before, so adding more questions sounds not great. I would love to understand more about "happiness" with web components, but perhaps that's a different survey? As to this suggestion:
I'm concerned about getting actionable results. With the "old" version of question 1, I can partition the results by self-reported skill level. For example, if something is important only for people who are only occasional users, perhaps that's a documentation problem. Whereas a priority also for experienced users might be more of a missing feature. With your proposed questions, I'm not sure how I make distinctions between users and authors. If something is a priority for either of those groups, it still seems like a priority. And I don't have a sense for the level of authoritativeness of the survey taker. How are you planning to use the results of this survey? Perhaps we're driving toward different goals. I'm trying to use it to help prioritize future work. |
|
closing since this now all done & compete |
Cross-posted from https://github.com/mdn/short-surveys/discussions/9 by request from @dontcallmedom
In web-platform-tests/interop#246 the top choice for the APIs and JavaScript survey was "Web Components (custom elements, Shadow DOM, etc.)", picked by ~39% of survey takers.
Web Components is a collection of technologies, so we'd like to learn more. @mfreed7 has proposed this:
Survey prompt before expanding: Browser vendors are working together to improve support across browsers for Web Components. Shape the future of Web Components by taking this 2-question survey!
Question 1: What's your experience with Web Components (custom elements, Shadow DOM, etc.)?
Question 2: For your needs as a developer, which Web Components features should be improved across browsers in the coming year? This means enabling support or fixing bugs. Please choose up to 5. (This is not a complete list, other features may also be planned.)
List:
The text was updated successfully, but these errors were encountered: