Content suggestion: Documenting testing strategies

[dontcallmedom]

(Not a very specific proposal yet, but more a fuzzy idea that would benefit from refining before making an actual RFC.)

What is the new suggestion?

Have a structured approach to documenting how to set up and run tests for Web technologies that need specific approaches, e.g. because of hardware integration à la WebNFC, WebBluetooth, Media Capture, WebXR, WebAuthn, or because they expose a different execution environment (*Workers, shaders in WebGL/WebGPU, WebAssembly, …)

Why is it important or useful?

A reasonable reading of the MDN Web Testing Report would be that it any friction that exists in the process of setting up and running tests make it more likely it won't get done or will get done poorly. MDN may have a role to play there by making the topic more prominent and spreading the word about practical approaches.

[chrisdavidmills]

This is a really good idea @dontcallmedom . I could imagine:

1. An optional section being added to API landing pages titled something like "Testing strategies", which will explan high-level approaches to testing that API, what is needed, etc.
2. A standard guide type called "Testing the ", which would be created as a child page of the main API landing page, and contain further information on more testing strategy specifics, if the information was too long to go in a subsection on the page itself.
3. These sections/pages only being included in cases where the API requires some kind of special testing setup, or there are specific gotchas to know about. Cases where the answer isn't just "load your page in the browser you want to test it in".

[wbamberg]

@dontcallmedom and I had a call today about this, and here are my notes:

---

Overall scope

The initial proposal is about testing strategies for specific Web APIs. But maybe it's worth expanding this out to how MDN documents testing in general. As such it might have a research component:

• what testing documentation do we currently have on MDN? (notably https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing, anything else?)
• assessing the needs implied by the MDN testing report: https://insights.developer.mozilla.org/#web-testing
• designing documentation to fill the gaps

The other strand is about testing strategies for specific Web APIs that have particular testing requirements. Here we might (as Chris suggested: mdn/content#3917 (comment)) have a "Testing the ... API" guide under the overview page of such APIs.

• it would be very helpful to work with domain experts here. Would love a model where OWD could figure out which docs are needed at a high level and work with various domain experts for various APIs, helping them navigate MDN and help write stuff.
• domain experts could be found through the organizations associated with OWD SC members.

How much detail should we go into for testing strategies?

There's a big range which makes a big difference to the amount of work it is.

• maybe best to experiment here: pick an API, write an outline, get feedback. Start with a smaller guide in a first phase.

Which APIs are promising candidates?

• Web Authentication: there's a proposed WebAuthN Adoption CG (https://www.w3.org/community/blog/2020/04/21/proposed-group-webauthn-adoption-community-group/) which would be very interested in helping to have testing docs on MDN. Also the FIDO alliance (https://fidoalliance.org/)

• WebXR/Immersive Web: strong community push for adoption

• WebRTC: also strong desire to reduce barriers to adoption.

Accessibility testing

Another possible item in here is accessibility testing, which would be very much in our remit, and for which we could certainly get help from OWD allies. This seems like a third possible strand, alongside (1)"overall Web testing strategies" and (2)"testing strategies for specific APIs".

Research project?

Should we consider for Q3, for this to be a research project that determines what testing-related content we are missing on MDN and prioritizes it?

[hamishwillee]

Slightly hesitant to comment on this one. Here's my gut reaction anyway (perhaps worth moving this whole issue to a discussion):

• Yes we need to invest in this. Testing is one of the pillars of software development and it is clearly under-served.
• The cross browser testing stuff is a huge problem all on its own. If it were me I would "pick an API, write an outline, get feedback. Start with a smaller guide in a first phase.".
• I would run a separate project for MDN.
• The whole thing needs to be very iterative. We probably want a constrained scope even on the research report.

And some impressions from my last 6 months ...

• MDN mostly collates client side testing tools running in a particular (firefox) browser.
  • It does talk about linters etc. But it doesn't cover the big picture "cross browser" compatibility issues highlighted by the mdn testing report.
  • I think maybe there is a missing page on types of testing, that then leads to finding the tools you need to do different things.
    • We have useful scattered resources that could be better marshalled. For example we might better "surface" features like: https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#browser_compat_warnings
    • we also jump into topics like "understanding client side frameworks" from top level testing topic. Perhaps not focused enough on the end goal.
  • It took me a fair while in the job (and pointers from Daniel Beck) to discover some of the online tools you can use for cross-browser testing. You might argue that once BCD is in place others should not need these. IMO worth documenting.
  • What would be really nice is a service that can parse your codebase and it spits out a BCD report on what you actually used.
• Whenever I get to a problem case that requires extra hardware I do not have I am blocked. So real problems being highlighted.

[wbamberg]

I'm glad you did comment, Hamish! In this:

• The cross browser testing stuff is a huge problem all on its own. If it were me I would "pick an API, write an outline, get > feedback. Start with a smaller guide in a first phase.".
• I would run a separate project for MDN.

...what do you mean by "I would run a separate project for MDN"?

I always prefer it when projects have concrete deliverables that help our users, but it looks like we really do need quite an open research project for figuring out what testing documentation we are missing overall.

But I wonder if we should also have - separately and in parallel - a more concrete project to document testing strategies for "difficult" Web APIs like WebAuthN. This one could go, for example:

• pick three likely-looking APIs (we have three up there^)
• find domain experts willing to help
• for each:
  • draft an outline of the docs we need
  • revise from feedback
  • write a guide

[hamishwillee]

...what do you mean by "I would run a separate project for MDN"?

I mean that MDN has a heap of work that could be done just to establish where we are, what is present but not well linked, where we would like to be. IMO the people who work on this and the type of work they would do would be quite different to those working on the more specific investigation of "difficult web APIs". There is no need to treat them as part of the same work, and no real benefit in doing so.

But I wonder if we should also have - separately and in parallel - a more concrete project to document testing strategies for "difficult" Web APIs like WebAuthN.

Yes. Suck it and see :-)

[Ryuno-Ki]

)It might help to have one comment per topic, so we can properly thread here)

Chiming in on the question of what is meant to be tested, I'd wonder, what kind of tests are we talking about here (in terms of testing pyramid). The above would make me think about unit or integration tests.

I experienced a lack of guidance when I was working on web extensions (which turn out to be really hard to test end-to-end with tooling - or I missed the silver bullet somehow).

Are there communities we could ask? I'm thinking of Cypress folks here. Back in angular/protractor#5502 the community also suggested several great projects that could be asked for feedback.

A separate project to MDN would then allow to collaborate with the testing community. But we should think about the interface we would need to integrate the results and learnings. A JSON API, some Markdown files, iFrames or what-have-you.