Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OWD BCD project: increase mdn-bcd-collector coverage to 90% #161

Closed
queengooborg opened this issue Jun 23, 2023 · 4 comments
Closed

OWD BCD project: increase mdn-bcd-collector coverage to 90% #161

queengooborg opened this issue Jun 23, 2023 · 4 comments
Assignees
Labels
H1 2024 Projects for the first half of 2024

Comments

@queengooborg
Copy link
Member

queengooborg commented Jun 23, 2023

Problem statement

Projects such as #85 and #160 that work to synchronize collector results with BCD are great. However, if the collector only covers a small set of the features BCD tracks, we can never guarantee complete accuracy.

And if we run the collector for new browser releases (#168), then ideally we should be able to update as many BCD data points as possible.

Command to get coverage stats: npm run feature-coverage -- -p api -p css -p html -p javascript -p mathml -p svg -c

Proposed solutions

This project is all about writing new tests to increase the collector's coverage of features tracked in BCD. The goal should be to get above 90% of overall coverage for testable features.

Task list

  • Increase test coverage for APIs
  • Increase test coverage for CSS
  • Increase test coverage for HTML
  • Increase test coverage for javascript
  • If possible, add tests for MathML
  • Increase test coverage for SVG

Progress Report

v10.10.2

api, css, html, javascript, mathml, svg: collector covers 12236 (90.02%) of 13593 entries tracked in bcd (1357 (9.98%) missing)

api: collector covers 8201 (95.15%) of 8619 entries tracked in bcd (418 (4.85%) missing)
css: collector covers 1841 (81.64%) of 2255 entries tracked in bcd (414 (18.36%) missing)
html: collector covers 549 (76.36%) of 719 entries tracked in bcd (170 (23.64%) missing)
javascript: collector covers 1202 (87.42%) of 1375 entries tracked in bcd (173 (12.58%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 414 (83.98%) of 493 entries tracked in bcd (79 (16.02%) missing)

v10.10.0 / v10.10.1

api, css, html, javascript, mathml, svg: collector covers 12202 (89.48%) of 13637 entries tracked in bcd (1435 (10.52%) missing)

api: collector covers 8167 (94.29%) of 8662 entries tracked in bcd (495 (5.71%) missing)
css: collector covers 1841 (81.64%) of 2255 entries tracked in bcd (414 (18.36%) missing)
html: collector covers 549 (76.25%) of 720 entries tracked in bcd (171 (23.75%) missing)
javascript: collector covers 1202 (87.42%) of 1375 entries tracked in bcd (173 (12.58%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 414 (83.98%) of 493 entries tracked in bcd (79 (16.02%) missing)

v10.9.0

api, css, html, javascript, mathml, svg: collector covers 11642 (88.30%) of 13184 entries tracked in bcd (1542 (11.70%) missing)

api: collector covers 8133 (94.34%) of 8621 entries tracked in bcd (488 (5.66%) missing)
css: collector covers 1369 (76.95%) of 1779 entries tracked in bcd (410 (23.05%) missing)
html: collector covers 549 (76.57%) of 717 entries tracked in bcd (168 (23.43%) missing)
javascript: collector covers 1173 (85.43%) of 1373 entries tracked in bcd (200 (14.57%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 389 (69.22%) of 562 entries tracked in bcd (173 (30.78%) missing)

v10.8.0

api, css, html, javascript, mathml, svg: collector covers 11613 (88.08%) of 13184 entries tracked in bcd (1571 (11.92%) missing)

api: collector covers 8128 (94.28%) of 8621 entries tracked in bcd (493 (5.72%) missing)
css: collector covers 1369 (76.95%) of 1779 entries tracked in bcd (410 (23.05%) missing)
html: collector covers 525 (73.22%) of 717 entries tracked in bcd (192 (26.78%) missing)
javascript: collector covers 1173 (85.43%) of 1373 entries tracked in bcd (200 (14.57%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 389 (69.22%) of 562 entries tracked in bcd (173 (30.78%) missing)

v10.7.2

api, css, html, javascript, mathml, svg: collector covers 11381 (84.83%) of 13416 entries tracked in bcd (2035 (15.17%) missing)

api: collector covers 8110 (92.06%) of 8809 entries tracked in bcd (699 (7.94%) missing)
css: collector covers 1174 (64.65%) of 1816 entries tracked in bcd (642 (35.35%) missing)
html: collector covers 524 (73.39%) of 714 entries tracked in bcd (190 (26.61%) missing)
javascript: collector covers 1155 (83.51%) of 1383 entries tracked in bcd (228 (16.49%) missing)
mathml: collector covers 29 (21.97%) of 132 entries tracked in bcd (103 (78.03%) missing)
svg: collector covers 389 (69.22%) of 562 entries tracked in bcd (173 (30.78%) missing)

v10.7.1

2227/13378 (16.65%) entries missing from collector that are in bcd for api, css, html, javascript, mathml, svg

727/8777 (8.28%) entries missing from collector that are in bcd for api
660/1810 (36.46%) entries missing from collector that are in bcd for css
190/715 (26.57%) entries missing from collector that are in bcd for html
347/1384 (25.07%) entries missing from collector that are in bcd for javascript
132/132 (100.00%) entries missing from collector that are in bcd for mathml
171/560 (30.54%) entries missing from collector that are in bcd for svg

v10.7.0

all: 2324/13356 (17.40%) entries missing from collector that are in bcd for api, css, html, javascript, mathml, svg

api: 724/8758 (8.27%) entries missing from collector that are in bcd
css: 670/1811 (37.00%) entries missing from collector that are in bcd
html: 257/711 (36.15%) entries missing from collector that are in bcd
javascript: 370/1384 (26.73%) entries missing from collector that are in bcd
mathml: 132/132 (100.00%) entries missing from collector that are in bcd
svg: 171/560 (30.54%) entries missing from collector that are in bcd
@Elchi3 Elchi3 added proposal (actionable) Enough information is provided and the work is scoped well. Actionable but not prioritized right now and removed not ready labels Jan 19, 2024
@Elchi3 Elchi3 changed the title OWD BCD project: increase mdn-bcd-collector coverage OWD BCD project: increase mdn-bcd-collector coverage to 90% Jan 19, 2024
@Elchi3 Elchi3 added H1 2024 Projects for the first half of 2024 and removed proposal (actionable) Enough information is provided and the work is scoped well. Actionable but not prioritized right now labels Jan 19, 2024
@Elchi3 Elchi3 mentioned this issue Feb 1, 2024
@queengooborg
Copy link
Member Author

As of mdn-bcd-collector v10.10.2, we have now reached over 90% feature coverage! 🎉

@Elchi3
Copy link
Member

Elchi3 commented Mar 12, 2024

@queengooborg can we update the feature coverage script to list untestable features along with the other stats?

So something like:

api, css, html, javascript, mathml, svg: collector covers 12236 (90.02%) of 13593 entries, 1357 (9.98%) missing, and XXX (x.xx%) are untestable.

@queengooborg
Copy link
Member Author

I can update the script, though I'd have to change the output format (untestable features are omitted from the stats)!

@Elchi3
Copy link
Member

Elchi3 commented May 29, 2024

We've finished this project. Thanks again to funding from the Sovereign Tech Fund making this work possible.

From our final report:

Deliverable 2: Increase the coverage of the mdn-bcd-collector to 90% (completed)

OWD’s mdn-bcd-collector currently monitors 90.15% of all web platform features recorded in browser-compat-data. At the beginning of the second round of the challenge, our compatibility data collector monitored 82% of all web platform features in the browser-compat-data project. We’ve surpassed the 90% coverage deliverable, by expanding the scope of our test suite. Notably, the collector now includes tests for all CSS property values in addition to CSS properties.

While the ultimate goal would be 100% coverage, we discovered some features (2.27% of all features) that can not be covered. Unfortunately, there are a limited set of features that can not be tested programmatically, either because they require specific hardware or they require user interactions that are not machine replicable. We created a list of untestable features that includes the features limiting factors. We will be addressing the remaining (7.58%) features in the near future; including global events and worker support for API interfaces.

@Elchi3 Elchi3 closed this as completed May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
H1 2024 Projects for the first half of 2024
Projects
None yet
Development

No branches or pull requests

2 participants