docs: partial reports can be submitted #3179
Conversation
Signed-off-by: Mattia Lavacca <lavacca.mattia@gmail.com>
k8s-ci-robot
added
release-note-none
k8s-ci-robot
added
the
approved
There was a problem hiding this comment.
After thinking it over, I do agree that it's better to get partial reports rather than no reports because this will increase our awareness of implementations in the field and enable more opportunities to engage with them.
I do think we may need to adjust the display to accommodate (make sure it's clear, and perhaps delineated when an implementation is reporting partial). I approve,
/approve
However I would like us to make sure we have a follow-up issue in place for considering how the display should look like for this (I don't consider it a blocker though) before we merge:
/hold
k8s-ci-robot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mlavacca, shaneutt The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Yes, it makes sense. I've opened #3183 to capture this. Let me know if it fits with what you had in mind :) |
|
LGTM /unhold |
k8s-ci-robot
removed
the
do-not-merge/hold
| that all the core conformance tests have been successfully run as well as all | ||
| the tests related to the supported extended features. No reports with partial | ||
| or failing results can be accepted. | ||
| need to have the `result` fields (core and extended) set to `success` or `partial`. |
There was a problem hiding this comment.
It seems like we probably need to include some guidelines for how we handle partial conformance results. Maybe it's sufficient to just say that they'll be displayed separately?
conformance/reports/README.md
Outdated
| It means that all the core conformance tests have been successfully run or properly | ||
| skipped as well as all the tests related to the supported extended features. No | ||
| reports with failing results can be accepted. |
There was a problem hiding this comment.
I think there's another reason for partial conformance - the tests may all pass, but the reproduction required some kind of additional unexpected steps.
There was a problem hiding this comment.
👍 I reworded this section to include this information
Signed-off-by: Mattia Lavacca <lavacca.mattia@gmail.com>
k8s-ci-robot
added
size/S
|
All of @robscott's feedback is addressed, going to get this one in. /lgtm |
k8s-ci-robot
added
the
lgtm
What type of PR is this?
/kind documentation
/area conformance
What this PR does / why we need it:
After discussing the possibility of accepting partial reports in #3021, we eventually decided to encourage implementations to start submitting reports even if there is no full conformance. This PR updates the documentation to reflect this final decision.
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: