Proposal: Clearer distinction between "verification failed" and "test failed"

[jviide]

There is room for extra confusion because a test can FAIL when a stub outputs VERIFY SUCCESS. But a test may also FAIL when a stub outputs VERIFY FAILURE. Anecdotally, this seems to confuse people. The very specific meaning reserved for VERIFY FAILURE doesn't help - it shouldn't be used to signal, for example, that verification failed because the certificate couldn't be fetched thanks to a network error.

I suggest that we rethink our terminology to distinguish test FAIL more clearly from stub VERIFY FAILURE. Fail/failure seems to be a common word used in e.g. unit testing, so maybe we could focus to the stub output protocol.

One option is to use terms accept and reject to refer whether the certificate was accepted or rejected. In pull request #82 (included in version 0.1.0) the trytls tool already started using accept/reject-terminology to tell whether VERIFY SUCCESS or VERIFY FAILURE was the "correct" answer. So there's some precedent and no-one has yet complained 😉

Of course there are other options, and this whole suggestion might be completely misguided. Thoughts?

[Mamietti]

(VERIFY?) ACCEPT and (VERIFY?) REJECT sounds pretty clear.

Word reject is more "I don't want to", failure is "I can't".

[Mamietti]

The current laws of Not-Quite-Asiimov:

1. Cannot be SUCCESS/FAILURE or PASS/FAIL: mixes with runner result
2. Cannot be pseudo-English (VERIFY ACCEPT I have now)
3. One word wonders are unclear (ACCEPT/REJECT, REJECT ≈ UNSUPPORTED?)

We might have to make a some sort of compromise.

[Mamietti]

We'll use ACCEPT/REJECT from now on.

[jviide]

Pull request #120 implements the ACCEPT/REJECT support for the Python-based trytls runner (and removes the VERIFY SUCCESS/VERIFY FAILURE support).

[Mamietti]

b93c7cb edited the stubs for that.