Add an OpenSSL harness

[woodruffw]

Still a WIP. This just gives us another thing to sanity-check against.

Closes #14.

[woodruffw]

This is a chunky PR, but most of it is two single-header libraries that can be ignored for review purposes.

Key changes:

1. We now have another test harness, this one for OpenSSL.
2. We now have JSON schema support for testcase "result" models.
3. The OpenSSL harness emits testcase results in the format specified by (2), to a ./results.json file.

I'm thinking that, once this is merged, we should probably update the crypto/x509 harness to behave similarly. That way, we can begin comparing the two in a more structured manner (as well as putting them into CI in a more structured manner than just uncoordinated running). Thoughts?

[woodruffw]

NB: Got lazy; this assumes macOS with Homebrew at the moment.

[alex]

When you say a schema for results, do you mean expected results (i.e., "This testcase should validate, this one should fail") or do you mean for the results of running a test?

[woodruffw]

When you say a schema for results, do you mean expected results (i.e., "This testcase should validate, this one should fail") or do you mean for the results of running a test?

The latter -- the testcase schema contains the expected results (via expected_result), and the "results" schema contains the "actual" results (via actual_result). Terrible naming, sorry.

[alex]

Why do we want/need that? Seems like how you consume results will be very project specific (e.g., for pyca/cryptography it'll just be part of our pytest suite)

…

On Wed, Jul 5, 2023 at 11:52 AM William Woodruff ***@***.***> wrote: When you say a schema for results, do you mean expected results (i.e., "This testcase should validate, this one should fail") or do you mean for the results of running a test? The latter -- the testcase schema contains the expected results (via expected_result), and the "results" schema contains the "actual" results (via actual_result). Terrible naming, sorry. — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBEQVQU2FM2XHXYH7V3XOWESBANCNFSM6AAAAAAZ5DIRZY> . You are receiving this because you commented.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[woodruffw]

Why do we want/need that? Seems like how you consume results will be very project specific (e.g., for pyca/cryptography it'll just be part of our pytest suite)

I don't think it'll useful for pyca/cryptography's suite, but my thinking was that it'd be useful for quickly diffing/visualizing the expected behavior here (i.e. as we're getting stood up, it'll be nice to have a structured way to compare how Go and OpenSSL behave differently).

In other words I think end consumers probably won't need the result models; they're mostly just useful for our own CI/diagnosing as we build out testcases here.

[woodruffw]

Relaying from chat: @alex brought up a good point that the result models probably won't be externally useful, so I'm going to exclude them from the generated limbo-schema.json.

[tnytown]

the result models probably won't be externally useful, so I'm going to exclude them from the generated limbo-schema.json.

One thing to note is that the Go implementation parses the test cases with structs automatically generated from JSON schema. We can use an unstructured map[string]interface{} for results or write some structs manually, but I'd prefer some kind of synchronized JSON structure.

Dumb question: other than the test harnesses, where is the Limbo schema going to be used?

[woodruffw]

We can use an unstructured map[string]interface{} for results or write some structs manually, but I'd prefer some kind of synchronized JSON structure.

Let's write them manually -- we don't expect this result schema to change significantly and, if it does, changes to it should be considered part of our internal use.

Dumb question: other than the test harnesses, where is the Limbo schema going to be used?

Not a dumb question! The ultimate idea here is to make both limbo.json and limbo-schema.json reusable within third-party libraries/X.509 validation implementations. In other words: our own harnesses won't be the source of ground truth; each upstream should be consuming the testcases and integrating them into their own testsuites.

That's the medium-term plan with pyca/cryptography as well.

[tnytown]

Not very familiar with the OpenSSL APIs. Looks good overall, build nits notwithstanding.

[woodruffw]

@tnytown JFYI, I've written an informal description of each harness's expected behavior here: 55b3f5c