Add support for DSSE

[adityasaky]

Description of issue or feature request:

Dead Simple Signing Envelope (DSSE) is the new signature wrapper recommended for use in in-toto (https://github.com/in-toto/ITE/blob/master/ITE/5/README.adoc) and TUF (pending theupdateframework/taps#138). Naturally, it makes sense for its implementation to live in securesystemslib.

With ITE-5 up for acceptance (in-toto/ITE#22), we are trying to define a transition period for in-toto during which the reference implementation supports generating and verifying metadata in both formats. This time period is dependent at least in part on the DSSE functionality being present in securesystemslib for in-toto to use. We can tentatively define a relative transition period, perhaps a year after in-toto starts supporting both wrappers.

Current behavior:

securesystemslib provides some crypto-specific primitives used by TUF and in-toto. Both projects maintain their own implementation of the current signature wrapper.

Expected behavior:

securesystemslib continues providing the functionality it currently does and adds DSSE support, allowing the python implementations of both TUF and in-toto to transition to the new wrapper.

cc @SantiagoTorres @joshuagl @mnm678 @trishankatdatadog

[adityasaky]

For reference, there's currently support for DSSE in Go: https://github.com/secure-systems-lab/go-securesystemslib. The DSSE specification also includes an implementation we can likely borrow from here: https://github.com/secure-systems-lab/dsse/tree/master/implementation

[lukpueh]

Related: #272

[PradyumnaKrishna]

@lukpueh, This issue is closed by in-toto#1 which is a small part of this issue. I believe closing this issue was not intended.

[lukpueh]

A working DSSE implementation is available in master...in-toto:securesystemslib:master courtesy of @PradyumnaKrishna! ❤️ (see GSoC blog post for context). The implementation was tailored for in-toto (in-toto/in-toto#503) but should work for python-tuf as well.

The result is a ~1,4K LOC diff, which I have reviewed in the course of multiple downstream PRs. Other maintainers should still take a look before we upstream. So I suggest to either:

• PR all at once and keep the nicely curated commit history, or
• PR module by module and make reviews more fun.

If we do the latter, I'm happy to coordinate the PRs and make sure that important information from the original commit history is preserved.

Since the decision depends mostly on how much we can ask from potential reviewers, they should voice their preference.

cc @jku, @joshuagl, @MVrachev, @adityasaky

---

Diff overview (ordered by degree of dependence from least to most, w/o tests)

• util.py -- DSSE-compliant b64enc and b64dec methods

• exceptions.py -- DSSE signature verification error and generic de/serialization errors

  TODO: Coordinate new signature verification error added here and in #456.

• key.py -- Abstract Key interface plus SSlibKey and GPGKey implementations

  TODO: Drop in favor of very similar *Key implementations from #456. GPGKey should be dropped altogether but may be upstreamed at a later point e.g. as BetterGPGKey (see #456 discussion).

• serialization.py -- Port of tuf.api.serialization

  NOTE: The main difference is that the securesystemslib JSON deserializer only calls json.loads but does not instantiate a user-specific object (such as Metadata). The latter would need to happen in a subclass (e.g. PayloadDeserializer, AnyMetadataDeserializer, EnvelopeJSONDeserializer)

  TODO (optional): Assess whether this is useful for the tuf metadata api, which served as inspiration. Regardless, this should be in securesystemslib, because it is useful for in-toto and DSSE.

• metadata.py -- DSSE envelope with methods to create and verify signatures, and to de/serialize envelope and payload

  TODO:

  • Replace match_keyid with simple keyid ==-matching (see #456 discussion)
  • Don't case handle SIGNATURE_SCHEMA/GPG_SIGNATURE_SCHEMA (DSSE should use a GPGSigner that generates normal Signature objects)
  • Maybe find better names than create_sig and verify_sigs (sign and verify_signatures are already taken, see in-toto@f418b3f)

[adityasaky]

I feel bad about the commit history, but I think I'd prefer it to be module by module personally.

[lukpueh]

Now that #456 is merged and we are starting to get a good feel for it (#442, #472) we should take a stab at this here too.

I suggest to just start with a single draft PR after all. This should make fixing conflicts easier (see TODOs in comment above). And maybe the diff becomes small enough to review all at once. If not we can still split by modules.

@PradyumnaKrishna, are you still up for the task?

[PradyumnaKrishna]

Yes, I am up for it.