Deterministic hashing / serialization

[yusefnapora]

To support signing of metadata, we need a "canonical" serialization format that will deterministically produce the same binary output for equal inputs. IPLD uses CBOR as its canonical representation, since they need consistent hashes as well.

The yn-graph-io branch adds CBOR serialization support and adds a slightly modified serializer for json4s AST values that will sort object keys lexicographically before serializing.

We need to test and verify a few things before we can rely on it:

• duplicate property keys should result in consistent output
  • see this note about duplicate keys in the ipld spec
• test serialization of numeric types.
  • The canonical CBOR section of the CBOR spec suggests that you always use the smallest possible numeric type that will support the value you're encoding.
  • The jackson-cbor module we're using has a flag WRITE_MINIMAL_INTS to use the smallest possible integer type, which is enabled in our serializer. However, there's no floating point / decimal support, so we may need to figure that out ourselves.
• test json input with various whitespace & key ordering differences to verify that they produce the same CBOR output

Also, at some point in the future, we'd like to produce actual IPLD objects. The IPLD canonical format uses a CBOR tag for "link" fields. The jackson CBOR module doesn't support tags, so we'll need to figure out how to add the link tag.

[parkan]

Let's park this work for now until IPLD lands in go-ipfs master; at that point, I think the IPLD writer can be a separate sub-project building on this work.

[yusefnapora]

sounds good to me 👍