Atomic Metadata

[zathras-crypto]

This PR serves to prototype the ability to publish and interpret atomic metadata via the Omni Layer.

The purpose is to remove duplication of metadata in future transaction types that may permit attaching metadata. For example the string "This transaction means X, find out more at Y.com" could be included with transactions relating to X but this is incredibly inefficient. Instead the string could be published just once and then referenced.

"Atomic metadata" in this context is metadata which is independent of any transaction and even independent of the Omni state. To put it another way, it is not consensus-affecting. Publishing metadata on-chain is a contentious issue, and the methodology used is intended to provide the functionality as an extension outside of the consensus state all clients must share.

The outline is fairly straight forward. Any arbitrary string (assume max length of datacarrier) can be hashed (SHA256) and the first and last bytes of the hash used to form an identifier. This is a deterministic outcome and has no relation to the Omni state. Should that same string be published as metadata via the Omni Layer, this two byte identifier could then be used in future transactions to reference the published string.

Since this process is isolated from the state, any future transaction can reference metadata but it cannot require it. References to non-existent metadata will simply return an empty string. Clients can choose to offer users the choice on whether to use and display metadata, and clients can even ignore metadata completely during processing without affecting the Omni state.

Example usage:

We can get the metadataid for a given string at any point (again independent so no synced chain required):

omni_getmetadataid "This is Core-ID metadata."
  {
    "metadataid": "ad0d",
    "metadata": "This is Core-ID metadata."
  }

We see the metadataid for this string is ad0d. If we modify the string slightly, the hash will change and we will receive a different metadataid:

omni_getmetadataid "This is Core ID metadata."
  {
    "metadataid": "8096",
    "metadata": "This is Core ID metadata."
  }

Let's say we wish to publish this string on-chain so we can reference it in future transactions. We would do that with omni_sendpublishmetadata:

omni_sendpublishmetadata myyqsvHWpQ31ru3R7napDD2E74fxzKA27o "This is Core ID metadata."
  89f602dcd95c17cdafe9303c6a5bd33ae641dbc29db08f5c2dbbf4483b3df47f

The metadata is now available to be referenced by a future transaction. We can retrieve the metadata by passing the address and the metadataid to omni_getmetadata:

omni_getmetadata myyqsvHWpQ31ru3R7napDD2E74fxzKA27o 8096
{
  "address": "myyqsvHWpQ31ru3R7napDD2E74fxzKA27o",
  "metadataid": "8096",
  "metadata": "This is Core ID metadata."
}

If we inspect the transaction we will see it is of a new type (80 - Publish Metadata):

omni_gettransaction 89f602dcd95c17cdafe9303c6a5bd33ae641dbc29db08f5c2dbbf4483b3df47f
  {
    "txid": "89f602dcd95c17cdafe9303c6a5bd33ae641dbc29db08f5c2dbbf4483b3df47f",
    "fee": "0.00004740",
    "sendingaddress": "myyqsvHWpQ31ru3R7napDD2E74fxzKA27o",
    "ismine": true,
    "version": 0,
    "type_int": 80,
    "type": "Publish Metadata",
    "metadataid": "8096",
    "metadata": "This is Core ID metadata.",
    "valid": true,
    "blockhash": "6a8e5f8dfe654a7d88916662909518c033be7602168af7a8283dffa333cf3e7a",
    "blocktime": 1488182950,
    "positioninblock": 2,
    "block": 538,
    "confirmations": 1
  }

It may also prove useful to obtain all the metadata a given address has published, and this can be done with the omni_getaddressmetadata call, for example:

omni_getaddressmetadata myyqsvHWpQ31ru3R7napDD2E74fxzKA27o
{
  "address": "myyqsvHWpQ31ru3R7napDD2E74fxzKA27o",
  "addressmetadata": [
    {
      "metadataid": "8096",
      "metadata": "This is Core ID metadata."
    }, 
    {
      "metadataid": "b44d",
      "metadata": "This is a test piece of metadata."
    }, 
    {
      "metadataid": "c4e1",
      "metadata": "This is some test metadata."
    }, 
    {
      "metadataid": "da3e",
      "metadata": "This is a test string."
    }
  ]
}

Notes

• Internally the identifier is stored as a concatenation of the address, a field delim and the metadataid
• References will resolve only the most recent publishing of metadata corresponding to a given metadataid. Publishing the same metadata, or new metadata with a colliding metadataid will replace the existing metadata.
• The metadataid is intentionally short for a number of reasons:
  • To minimize the number of bytes needed to reference a piece of metadata in future transactions
  • To intentionally allow collisions:
    • (whilst discouraged) metadata can be updated if necessary - with a little manipulation (rotating 2 bytes) then a different string with a colliding metadataid can be published to update old metadata
    • To enforce the behaviour that this is for deduplicating metadata usage only. A collision in regular use for an address publishing a few entries is unlikely (and can be worked around with a simple change to the string eg adding a space or full stop) however trying to use this for lots of data or unique data (UUIDs for example) and collisions will get in the way very quickly, which should prevent those types of engineering decisions from those using the technology.
• This PR is fully functional but is also a prototype. Needs to be cleaned up before taking it on for real.
• This PR is against a new develop branch within the core-id repo, with a potential pull/merge flow of core-id/featurebranch > core-id/develop > omnilayer/develop > omnilayer/master if suitable.