Skip to content

Commit

Permalink
feat: add TIP-48: Metadata Encoding Standard v1
Browse files Browse the repository at this point in the history
  • Loading branch information
@Tuditi
Tuditi committed Jun 20, 2023
1 parent e30ee79 commit 7c32f44
Show file tree
Hide file tree
Showing 2 changed files with 300 additions and 0 deletions.
19 changes: 19 additions & 0 deletions tips/TIP-0048/irc48.schema.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/iotaledger/tips/main/tips/TIP-0038/irc38.schema.json",
"title": "JSON Schema to define layer-3 JSON standards",
"description": "JSON Schema to define layer-3 JSON standards",
"type": "object",
"properties": {
"standard": {
"type": "string",
"description": "identifier for the metadata standard used",
},
"version": {
"type": "string",
"description": "version of the standard used",
"pattern": "^v\\d+.\\d+$"
}
},
"required": [ "standard" ]
}
281 changes: 281 additions & 0 deletions tips/TIP-0048/tip-0048.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,281 @@
---
tip: 48
title: Metadata Encoding Standard v1 (MES-1)
description: A standard for encoding and decoding data in the metadata field of an output
author: Nicole O'Brien (@nicole-obrien) <cheerfulnicole9@gmail.com>, David De Troch (@tuditi) <david.detroch@pm.me>, Matthew Maxwell (@maxwellmattryan) <maxwellmattryan@gmail.com>
discussions-to: https://github.com/iotaledger/tips/discussions/131
status: Draft
type: Process
created: 2023-05-12
---

# Abstract

The TIP describes the Metadata Encoding Standard v1 (MES-1), a standard for encoding and decoding data in the metadata field of an output. The lack of a standard for metadata encoding/decoding poses a problem for applications that interpret metadata of an output/transaction across the ecosystem. The specification outlines a layered approach to encoding, with each encoding level being a layer of the onion. Level 1 determines the encoding of the data type, with backwards compatibility for existing metadata that uses JSON. Level 2 defines the format of the data. Further levels of encoding can be defined for specific standards. The specification also proposes a flexible system for defining the bytes that tell us how to decode the data, which allows to add future standards. The proposed encoding uses the first bit as a flag and the remaining bits to denote the encoding type.

# Motivation

Currently, applications don't have a standard that helps them encode and decode the data fields of a Transaction or Output (e.g. the metadata field of an Alias). Each application is free to define data objects as they see fit, even if the data format already adheres to an RFC standard (e.g. JSON, UTF-8, CBOR, ...). This poses a problem for applications that interpret metadata across the ecosystem. We think it's beneficial to introduce a TIP that standardizes encoding metadata.

## Encoding / Decoding

This TIP defines several levels of encoding, much like the layers of an onion. Each layer has to be decoded in order to understand the full message. For simplicity we will order the layers in the order required to decode and read the data at the center of the onion.

This TIP refers to the State Metadata field of an Alias Output and the Metadata Feature as "metadata" throughout the document. [TIP-18](https://github.com/iotaledger/tips/blob/main/tips/TIP-0018/tip-0018.md) defines the byte format of the metadata on a lower level.

### Layer 1 - Encoding

Layer 1 is used to determine how we should decode the metadata e.g. should we treat the data as a byte array or a UTF8 string or something else entirely.

Initial sensible layer 1 encodings can be defined as:

| Value | Hex | Encoding | Check Next Encoding | Type | Comment |
| ----- | --- | ---------------- | --- | --------- | --------------------------------------------- |
| 0 | 00 | Byte Array | True | Bytes | |
| 8 | 08 | UTF8 | True | Text | |
| 16 | 10 | UTF16 | True | Text | |
| 18 | 12 | GB18030 | True | Text | |
| 32 | 20 | UTF32 | True | Text | |
| 34 | 22 | UTF8/JSON/string | False | Text/JSON | Special case for JSON backwards compatibility |
| 49 | 31 | Bytes/CBOR | False | Bytes | |
| 91 | 5B | UTF8/JSON/array | False | Text/JSON | Special case for JSON backwards compatibility |
| 123 | 7B | UTF8/JSON/object | False | Text/JSON | Special case for JSON backwards compatibility |

### Layer 2 - Format

Layer 2 determines the format of the data that the application wants to read. This could represent an existing data or file format or a new custom proprietary one that can be linked in this TIP e.g. JSON, CBOR, XML.

##### Byte Array - 00

| Value | Hex | Encoding | Check Next Byte | Type | Comment |
| ----- | --- | ----------------------- | --- | ----------- | --------------------------------------------------------------- |
| 0 | 00 | Bytes | False | Plain Bytes | |
| 1 | 01 | ISC | true | Plain Bytes | Need to define the standards for smart contract metadata fields |
49 | 31 | CBOR | False | Bytes

##### Text Format - 08, 10, 12, 20

| Value | Hex | Encoding | Check Next Byte | Type | Comment |
| ----- | --- | -------- | --- | ---------- | ------- |
| 0 | 00 | String | False | Plain Text | |
| 1 | 01 | JSON | False | Document | |

### Layer 3 - Standard

Layer 3 is an optional standard that can be defined to allow programs to understand the data. Decoding the standards will be specific to the data format chosen. E.g IRC30, defined in [TIP-30](https://github.com/iotaledger/firefly/pull/6991/files), contains the definition of how to store metadata of a native token and currently only contains a definition for JSON format but it could be extended to include a decoding for CBOR etc.

##### JSON

In JSON a standard can be defined using the key `standard` and an optional version defined using the key `version`. The [JSON scheme](#json-schema-for-layer-3-standards) at the end of this document provides information around its structure. Currently we have two defined standards for JSON formatted metadata:

Standard | Version | Link |
| ----- | --- | -------- |
| IRC27 | v1.0 | https://github.com/iotaledger/tips/blob/main/tips/TIP-0027/irc27.schema.json
| IRC30 | | https://github.com/iotaledger/tips/blob/main/tips/TIP-0030/irc30.schema.json

##### ISC
| Value | Hex | Encoding | Check Next Byte | Link |
| ----- | --- | -------------------- | --------------- | ------------------------------------------------- |
| 0 | 00 | ISC state | False | |
| 1 | 01 | ISC on chain request | False | https://hackmd.io/SWpeUi_eTDKbnvK623vMmw#Metadata |


#### Combinations

We also have the ability to shorten encodings to fewer bytes by defining a specific byte for a combination of the above encodings. And in specific cases as we will describe later for backwards compatibility the the byte itself could form part of the data to be parsed.

### Encoding bytes

The metadata is represented by binary data, so bytes are a logical choice for markers or flags to decode the data. Therefore we introduce a system to define the bytes which tell us how to decode the data.

#### Extendable bytes

The system is flexible so new standards can be added in the future. The proposed encoding uses the first bit as a flag and the remaining bits to denote the encoding type. If the flag bit is unset (flag bit = 0), only a single byte signals the format decoding type. Otherwise (flag bit = 1), the decoding also considers the next byte for decoding the format.

![](https://i.imgur.com/HXZ84PZ.jpg)

![](https://i.imgur.com/IyCaCFU.jpg)

This approach provides 128 single byte values to define standards for. An additional 32,760 double byte values ensure that the encoding is future proof.

# Rationale

Currently, two application level metadata standards exist: the IRC27 and IRC30 metadata standards for NFTs and Foundries. They don't specify how they are encoded, but applications assume that the metadata field consists of hex encoded JSON objects.

Soon we will introduce three more metadata standards. These relate to the anchoring of smart contract state and DID records and the interaction with the smart contract chains on layer 1. To our knowledge the two smart contract standards save space through proprietary encoding. (There is no need to store a key value pair / object if you know all the fields and can enforce the size and order).

These will not be the only metadata standards introduced in IOTA's lifetime, and applications should decode the metadata without using trial and error. Therefore, we propose a metadata encoding standard before adding any more functionality.

This design represents an efficient encoding that optimizes the space used on the Tangle, but allows future additions to the standard. The standard is backwards compatible with the basic JSON metadata to minimize changes for the TIPs concerning token foundries or NFTs.

# Backwards Compatibility

Currently there is no standard for encoding metadata, and each individual standard, such as IRC-27 and IRC-30 states how they encode the metadata field. By introducing this encoding standard, previous stored metadata may contradict the standard.

MES-1 covers the current known metadata standards on the Shimmer network. The only two published standards, to the best of our knowledge, are:
- IRC27
- IRC30

The analysis of these standards resulted in exceptions to the layered encoding approach while also minimizing the bytes used for more common standards.

IRC27 and IRC30 metadata use UTF8/JSON encoding to represent objects in JSON. The JSON standard determines what the initial characters can be. MES-1 supports JSON objects, arrays and strings (i.e., objects are initialized by `{`, arrays by `[` and strings by `"`). The layer 1 encoding reserves the hex value of these starting characters to ensure maximum backwards compatibility.

* `{` is represented by `123` in UTF8, therefore the hex code reserved is `7B`
* `[` is represented by `91` in UTF8, therefore the hex code reserved is `5B`
* `"` is represented by `34` in UTF8, therefore the hex code reserved is `22`

# Examples

## Byte Array

#### Metadata:
`[01010100, 00000110, 01010111, 10000110, 00010110, 11010111, 00000110, 11000110, 01010010, 11100110, 00110110, 11110110, 11010010, 00100010, 11000000, 10100010, 00000010, 00001111]`

#### HEFS

| Layer | Encoding |
| ------------ | -------- |
| 0 - Hex Flag | 0x |
| 1 - Encoding | 00 |
| 2 - Format | 00 |

#### Encoded Metadata
`0x00005406578616d706c652e636f6d222c0a2020f`

## UTF-8 String

#### Metadata

``'Hello World'``

#### HEFS

| Layer | Encoding |
| ------------ | -------- |
| 0 - Hex Flag | 0x |
| 1 - Encoding | 08 |
| 2 - Format | 00 |

#### Encoded Metadata

`0x080048656c6c6f20576f726c64`

## JSON Object

#### Metadata

```json
{
"id": "12345",
"name": "John Doe",
"age": 30,
"email": "johndoe@example.com",
"address": {
"street": "123 Main St",
"city": "Anytown",
"state": "CA",
"zip": "12345"
}
}
```

#### HEFS

| Layer | Encoding |
| ------------ | -------- |
| 0 - Hex Flag | 0x |
| 1/2 - Encoding | 7B |

#### Encoded Metadata

Results in the following metadata:

`0x7b0a2020226964223a20223132333435222c0a2020226e616d65223a20224a6f686e20446f65222c0a202022616765223a2033302c0a202022656d61696c223a20226a6f686e646f65406578616d706c652e636f6d222c0a20202261646472657373223a207b0a2020202022737472656574223a2022313233204d61696e205374222c0a202020202263697479223a2022416e79746f776e222c0a20202020227374617465223a20224341222c0a20202020227a6970223a20223132333435220a20207d0a7d`

## IRC-30 Token Standard

#### Metadata

```json
{
"standard": "IRC30",
"name": "Metadata Token",
"symbol": "MES1",
"decimals": 0
}
```

#### HEFS

| Layer | Encoding |
| ------------ | -------- |
| 0 - Hex Flag | 0x |
| 1/2 - Encoding | 7B |
| 3 - Standard | IRC30 |

#### Encoded Metadata

`0x7b227374616e64617264223a224952433330222c226e616d65223a224d6574616461746120546f6b656e222c2273796d626f6c223a224d455331222c22646563696d616c73223a307d`

## Iota Smart Contract On Chain Request

#### Metadata

```json
{
"senderContract": 0,
"targetContract": 1011572226,
"entryPoint": 603251617,
"params": {
"Items": [
{
"key": "0x61",
"value": "0x03cbcd6d8659ed1998a452335ae53904dc0af1c99b"
}
]
},
"allowance": {
"baseTokens": 25000000,
"nativeTokens": [],
"nfts": []
},
"gasBudget": 500000
}
```

#### HEFS

| Layer | Encoding |
| ------------ | -------- |
| 0 - Hex Flag | 0x |
| 1 - Encoding | 00 |
| 2 - Format | 01 |
| 3 - Standard | 01 |

#### Encoded Metadata

`0x00010100000000025e4b3ca1e3f42320a1070000000000010000000100611500000003cbcd6d8659ed1998a452335ae53904dc0af1c99b0040787d0100000000020000000000`

### [JSON Schema for Layer-3 standards](#json-schema-for-layer-3-standards)

```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/iotaledger/tips/main/tips/TIP-0038/irc38.schema.json",
"title": "JSON Schema to define layer-3 JSON standards",
"description": "JSON Schema to define layer-3 JSON standards",
"type": "object",
"properties": {
"standard": {
"type": "string",
"description": "identifier for the metadata standard used",
},
"version": {
"type": "string",
"description": "version of the standard used",
"pattern": "^v\\d+.\\d+$"
}
},
"required": [ "standard" ]
}
```

0 comments on commit 7c32f44

Please sign in to comment.