The unified multicodecs theory #16
+134
−223
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,15 +4,14 @@ | |
| [](http://github.com/multiformats/multiformats) | ||
| [](http://webchat.freenode.net/?channels=%23ipfs) | ||
|
|
||
| > compact self-describing codecs. Save space by using predefined multicodec tables. | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Motivation](#motivation) | ||
| - [How does it work? - Protocol Description](#how-does-it-work---protocol-description) | ||
| - [Multicodec tables](#multicodec-tables) | ||
| - [Standard multicodec table](#standard-mcp-protocol-table) | ||
| - [Implementations](#implementations) | ||
| - [FAQ](#faq) | ||
| - [Maintainers](#maintainers) | ||
|
|
@@ -21,136 +20,162 @@ | |
|
|
||
| ## Motivation | ||
|
|
||
| [Multistreams](https://github.com/multiformats/multistream) are self-describing protocol/encoding streams. Multicodec uses an agreed-upon "protocol table". It is designed for use in short strings, such as keys or identifiers (i.e [CID](https://github.com/ipld/cid)). | ||
|
|
||
| ## Protocol Description - How does the protocol work? | ||
|
|
||
| `multicodec` is a _self-describing multiformat_, it wraps other formats with a tiny bit of self-description. A multicodec identifier is both a varint and the code identifying the following data, this means that the most significant bit of every multicodec code is reserved to signal the continuation. | ||
|
There was a problem hiding this comment. It describes only the packed format, I think we aren't yet removing the path format, are we? |
||
|
|
||
| This way, a chunk of data identified by multicodec will look like this: | ||
|
|
||
| ```sh | ||
| <multicodec-varint><encoded-data> | ||
| # To reduce the cognitive load, we sometimes might write the same line as: | ||
| <mcp><data> | ||
| ``` | ||
|
|
||
| Another useful scenario is when using the multicodec-packed as part of the keys to access data, example: | ||
|
|
||
| ``` | ||
| # suppose we have a value and a key to retrieve it | ||
| "<key>" -> <value> | ||
|
|
||
| # we can use multicodec-packed with the key to know what codec the value is in | ||
|
|
||
| "<mcp><key>" -> <value> | ||
| ``` | ||
|
|
||
| It is worth noting that multicodec-packed works very well in conjunction with [multihash](https://github.com/multiformats/multihash) and [multiaddr](https://github.com/multiformats/multiaddr), as you can prefix those values with a multicodec-packed to tell what they are. | ||
|
|
||
| ## MulticodecProtocol Tables | ||
|
|
||
| Multicodec uses "protocol tables" to agree upon the mapping from one multicodec code (a single varint). These tables can be application specific, though -- like [with](https://github.com/multiformats/multihash) [other](https://github.com/multiformats/multibase) [multiformats](https://github.com/multiformats/multiaddr) -- we will keep a globally agreed upon table with common protocols and formats. | ||
|
|
||
| ## Multicodec table | ||
|
|
||
| ```csv | ||
| codec, description, code | ||
|
|
||
| miscelaneous | ||
| bin, raw binary, 0x55 | ||
|
|
||
| bases encodings | ||
| base1, unary, 0x01 | ||
| base2, binary (0 and 1), 0x55 | ||
| base8, octal, 0x07 | ||
| base10, decimal, 0x09 | ||
| base16, hexadecimal, 0x | ||
| base32, rfc4648, 0x | ||
| base32hex, rfc4648, 0x | ||
| base58flickr, base58 flicker, 0x | ||
| base58btc, base58 bitcoin, 0x | ||
| base64, rfc4648, 0x | ||
| base64url, rfc4648, 0x | ||
|
There was a problem hiding this comment. The codes of multibase should be the same as https://github.com/multiformats/multibase/blob/master/multibase.csv There was a problem hiding this comment. Not possible, the multibase.csv are chars available in the encodec space of the base, not hex values. There was a problem hiding this comment. IIRC we were removing multibases from multicodec packed. a i misremembering, @diasdavid ? |
||
|
|
||
| serialization formats | ||
| cbor, CBOR, 0x | ||
| bson, Binary JSON, 0x | ||
| ubjson, Universal Binary JSON, 0x | ||
| protobuf, Protocol Buffers, 0x | ||
| capnp, Cap-n-Proto, 0x | ||
| flatbuf, FlatBuffers, 0x | ||
| rlp, recursive length prefix, 0x60 | ||
|
|
||
| multiformats | ||
| multicodec, , 0x30 | ||
| multihash, , 0x31 | ||
| multiaddr, , 0x32 | ||
| multibase, , 0x33 | ||
|
|
||
| multihashes | ||
| sha1, , 0x11 | ||
| sha2-256, , 0x12 | ||
| sha2-512, , 0x13 | ||
| sha3-224, , 0x17 | ||
| sha3-256, , 0x16 | ||
| sha3-384, , 0x15 | ||
| sha3-512, , 0x14 | ||
| shake-128, , 0x18 | ||
| shake-256, , 0x19 | ||
| keccak-224, , 0x1A | ||
| keccak-256, , 0x1B | ||
| keccak-384, , 0x1C | ||
| keccak-512, , 0x1D | ||
| Note: keccak has variable output length, instead the number specifies the core length,, | ||
| blake2b, , 0x40 | ||
| blake2s, , 0x41 | ||
| reserved for apps, appl specific range, 0x4000-0x40f0 | ||
|
|
||
| multiaddrs | ||
| ip4, , 0x04 | ||
| ip6, , 0x29 | ||
| tcp, , 0x06 | ||
| udp, , 0x0111 | ||
| dccp, , 0x21 | ||
| sctp, , 0x84 | ||
| udt, , 0x012D | ||
| utp, , 0x012E | ||
| ipfs, , 0x2A | ||
| http, , 0x01E0 | ||
| https, , 0x01BB | ||
| ws, , 0x01DD | ||
| onion, , 0x01BC | ||
|
|
||
| archiving formats | ||
| tar, , 0x | ||
| zip, , 0x | ||
|
|
||
| image formats | ||
| png, , 0x | ||
| jpg, , 0x | ||
|
|
||
| video formats | ||
| mp4, , 0x | ||
| mkv, , 0x | ||
|
|
||
| IPLD formats | ||
| dag-pb, MerkleDAG protobuf, 0x70 | ||
| dag-cbor, MerkleDAG cbor, 0x71 | ||
| eth-block, Ethereum Block (RLP), 0x90 | ||
|
There was a problem hiding this comment. I don't think these should be above There was a problem hiding this comment. Because that means they take up two bytes when encoded. Anything that is going to be commonly used we would really love to have take only a single byte. There was a problem hiding this comment. I noticed There was a problem hiding this comment. Transport multicodecs are 'more okay' because they get transferred less, in the other hand, format multicodecs get transferred every time a block is transferred, so a byte actually means a lot. There was a problem hiding this comment. @whyrusleeping is this still a concern? I believe it stopped being as soon you merged IPLD in go-ipfs into master, correct? There was a problem hiding this comment. and, there are simply more than 127 things we "really care about". so 2 bytes is in the long run unavoidable. |
||
| eth-tx, Ethereum Tx (RLP), 0x91 | ||
| bitcoin-block, Bitcoin Block, 0xb0 | ||
| bitcoin-tx, Bitcoin Tx, 0xb1 | ||
| stellar-block, Stellar Block, 0xd0 | ||
| stellar-tx, Stellar Tx, 0xd1 | ||
| ``` | ||
|
There was a problem hiding this comment. this table should move to a CSV, and the readme should point to it or embed it. |
||
|
|
||
| ## Implementations | ||
|
|
||
| - [go](https://github.com/multiformats/go-multicodec/) | ||
| - [JavaScript](https://github.com/multiformats/js-multicodec) | ||
| - [Add yours today!](https://github.com/multiformats/multicodec/edit/master/multicodec.md) | ||
|
|
||
| ## Multicodec Path, also known as [`multistream`](https://github.com/multiformats/multistream) | ||
|
|
||
| Multicodec defines a table for the most common data serialization formats that can be expanded overtime or per application bases, however, in order for two programs to talk with each other, they need to know before hand which table or table extension is being used. | ||
|
|
||
| In order to enable self descriptive data formats or streams that can be dynamically described, without the formal set of adding a binary packed code to a table, we have [`multistream`](https://github.com/multiformats/multistream), so that applications can adopt multiple data formats for their streams and with that create different protocols. | ||
|
|
||
| ## FAQ | ||
|
|
||
| > **Q. I have questions on multicodec, not listed here.** | ||
|
|
||
| That's not a question. But, have you checked the proper [multicodec FAQ](./README.md#faq)? Maybe your question is answered there. This FAQ is only specifically for multicodec-packed. | ||
|
|
||
| > **Q. Why?** | ||
|
|
||
| Because [multistream](https://github.com/multiformats/multistream) is too long for identifiers. We needed something shorter. | ||
|
|
||
| > **Q. Why varints?** | ||
|
|
||
| So that we have no limitation on protocols. Implementation note: you do not need to implement varints until the standard multicodec table has more than 127 functions. | ||
|
There was a problem hiding this comment. yes, we probably need to remove this, and add multicodec impls to all the use-case specific multiformats IF their table requires it. |
||
|
|
||
| > **Q. What kind of varints?** | ||
|
|
||
| An Most Significant Bit unsigned varint, as defined by the [multiformats/unsigned-varint](https://github.com/multiformats/unsigned-varint). | ||
|
|
||
| > **Q. Don't we have to agree on a table of protocols?** | ||
|
|
||
| Yes, but we already have to agree on what protocols themselves are, so this is not so hard. The table even leaves some room for custom protocol paths, or you can use your own tables. The standard table is only for common things. | ||
|
|
||
| ## Maintainers | ||
|
|
||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do all protocols need to be in 'agreed upon "protocol table"' or only those using packed format?