From d98e64c2ea06b575a23c8e16a6148d61a94e9e71 Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Thu, 11 May 2023 12:35:13 +0200 Subject: [PATCH 1/9] Clarify the childTrie parameter of chainHead_unstable_storage --- src/api/chainHead_unstable_storage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index 0a2daa5..7d7808f 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -5,12 +5,12 @@ - `followSubscription`: An opaque string that was returned by `chainHead_unstable_follow`. - `hash`: String containing an hexadecimal-encoded hash of the header of the block whose storage to fetch. - `key`: String containing the hexadecimal-encoded key to fetch in the storage. -- `childKey`: `null` for main storage look-ups, or a string containing the hexadecimal-encoded key of the trie key of the trie that `key` refers to. **TODO**: I don't know enough about child tries to design this properly +- `childTrie`: `null` for main storage look-ups, or a string containing the hexadecimal-encoded key of the child trie of the "default" namespace. - `networkConfig` (optional): Object containing the configuration of the networking part of the function. See [here](./api.md) for details. Ignored if the JSON-RPC server doesn't need to perform a network request. Sensible defaults are used if not provided. **Return value**: String containing an opaque value representing the operation. -The JSON-RPC server must start obtaining the value of the entry with the given `key` (and possibly `childKey`) from the storage. +The JSON-RPC server must start obtaining the value of the entry with the given `key` from the storage, either from the main trie of from `childTrie`. The operation will continue even if the given block is unpinned while it is in progress. From e540aed139200a1a5a494de1075867e39cbb70e4 Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Mon, 15 May 2023 14:01:28 +0200 Subject: [PATCH 2/9] Make `chainHead_unstable_storage` more powerful --- src/api/chainHead_unstable_storage.md | 66 ++++++++++++++++--- src/api/chainHead_unstable_storageContinue.md | 13 ++++ 2 files changed, 71 insertions(+), 8 deletions(-) create mode 100644 src/api/chainHead_unstable_storageContinue.md diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index 7d7808f..f622c87 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -6,11 +6,12 @@ - `hash`: String containing an hexadecimal-encoded hash of the header of the block whose storage to fetch. - `key`: String containing the hexadecimal-encoded key to fetch in the storage. - `childTrie`: `null` for main storage look-ups, or a string containing the hexadecimal-encoded key of the child trie of the "default" namespace. +- `type`: String equal to one of: `value`, `hash`, `closest-ancestor-merkle-value`, `descendants-values`, `descendants-hashes`. - `networkConfig` (optional): Object containing the configuration of the networking part of the function. See [here](./api.md) for details. Ignored if the JSON-RPC server doesn't need to perform a network request. Sensible defaults are used if not provided. **Return value**: String containing an opaque value representing the operation. -The JSON-RPC server must start obtaining the value of the entry with the given `key` from the storage, either from the main trie of from `childTrie`. +The JSON-RPC server must start obtaining the value of the entry with the given `key` from the storage, either from the main trie of from `childTrie`. If `type` is `descendants-value` or `descendants-hashes`, then it must also obtain the values of all the descendants of the entry. The operation will continue even if the given block is unpinned while it is in progress. @@ -18,6 +19,14 @@ This function should be seen as a complement to `chainHead_unstable_follow`, all For optimization purposes, the JSON-RPC server is allowed to wait a little bit (e.g. up to 100ms) before starting to try fulfill the storage request, in order to batch multiple storage requests together. +One `{"event": "item"}` notification will be generated for each value found in the storage. If `type` is `value` or `hash`, then either 0 or 1 `"item"` notification will be generated. If `type` is `closest-ancestor-merkle-value` then exactly 1 `"item"` notification will be generated. If `type` is `descendants-values` or `descendants-hashes`, then one `"item"` notifications that will be generated for each descendant of the `key` (including the `key` itself). + +If `type` is `hash` or `descendants-hashes`, then the cryptographic hash of each item is provided rather than the full value. The hashing algorithm used is the one of the chain, which is typically blake2b. This can lead to significantly less bandwidth usage and can be used in order to compare the value of an item with a known hash and querying the full value only if it differs. + +If `type` is `closest-ancestor-merkle-value`, then the so-called trie Merkle value of the `key` is provided. If `key` doesn't exist in the trie, then the Merkle value of the closest ancestor of `key` is provided. Contrary to `hash`, a `closest-ancestor-merkle-value` always exists for every `key`. The Merkle value is similar to a hash of the value and all of its descendants together. + +If a `{"event": "waiting-for-continue"}` notification is generated, the subscription will not generate any more notification unless the JSON-RPC client calls the `chainHead_unstable_storageContinue` JSON-RPC function. The JSON-RPC server is encouraged to generate this event after having sent a certain number of bytes to the JSON-RPC client in order to avoid head-of-line-blocking issues. + ## Notifications format This function will later generate notifications in the following format: @@ -35,18 +44,56 @@ This function will later generate notifications in the following format: Where `subscription` is equal to the value returned by this function, and `result` can be one of: +### item + +```json +{ + "event": "item", + "key": "0x0000000...", + "value": "0x0000000...", + "hash": "0x0000000...", + "closest-ancestor-merkle-value": "0x000000...", +} +``` + +Yields an item that was found in the storage. + +The `key` field is a string containing the hexadecimal-encoded key of the value that was found. +If the `type` parameter was `"value"`, `"hash"`, `"descendants-values"` or `"descendants-hashes"`, this `key` is guaranteed to start with the `key` provided as parameter. +If the `type` parameter was `"value"` or `"hash"`, then it is also guaranteed to be equal to the `key` provided as parameter. +If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `key` provided as parameter is guaranteed to start with the value in the `key` field. + +If the `type` parameter was `"value"` or `"descendants-values"`, then the `value` field is set. The `value` field a string containing the hexadecimal-encoded value of the storage entry. + +If the `type` parameter was `"hash"` or `"descendants-hashes"`, then the `hash` field is set. The `hash` field a string containing the hexadecimal-encoded hash of the storage entry. + +If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `closest-ancestor-merkle-value` field is set. The `closest-ancestor-merkle-value` field a string containing the hexadecimal-encoded Merkle value of the storage entry or its closest ancestor. + +Only one of `value`, `hash` or `closest-ancestor-merkle-value` are set at any given time. + +### waiting-for-continue + +```json +{ + "event": "waiting-for-continue" +} +``` + +The `waiting-for-continue` event is generated after at least one `"item"` event has been generated, and indicates that the JSON-RPC client must call `chainHead_unstable_storageContinue` before more events are generated. + +This event only ever happens if the `type` parameter was `descendants-values` or `descendants-hashes`. + ### done ```json { - "event": "done", - "value": "0x0000000..." + "event": "done" } ``` -The `done` event indicates that everything went well. The `value` field contains the requested value. +The `done` event indicates that everything went well and all values have been provided through `item` events in the past. -`value` is either `null` if the storage doesn't contain a value at the given key, or a string containing the hexadecimal-encoded value of the storage entry. +If no `item` event was yielded, then the storage doesn't contain a value at the given key. No more event will be generated with this `subscription`. @@ -71,13 +118,14 @@ No more event will be generated with this `subscription`. } ``` -The `error` event indicates a problem during the storage access, such as failing to parse the block header to obtain the state root hash. +The `error` event indicates a problem during the storage access, such as failing to parse the block header to obtain the state root hash or the trie being empty when `type` was `closest-ancestor-merkle-value`. Contrary to the `inaccessible` event, querying the same storage value again in the future will not succeed. `error` is a human-readable error message indicating why the call has failed. This string isn't meant to be shown to end users, but is for developers to understand the problem. -No more event will be generated with this `subscription`. +This can only be the first event generated by this subscription. +No other event will be generated with this subscription. ### disjoint @@ -89,7 +137,8 @@ No more event will be generated with this `subscription`. The `disjoint` event indicates that the provided `followSubscription` is invalid or stale. -No more event will be generated with this `subscription`. +This can only be the first event generated by this subscription. +No other event will be generated with this subscription. ## Possible errors @@ -98,3 +147,4 @@ No more event will be generated with this `subscription`. - If the `followSubscription` is invalid or stale, then a `{"event": "disjoint"}` notification is generated (as explained above). - A JSON-RPC error is generated if the block hash passed as parameter doesn't correspond to any block that has been reported by `chainHead_unstable_follow`. - A JSON-RPC error is generated if the `followSubscription` is valid but the block hash passed as parameter has already been unpinned. +- If the trie is empty and `type` is `closest-ancestor-merkle-value`, then a `{"event": "error"}`. diff --git a/src/api/chainHead_unstable_storageContinue.md b/src/api/chainHead_unstable_storageContinue.md new file mode 100644 index 0000000..df07dea --- /dev/null +++ b/src/api/chainHead_unstable_storageContinue.md @@ -0,0 +1,13 @@ +# chainHead_unstable_storageContinue + +**Parameters**: + +- `subscription`: An opaque string that was returned by `chainHead_unstable_storage`. + +**Return value**: *null* + +Resumes a storage fetch started with `chainHead_unstable_storage`. If the storage fetch was still in progress, this interrupts it. If the storage fetch was already finished, this call has no effect. + +## Possible errors + +- A JSON-RPC error is generated if the `subscription` is invalid or hasn't generated a `waiting-for-continue` event. From 3b20fb1b22a8f38d5decfcf6a804f4941e15272e Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 10:22:56 +0200 Subject: [PATCH 3/9] Fix `chainHead_unstable_storageContinue` text --- src/api/chainHead_unstable_storageContinue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/api/chainHead_unstable_storageContinue.md b/src/api/chainHead_unstable_storageContinue.md index df07dea..638608e 100644 --- a/src/api/chainHead_unstable_storageContinue.md +++ b/src/api/chainHead_unstable_storageContinue.md @@ -6,7 +6,7 @@ **Return value**: *null* -Resumes a storage fetch started with `chainHead_unstable_storage`. If the storage fetch was still in progress, this interrupts it. If the storage fetch was already finished, this call has no effect. +Resumes a storage fetch started with `chainHead_unstable_storage` after it has generated a `waiting-for-continue` event. ## Possible errors From eb7f7f43bad02bbf27f32566ba4ec8db8a8a7429 Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 10:30:10 +0200 Subject: [PATCH 4/9] The field is now just `merkle-value` rather than `closest-ancestor-merke-value` --- src/api/chainHead_unstable_storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index f622c87..a268d60 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -52,7 +52,7 @@ Where `subscription` is equal to the value returned by this function, and `resul "key": "0x0000000...", "value": "0x0000000...", "hash": "0x0000000...", - "closest-ancestor-merkle-value": "0x000000...", + "merkle-value": "0x000000...", } ``` @@ -67,9 +67,9 @@ If the `type` parameter was `"value"` or `"descendants-values"`, then the `value If the `type` parameter was `"hash"` or `"descendants-hashes"`, then the `hash` field is set. The `hash` field a string containing the hexadecimal-encoded hash of the storage entry. -If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `closest-ancestor-merkle-value` field is set. The `closest-ancestor-merkle-value` field a string containing the hexadecimal-encoded Merkle value of the storage entry or its closest ancestor. +If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `merkle-value` field is set and the `key` field indicates which closest ancestor has been found. The `merkle-value` field a string containing the hexadecimal-encoded Merkle value of the storage entry or its closest ancestor. -Only one of `value`, `hash` or `closest-ancestor-merkle-value` are set at any given time. +Only one of `value`, `hash` or `merkle-value` are set at any given time. ### waiting-for-continue From 3313139a90ba57a2158b0f403bcdafa70ad5f0c3 Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 10:31:22 +0200 Subject: [PATCH 5/9] Text tweak --- src/api/chainHead_unstable_storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index a268d60..9cfcc62 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -67,7 +67,7 @@ If the `type` parameter was `"value"` or `"descendants-values"`, then the `value If the `type` parameter was `"hash"` or `"descendants-hashes"`, then the `hash` field is set. The `hash` field a string containing the hexadecimal-encoded hash of the storage entry. -If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `merkle-value` field is set and the `key` field indicates which closest ancestor has been found. The `merkle-value` field a string containing the hexadecimal-encoded Merkle value of the storage entry or its closest ancestor. +If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `merkle-value` field is set and the `key` field indicates which closest ancestor has been found. The `merkle-value` field a string containing the hexadecimal-encoded Merkle value of the storage item indicated by the `key` field. Only one of `value`, `hash` or `merkle-value` are set at any given time. From 7f949f916bf7e18be442b0333a9f60608f9c6a8a Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 11:20:41 +0000 Subject: [PATCH 6/9] Update src/api/chainHead_unstable_storage.md Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> --- src/api/chainHead_unstable_storage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index 9cfcc62..fd9faf5 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -63,11 +63,11 @@ If the `type` parameter was `"value"`, `"hash"`, `"descendants-values"` or `"des If the `type` parameter was `"value"` or `"hash"`, then it is also guaranteed to be equal to the `key` provided as parameter. If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `key` provided as parameter is guaranteed to start with the value in the `key` field. -If the `type` parameter was `"value"` or `"descendants-values"`, then the `value` field is set. The `value` field a string containing the hexadecimal-encoded value of the storage entry. +If the `type` parameter was `"value"` or `"descendants-values"`, then the `value` field is set. The `value` field is a string containing the hexadecimal-encoded value of the storage entry. -If the `type` parameter was `"hash"` or `"descendants-hashes"`, then the `hash` field is set. The `hash` field a string containing the hexadecimal-encoded hash of the storage entry. +If the `type` parameter was `"hash"` or `"descendants-hashes"`, then the `hash` field is set. The `hash` field is a string containing the hexadecimal-encoded hash of the storage entry. -If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `merkle-value` field is set and the `key` field indicates which closest ancestor has been found. The `merkle-value` field a string containing the hexadecimal-encoded Merkle value of the storage item indicated by the `key` field. +If the `type` parameter was `"closest-ancestor-merkle-value"`, then the `merkle-value` field is set and the `key` field indicates which closest ancestor has been found. The `merkle-value` field is a string containing the hexadecimal-encoded Merkle value of the storage item indicated by the `key` field. Only one of `value`, `hash` or `merkle-value` are set at any given time. From 367d64c7f042958a86b573cee6654bb7154c106a Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 11:20:51 +0000 Subject: [PATCH 7/9] Update src/api/chainHead_unstable_storage.md Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com> --- src/api/chainHead_unstable_storage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index fd9faf5..18b03ee 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -11,7 +11,7 @@ **Return value**: String containing an opaque value representing the operation. -The JSON-RPC server must start obtaining the value of the entry with the given `key` from the storage, either from the main trie of from `childTrie`. If `type` is `descendants-value` or `descendants-hashes`, then it must also obtain the values of all the descendants of the entry. +The JSON-RPC server must start obtaining the value of the entry with the given `key` from the storage, either from the main trie or from `childTrie`. If `type` is `descendants-value` or `descendants-hashes`, then it must also obtain the values of all the descendants of the entry. The operation will continue even if the given block is unpinned while it is in progress. From 98ca08268c3edc5d060ed3e6c869421bdfba3919 Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 14:24:34 +0200 Subject: [PATCH 8/9] Generate `inaccessible` while waiting if necessary --- src/api/chainHead_unstable_storage.md | 2 ++ src/api/chainHead_unstable_storageContinue.md | 4 +++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/api/chainHead_unstable_storage.md b/src/api/chainHead_unstable_storage.md index 18b03ee..df93a3c 100644 --- a/src/api/chainHead_unstable_storage.md +++ b/src/api/chainHead_unstable_storage.md @@ -83,6 +83,8 @@ The `waiting-for-continue` event is generated after at least one `"item"` event This event only ever happens if the `type` parameter was `descendants-values` or `descendants-hashes`. +While the JSON-RPC server is waiting for a call to `chainHead_unstable_storageContinue`, it can generate an `inaccessible` event in order to indicate that it can no longer proceed with the request. If that is the case, the JSON-RPC client can simply try again. + ### done ```json diff --git a/src/api/chainHead_unstable_storageContinue.md b/src/api/chainHead_unstable_storageContinue.md index 638608e..77d01c4 100644 --- a/src/api/chainHead_unstable_storageContinue.md +++ b/src/api/chainHead_unstable_storageContinue.md @@ -8,6 +8,8 @@ Resumes a storage fetch started with `chainHead_unstable_storage` after it has generated a `waiting-for-continue` event. +Has no effect if the `subscription` is invalid or refers to a subscription that has emitted a `{"event": "inaccessible"}` event. + ## Possible errors -- A JSON-RPC error is generated if the `subscription` is invalid or hasn't generated a `waiting-for-continue` event. +- A JSON-RPC error is generated if the `subscription` hasn't generated a `waiting-for-continue` event. From c0f25acc3a5c0bdcd7b9abf04c4b86f49542f427 Mon Sep 17 00:00:00 2001 From: Pierre Krieger Date: Tue, 16 May 2023 14:29:29 +0200 Subject: [PATCH 9/9] Clarify continue error --- src/api/chainHead_unstable_storageContinue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/api/chainHead_unstable_storageContinue.md b/src/api/chainHead_unstable_storageContinue.md index 77d01c4..0751ed3 100644 --- a/src/api/chainHead_unstable_storageContinue.md +++ b/src/api/chainHead_unstable_storageContinue.md @@ -12,4 +12,4 @@ Has no effect if the `subscription` is invalid or refers to a subscription that ## Possible errors -- A JSON-RPC error is generated if the `subscription` hasn't generated a `waiting-for-continue` event. +- A JSON-RPC error is generated if the `subscription` is valid but hasn't generated a `waiting-for-continue` event.