Make chainHead_unstable_storage more powerful

[tomaka]

Close #21

Even though they're unrelated, this PR is based on top of #35. I expect #35 to be merged, and didn't want to have conflicts.

This PR enhances chainHead_unstable_storage by adding back the type field that was removed in #17.
You can now query not just the value of a storage item, but also the hash of its value, the values of the item and all its descendants, the hashes of item and all its descendants, or the Merkle value of the item or its closest ancestor.

Querying the descendants of the storage item makes it possible to enumerate all the keys under a certain prefix, which is important to know the content of maps.
In order to reproduce something similar to state_getKeysPaged, in the case when you enumerate descendants the server can now generate a waiting-for-continue event, after which the client must call chainHead_unstable_storageContinue in order to resume the enumeration. This makes it possible for the client to interrupt the enumeration at any point in time. Rather than letting the client decide of the number of items (like is the case for state_getKeysPaged), it's now the server which decides, which makes it possible to decide to generate this event based on the size of what was sent rather than the number of items.

The closest-ancestor-merkle-value query type makes it possible to know when the content of a map has changed. It can be seen as similar to hash, except that hash is only the hash of the value while closest-ancestor-merkle-value is the hash of the value plus the hash of all the descendants in the trie.
Basically, in order to watch a map, a JSON-RPC client would periodically query the closest-ancestor-merkle-value of the map, then redownload the list of keys if this Merkle value changes.

[lexnv]

Amazing!

This should give us parity between what substrate exposed previously wrt iterating over key storages, while having a more compact and robust API.

Hopefully, we'll be able to make a complete transition in the whole ecosystem to the new methods and deprecate the old RPC ones 👍

[tomaka]

I'm actually not completely happy with this design where the client has to call storageContinue.

The function achieves all its goals, but the problem is that the client can delay calling storageContinue indefinitely. There exists a limit to the number of active subscriptions, so I'm not worried about DoS attacks or anything like that.

What I'm worried about it a very specific situation: if a load balancer redirects a storage query to server A, and later wants to shut down server A because it is no longer needed or needs maintenance, the load balancer currently can generate a stop event for all active follow subscriptions, wait for all active requests to finish, then shut down server A.

But with the introduction of this storageContinue thing, the client can prevent this from happening by keeping a storage query active indefinitely.
I feel like there should be something similar to stop that the load balancer can generate. The error event currently means "fatal irrecoverable error", which isn't the case here as the client should simply try again. I think that inaccessible is appropriate, because the load balancer can now longer access the node in question.

[tomaka]

Since it's sitting there, I just wanted to mention that I can't merge this pull request myself as I'm not at Parity anymore.

[lexnv]

LGTM!

// cc @skunert @jsdw @bkchr

[bkchr]

If I understand it correctly, I find the naming "hard/confusing". Why not "closest-ancestor-merkle-node" and then also saying that this is the hash of the closest merkle node.

[tomaka]

I kept the same vocabulary as the spec. There's no such thing as a "merkle node". There are "trie nodes" that have a "node value" and a "merkle value" (the merkle value is either the node value or its hash).

"closest ancestor merkle value" means "the merkle value of the closest ancestor".

[bkchr]

I kept the same vocabulary as the spec.

Okay 👍

(the merkle value is either the node value or its hash).

What is the node value? The value attached to a node? Shouldn't we always be interested in the hash as we only use this for tracking changes?

[tomaka]

The node value is (basically) concat(node_type, partial_key, storage_value, merkle_values_of_children).

If the node value is less than 32 bytes then the Merkle value is the same as the node value. If the node value is 32 bytes or more, then it's the hash.
https://spec.polkadot.network/chap-state#defn-merkle-value

[bkchr]

Ahh okay, ty for the explanation 👍