Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Basic Proving Trie for the Runtime #3881

Merged
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
d50d5e4
initial ideas
shawntabrizi Mar 28, 2024
7f4422e
Update proving_trie.rs
shawntabrizi Mar 28, 2024
bdc0c84
create trait
shawntabrizi Mar 28, 2024
da385ab
use trait
shawntabrizi Mar 28, 2024
6ab4bfb
clean up trait and basic trie further
shawntabrizi Mar 28, 2024
1cfb29f
use trait in session historical
shawntabrizi Mar 28, 2024
3080c7b
fix api, add basic end to end test
shawntabrizi Mar 28, 2024
39b0fca
fix test
shawntabrizi Mar 28, 2024
599f576
Revert "use trait in session historical"
shawntabrizi Mar 28, 2024
a1c8886
Merge branch 'master' into shawntabrizi-proving-trie
shawntabrizi Mar 29, 2024
6bde2a3
Merge branch 'master' into shawntabrizi-proving-trie
Ank4n Apr 4, 2024
ead7951
Update substrate/primitives/runtime/src/proving_trie.rs
shawntabrizi Apr 4, 2024
0cac048
Merge branch 'master' into shawntabrizi-proving-trie
shawntabrizi Aug 15, 2024
acfe734
fix some feedback
shawntabrizi Aug 15, 2024
45f4287
update name
shawntabrizi Aug 15, 2024
efbe1c9
docs and multi value proof
shawntabrizi Aug 15, 2024
cfa62ce
improve test
shawntabrizi Aug 15, 2024
885fd94
add multi-value query test
shawntabrizi Aug 15, 2024
18a5c2e
Merge branch 'master' into shawntabrizi-proving-trie
shawntabrizi Aug 15, 2024
5e3e518
Create pr_3881.prdoc
shawntabrizi Aug 15, 2024
f35cacc
Merge branch 'shawntabrizi-proving-trie' of https://github.com/shawnt…
shawntabrizi Aug 15, 2024
e6c3759
use v1
shawntabrizi Aug 17, 2024
89679c3
Merge branch 'master' into shawntabrizi-proving-trie
shawntabrizi Aug 17, 2024
6e853ba
Merge branch 'master' into shawntabrizi-proving-trie
shawntabrizi Sep 2, 2024
00a7aa9
Merge remote-tracking branch 'upstream/master' into shawntabrizi-prov…
shawntabrizi Sep 3, 2024
ffbd656
use compact trie apis
shawntabrizi Sep 3, 2024
0217d0f
Update proving_trie.rs
shawntabrizi Sep 3, 2024
f20038a
more docs
shawntabrizi Sep 3, 2024
f934d8d
bad data test
shawntabrizi Sep 3, 2024
c92bd6a
Merge branch 'master' into shawntabrizi-proving-trie
shawntabrizi Sep 3, 2024
266a89b
additional warnings about layout
shawntabrizi Sep 3, 2024
3034aab
Update prdoc/pr_3881.prdoc
shawntabrizi Sep 3, 2024
1ceab64
remove lifetime
shawntabrizi Sep 3, 2024
c426a65
remove unused
shawntabrizi Sep 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions substrate/primitives/runtime/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ sp-arithmetic = { path = "../arithmetic", default-features = false }
sp-core = { path = "../core", default-features = false }
sp-io = { path = "../io", default-features = false }
sp-std = { path = "../std", default-features = false }
sp-trie = { path = "../trie", default-features = false }
sp-weights = { path = "../weights", default-features = false }
docify = { version = "0.2.7" }

Expand Down Expand Up @@ -66,6 +67,7 @@ std = [
"sp-state-machine/std",
"sp-std/std",
"sp-tracing/std",
"sp-trie/std",
"sp-weights/std",
]

Expand Down
1 change: 1 addition & 0 deletions substrate/primitives/runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ pub mod generic;
pub mod legacy;
mod multiaddress;
pub mod offchain;
pub mod proving_trie;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not too sure if it should be part of sp-runtime crate.
Thing is the choice of using this trie should not be a default for any runtime (I think it is not the best tree for most use case, and I would encourage more binary trie, anyway it is very use case specific).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a binary trie available in Substrate?

Does this trie work along side child tries?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the proof is to be run on child trie, there is no choice but to use this trie indeed.
(if using child trie root to prove, please be careful the chain will not be switching from trie V0 to trie V1 (should not matter for crowdloan as iirc the values are <= 32 byte), as a migration may change child trie root.
About this trie version, the simpliest would probably be to just pass the version in parameter.
When I read the PR I did not have this use case in mind but it is true that if what we are proving is from substrate state, then it could make sense to have this in runtime.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can I name this ProvingTrieBase16, and that would be better?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not too sure anymore what was my point 🤦
Probably at first reading I was thinking it should be an external crate, then realizing it is being tightly linked with substrate state could actually be fine.
ProvingTrieBase16 may not be a good idea as long as substrate does have a single trie backend.
Maybe it could simple be a module in sp-trie, but guess I am ok with things being this way.

pub mod runtime_logger;
mod runtime_string;
#[cfg(feature = "std")]
Expand Down
178 changes: 178 additions & 0 deletions substrate/primitives/runtime/src/proving_trie.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
// This file is part of Substrate.

// Copyright (C) Parity Technologies (UK) Ltd.
// SPDX-License-Identifier: Apache-2.0

// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

//! Types for a simple merkle trie used for checking and generating proofs.

use crate::{Decode, DispatchError, Encode};

use sp_std::vec::Vec;
use sp_trie::{
trie_types::{TrieDBBuilder, TrieDBMutBuilderV0},
LayoutV0, MemoryDB, Recorder, Trie, TrieMut, EMPTY_PREFIX,
};

/// A trait for creating a merkle trie for checking and generating merkle proofs.
pub trait ProvingTrie<Hashing, Hash, Key, Value>
shawntabrizi marked this conversation as resolved.
Show resolved Hide resolved
shawntabrizi marked this conversation as resolved.
Show resolved Hide resolved
shawntabrizi marked this conversation as resolved.
Show resolved Hide resolved
where
Self: Sized,
{
/// Create a new instance of a `ProvingTrie` using an iterator of key/value pairs.
fn generate_for<I>(items: I) -> Result<Self, DispatchError>
where
I: IntoIterator<Item = (Key, Value)>;
/// Access the underlying trie root.
fn root(&self) -> &Hash;
/// Check a proof contained within the current memory-db. Returns `None` if the
/// nodes within the current `MemoryDB` are insufficient to query the item.
fn query(&self, key: Key) -> Option<Value>;
/// Create the full verification data needed to prove a key and its value in the trie. Returns
/// `None` if the nodes nodes within the current `MemoryDB` are insufficient to create a proof.
shawntabrizi marked this conversation as resolved.
Show resolved Hide resolved
fn create_proof(&self, key: Key) -> Option<Vec<Vec<u8>>>;
/// Create a new instance of `ProvingTrie` from raw nodes. Nodes can be generated using the
/// `create_proof` function.
fn from_nodes(root: Hash, nodes: &[Vec<u8>]) -> Self;
}

/// A basic trie implementation for checking and generating proofs for a key / value pair.
pub struct BasicProvingTrie<Hashing, Hash, Key, Value>
where
Hashing: sp_core::Hasher<Out = Hash>,
{
db: MemoryDB<Hashing>,
root: Hash,
_phantom: core::marker::PhantomData<(Key, Value)>,
}

impl<Hashing, Hash, Key, Value> ProvingTrie<Hashing, Hash, Key, Value>
for BasicProvingTrie<Hashing, Hash, Key, Value>
where
Hashing: sp_core::Hasher<Out = Hash>,
Hash: Default + Send + Sync,
Key: Encode,
Value: Encode + Decode,
{
fn generate_for<I>(items: I) -> Result<Self, DispatchError>
where
I: IntoIterator<Item = (Key, Value)>,
{
let mut db = MemoryDB::default();
let mut root = Default::default();

{
let mut trie = TrieDBMutBuilderV0::new(&mut db, &mut root).build();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cheme why do we use a specific version here? Should this not be V1?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general the version does not matter much for proof as V0 can be use on a V1 trie and V1 can be use on a V0, and since it is a proof we mainly care for access node (both version should have same access patterns).
In this case though, because we return the root, we need to choose the right version (proof content will be the same but root will differ).
In the case of historical, I think the value where all 32 bytes so V0 and V1 would work the same, but I kept V0 to be conservative.
For a generic toolling, the version should be pass in parameter (even if in most case there is no reason to use V0).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you want me to make this a parameter we pass in? or make it V1, or leave it untouched?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either V1 only (would not be compatible in some case) or pass it as a parameter.

for (key, value) in items.into_iter() {
key.using_encoded(|k| value.using_encoded(|v| trie.insert(k, v)))
.map_err(|_| "failed to insert into trie")?;
}
}

Ok(Self { db, root, _phantom: Default::default() })
}

fn root(&self) -> &Hash {
&self.root
}

fn query(&self, key: Key) -> Option<Value> {
let trie = TrieDBBuilder::new(&self.db, &self.root).build();
key.using_encoded(|s| trie.get(s))
.ok()?
.and_then(|raw| Value::decode(&mut &*raw).ok())
}

fn create_proof(&self, key: Key) -> Option<Vec<Vec<u8>>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better passing Vec and allow call multiple query here.
For now we can simply rename to create_single_value_access_proof I think.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated name and also added the multi-value API

let mut recorder = Recorder::<LayoutV0<Hashing>>::new();
shawntabrizi marked this conversation as resolved.
Show resolved Hide resolved

{
let trie =
TrieDBBuilder::new(&self.db, &self.root).with_recorder(&mut recorder).build();

key.using_encoded(|k| {
trie.get(k).ok()?.and_then(|raw| Value::decode(&mut &*raw).ok())
})?;
}

Some(recorder.drain().into_iter().map(|r| r.data).collect())
}

fn from_nodes(root: Hash, nodes: &[Vec<u8>]) -> Self {
cheme marked this conversation as resolved.
Show resolved Hide resolved
use sp_trie::HashDBT;

let mut memory_db = MemoryDB::default();
for node in nodes {
HashDBT::insert(&mut memory_db, EMPTY_PREFIX, &node[..]);
}

Self { db: memory_db, root, _phantom: Default::default() }
}
}

#[cfg(test)]
mod tests {
use super::*;
use crate::traits::BlakeTwo256;
use sp_core::H256;
use sp_std::{collections::btree_map::BTreeMap, str::FromStr};

// A trie which simulates a trie of accounts (u32) and balances (u128).
type BalanceTrie = BasicProvingTrie<BlakeTwo256, H256, u32, u128>;

// The expected root hash for an empty trie.
fn empty_root() -> H256 {
H256::from_str("0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314")
.unwrap()
}

#[test]
fn empty_trie_works() {
let empty_trie = BalanceTrie::generate_for(Vec::new()).unwrap();
assert_eq!(*empty_trie.root(), empty_root());
}

#[test]
fn basic_end_to_end() {
// Create a map of users and their balances.
let mut map = BTreeMap::<u32, u128>::new();
for i in 0..10u32 {
map.insert(i, i.into());
}

// Put items into the trie.
let balance_trie = BalanceTrie::generate_for(map).unwrap();

// Root is changed.
let root = *balance_trie.root();
assert!(root != empty_root());

// Assert valid key is queryable.
assert_eq!(balance_trie.query(6u32), Some(6u128));
assert_eq!(balance_trie.query(9u32), Some(9u128));
// Invalid key returns none.
assert_eq!(balance_trie.query(69u32), None);

// Create a proof for a valid key.
let proof = balance_trie.create_proof(6u32).unwrap();
// Can't create proof for invalid key.
assert_eq!(balance_trie.create_proof(69u32), None);

// Create a new proving trie from the proof.
let new_balance_trie = BalanceTrie::from_nodes(root, &proof);

// Assert valid key is queryable.
assert_eq!(new_balance_trie.query(6u32), Some(6u128));
}
}
Loading