-
Notifications
You must be signed in to change notification settings - Fork 219
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(consensus)!: add balanced binary merkle tree (#5189)
Description --- Implement balanced binary merkle tree and use it instead of mmr for vns. It has no effect on the base layer functionality apart for being a breaking change (it changes the vn merkle root). In the base layer only the generation is used. The proofs are used only in `tari-dan` (signing votes). The advantages of MMR over binary tree is the time complexity when adding an element. But that's not our case. We regenerate the tree everytime. So it's much simpler to generate balanced binary tree. Which is the best option for this. The proof time/size for binary tree is always log n. For MMR the worst case scenario is 2*log n (when there is peak of every possible height). The merged proofs are part of another PR. Some comparison between balanced binary merkle tree vs merkle mountain range. The data set I used was number of VNs of 1000 to 1M (step by 1000), committee size from 10 to 200 (step by 10) (In reality the committee can be bigger, this is representing the number of proofs). For generating the proofs the random subset of VNs were selected (same for BMT and MMR). - **Tree generation takes 62.03% of the time**  - **Generating proofs takes 42.37% of the time** **Generating merged proofs takes 85.96% of the time** In the graph below the green represents only the merging time. But in reality we need to generate the proof anyway and then merge them (figure above takes this into account)  - **Verifying proofs takes 68.98% of the time** **Verifying merged proofs takes 54.77% of the time**  - **Proof size is 76.12% of the size** **Merged proof size is 9.47% of the size**  How Has This Been Tested? --- There are couple cargo tests. <!-- Does this include a breaking change? If so, include this line as a footer --> <!-- BREAKING CHANGE: Description what the user should do, e.g. delete a database, resync the chain --> BREAKING CHANGE: **The Merkle root for VNs is different.** It's equal in the genesis block, but once there is a single vn registration it starts to differ.
- Loading branch information
Showing
9 changed files
with
274 additions
and
32 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| // Copyright 2019. The Tari Project | ||
| // | ||
| // Redistribution and use in source and binary forms, with or without modification, are permitted provided that the | ||
| // following conditions are met: | ||
| // | ||
| // 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following | ||
| // disclaimer. | ||
| // | ||
| // 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the | ||
| // following disclaimer in the documentation and/or other materials provided with the distribution. | ||
| // | ||
| // 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote | ||
| // products derived from this software without specific prior written permission. | ||
| // | ||
| // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, | ||
| // INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
| // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
| // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR | ||
| // SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, | ||
| // WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE | ||
| // USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
|
||
| use std::marker::PhantomData; | ||
|
|
||
| use digest::Digest; | ||
| use tari_common::DomainDigest; | ||
|
|
||
| use crate::{common::hash_together, BalancedBinaryMerkleTree, Hash}; | ||
|
|
||
| #[derive(Debug)] | ||
| pub struct BalancedBinaryMerkleProof<D> { | ||
| pub path: Vec<Hash>, | ||
| pub node_index: usize, | ||
| _phantom: PhantomData<D>, | ||
| } | ||
|
|
||
| // Since this is balanced tree, the index `2k+1` is always left child and `2k` is right child | ||
|
|
||
| impl<D> BalancedBinaryMerkleProof<D> | ||
| where D: Digest + DomainDigest | ||
| { | ||
| pub fn verify(&self, root: &Hash, leaf_hash: Hash) -> bool { | ||
| let mut computed_root = leaf_hash; | ||
| let mut node_index = self.node_index; | ||
| for sibling in &self.path { | ||
| if node_index & 1 == 1 { | ||
| computed_root = hash_together::<D>(&computed_root, sibling); | ||
| } else { | ||
| computed_root = hash_together::<D>(sibling, &computed_root); | ||
| } | ||
| node_index = (node_index - 1) >> 1; | ||
| } | ||
| &computed_root == root | ||
| } | ||
|
|
||
| pub fn generate_proof(tree: &BalancedBinaryMerkleTree<D>, leaf_index: usize) -> Self { | ||
| let mut node_index = tree.get_node_index(leaf_index); | ||
| let mut proof = Vec::new(); | ||
| while node_index > 0 { | ||
| // Sibling | ||
| let parent = (node_index - 1) >> 1; | ||
| // The children are 2i+1 and 2i+2, so together are 4i+3, we substract one, we get the other. | ||
| let sibling = 4 * parent + 3 - node_index; | ||
| proof.push(tree.get_hash(sibling).clone()); | ||
| // Traverse to parent | ||
| node_index = parent; | ||
| } | ||
| Self { | ||
| path: proof, | ||
| node_index: tree.get_node_index(leaf_index), | ||
| _phantom: PhantomData, | ||
| } | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod test { | ||
| use tari_crypto::{hash::blake2::Blake256, hash_domain, hashing::DomainSeparatedHasher}; | ||
|
|
||
| use crate::{BalancedBinaryMerkleProof, BalancedBinaryMerkleTree}; | ||
| hash_domain!(TestDomain, "testing", 0); | ||
|
|
||
| #[test] | ||
| fn test_generate_and_verify_big_tree() { | ||
| for n in [1usize, 100, 1000, 10000] { | ||
| let leaves = (0..n) | ||
| .map(|i| [i.to_le_bytes().to_vec(), vec![0u8; 24]].concat()) | ||
| .collect::<Vec<_>>(); | ||
| let hash_0 = leaves[0].clone(); | ||
| let hash_n_half = leaves[n / 2].clone(); | ||
| let hash_last = leaves[n - 1].clone(); | ||
| let bmt = BalancedBinaryMerkleTree::<DomainSeparatedHasher<Blake256, TestDomain>>::create(leaves); | ||
| let root = bmt.get_merkle_root(); | ||
| let proof = BalancedBinaryMerkleProof::generate_proof(&bmt, 0); | ||
| assert!(proof.verify(&root, hash_0)); | ||
| let proof = BalancedBinaryMerkleProof::generate_proof(&bmt, n / 2); | ||
| assert!(proof.verify(&root, hash_n_half)); | ||
| let proof = BalancedBinaryMerkleProof::generate_proof(&bmt, n - 1); | ||
| assert!(proof.verify(&root, hash_last)); | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| // Copyright 2019. The Tari Project | ||
| // | ||
| // Redistribution and use in source and binary forms, with or without modification, are permitted provided that the | ||
| // following conditions are met: | ||
| // | ||
| // 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following | ||
| // disclaimer. | ||
| // | ||
| // 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the | ||
| // following disclaimer in the documentation and/or other materials provided with the distribution. | ||
| // | ||
| // 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote | ||
| // products derived from this software without specific prior written permission. | ||
| // | ||
| // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, | ||
| // INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE | ||
| // DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
| // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR | ||
| // SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, | ||
| // WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE | ||
| // USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
|
||
| use std::marker::PhantomData; | ||
|
|
||
| use digest::Digest; | ||
| use tari_common::DomainDigest; | ||
| use thiserror::Error; | ||
|
|
||
| use crate::{common::hash_together, ArrayLike, Hash}; | ||
|
|
||
| #[derive(Clone, Debug, PartialEq, Eq, Error)] | ||
| pub enum BalancedBinaryMerkleTreeError { | ||
| #[error("There is no leaf with the hash provided.")] | ||
| LeafNotFound, | ||
| } | ||
|
|
||
| // The hashes are perfectly balanced binary tree, so parent at index `i` (0-based) has children at positions `2*i+1` and | ||
| // `2*i+1`. | ||
| #[derive(Debug)] | ||
| pub struct BalancedBinaryMerkleTree<D> { | ||
| hashes: Vec<Hash>, | ||
| _phantom: PhantomData<D>, | ||
| } | ||
|
|
||
| impl<D> BalancedBinaryMerkleTree<D> | ||
| where D: Digest + DomainDigest | ||
| { | ||
| // There is no push method for this tree. This tree is created at once and no modifications are allowed. | ||
| pub fn create(leaves: Vec<Hash>) -> Self { | ||
| let leaves_cnt = leaves.len(); | ||
| if leaves_cnt == 0 { | ||
| return Self { | ||
| hashes: vec![], | ||
| _phantom: PhantomData, | ||
| }; | ||
| } | ||
| // The size of the tree of `n` leaves is `2*n+1` where the leaves are at the end of the array. | ||
| let mut hashes = Vec::with_capacity(2 * leaves_cnt - 1); | ||
| hashes.extend(vec![vec![0; 32]; leaves_cnt - 1]); | ||
| hashes.extend(leaves); | ||
| // Now we compute the hashes from bottom to up of the tree. | ||
| for i in (0..leaves_cnt - 1).rev() { | ||
| hashes[i] = hash_together::<D>(&hashes[2 * i + 1], &hashes[2 * i + 2]); | ||
| } | ||
| Self { | ||
| hashes, | ||
| _phantom: PhantomData, | ||
| } | ||
| } | ||
|
|
||
| pub fn get_merkle_root(&self) -> Hash { | ||
| if self.hashes.is_empty() { | ||
| D::digest(b"").to_vec() | ||
| } else { | ||
| self.hashes[0].clone() | ||
| } | ||
| } | ||
|
|
||
| pub(crate) fn get_hash(&self, pos: usize) -> &Hash { | ||
| &self.hashes[pos] | ||
| } | ||
|
|
||
| pub fn get_leaf(&self, leaf_index: usize) -> &Hash { | ||
| self.get_hash(self.get_node_index(leaf_index)) | ||
| } | ||
|
|
||
| pub(crate) fn get_node_index(&self, leaf_index: usize) -> usize { | ||
| leaf_index + (self.hashes.len() >> 1) | ||
| } | ||
|
|
||
| pub fn find_leaf_index_for_hash(&self, hash: &Hash) -> Result<usize, BalancedBinaryMerkleTreeError> { | ||
| let pos = self | ||
| .hashes | ||
| .position(hash) | ||
| .expect("Unexpected Balanced Binary Merkle Tree error") | ||
| .ok_or(BalancedBinaryMerkleTreeError::LeafNotFound)?; | ||
| if pos < (self.hashes.len() >> 1) { | ||
| // The hash provided was not for leaf, but for node. | ||
| Err(BalancedBinaryMerkleTreeError::LeafNotFound) | ||
| } else { | ||
| Ok(pos - (self.hashes.len() >> 1)) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod test { | ||
| use tari_crypto::{hash::blake2::Blake256, hash_domain, hashing::DomainSeparatedHasher}; | ||
|
|
||
| use crate::{balanced_binary_merkle_tree::BalancedBinaryMerkleTreeError, BalancedBinaryMerkleTree}; | ||
| hash_domain!(TestDomain, "testing", 0); | ||
|
|
||
| #[test] | ||
| fn test_empty_tree() { | ||
| let leaves = vec![]; | ||
| let bmt = BalancedBinaryMerkleTree::<DomainSeparatedHasher<Blake256, TestDomain>>::create(leaves); | ||
| let root = bmt.get_merkle_root(); | ||
| assert_eq!(root, vec![ | ||
| 72, 54, 179, 2, 214, 45, 9, 89, 161, 132, 177, 251, 229, 46, 124, 233, 32, 186, 46, 87, 127, 247, 19, 36, | ||
| 225, 191, 167, 189, 58, 58, 39, 74 | ||
| ]); | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_single_node_tree() { | ||
| let leaves = vec![vec![0; 32]]; | ||
| let bmt = BalancedBinaryMerkleTree::<DomainSeparatedHasher<Blake256, TestDomain>>::create(leaves); | ||
| let root = bmt.get_merkle_root(); | ||
| assert_eq!(root, vec![0; 32]); | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_find_leaf() { | ||
| let leaves = (0..100).map(|i| vec![i; 32]).collect::<Vec<_>>(); | ||
| let bmt = BalancedBinaryMerkleTree::<DomainSeparatedHasher<Blake256, TestDomain>>::create(leaves); | ||
| assert_eq!(bmt.find_leaf_index_for_hash(&vec![42; 32]).unwrap(), 42); | ||
| // Non existing hash | ||
| assert_eq!( | ||
| bmt.find_leaf_index_for_hash(&vec![142; 32]), | ||
| Err(BalancedBinaryMerkleTreeError::LeafNotFound) | ||
| ); | ||
| // This hash exists but it's not a leaf. | ||
| assert_eq!( | ||
| bmt.find_leaf_index_for_hash(&bmt.get_merkle_root()), | ||
| Err(BalancedBinaryMerkleTreeError::LeafNotFound) | ||
| ); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.