Groundwork for generalized warp sync #5454

rphmeier · 2017-04-13T18:28:00Z

Makes changes to the engines module that will allow warp sync, ancient block download, and light client sync to work for even complex engines.

Engines now detect and generate data for epoch changes. State proofs validating this data, along with the transitions themselves, are stored in the blockchain extras DB.

Snapshot block chunks are repurposed to a more general "secondary chunks" which engine-specific snapshot components will create and interpret as a means to corroborate the data within the snapshot state chunks. For proof-of-work engines, these chunks will be as specified in the wiki.

PoA engines like Aura and BasicAuthority will include all transitions in the snapshot, as described in #5427.
Ancient block download/light sync will make use of an EpochVerifier that caches an in-memory list of validators derived from the validated transitions obtained in the snapshot (or in the light client case, from the network). When epoch changes occur, we simply obtain a new EpochVerifier for that epoch. Ethash epochs don't currently make use of this mechanism, and just pretend that the chain is under one "epoch" so we never need to swap, but could be altered to in the future to ensure our DAG caching is completely optimal.

This approach isn't completely resilient (and indeed, won't really work for something like rinkeby's clique consensus; for that I'd wait on introducing engine-specific storage and have clique snapshots contain the entire headerchain.), but it will get the job done for now.

The "genesis epoch" has number 0, so any epochs beyond that must start with 1, otherwise warp and light sync will break.

keorn · 2017-04-19T16:40:16Z

ethcore/src/engines/authority_round.rs

+
+	#[test]
+	fn rejects_step_backwards() {
+				let tap = AccountProvider::transient_provider();


rphmeier · 2017-04-20T15:20:42Z

To give a quick run-down of what's still left to do:

Implementation of snapshot chunk and restore traits for an authority-based method.
Implementation of ancient block restoration which uses an EpochVerifier that changes whenever a header is a transition. Perhaps this will be done via another trait hanging off of SnapshotComponents
Ensure restoration database always contains genesis epoch data. Otherwise, contracts seeded with initial validators won't work when restoring from snapshot.
Tests!

This implementation will be in a separate branch.

rphmeier · 2017-04-20T16:46:09Z

@keorn

I'm mildly concerned about how Multi validator sets will play; there's a requirement that (for correct behavior) epochs should increase monotonically. My solution for now was to add the "inner" set's epoch number to the block number at which the transition occurred, although this will break for nested multi-sets and seems fragile.

keorn

Thanks, lg in general. I like the Unsure stuff to keep state out of Engine. For Mutli maybe handling the recursive case is possible as mentioned.

keorn · 2017-04-21T16:34:05Z

ethcore/src/engines/epoch_verifier.rs

+use error::Error;
+use header::Header;
+
+/// Verifier for all blocks within an epoch without accessing


keorn · 2017-04-24T09:41:12Z

ethcore/src/snapshot/consensus/mod.rs

+pub type ChunkSink<'a> = FnMut(&[u8]) -> io::Result<()> + 'a;
+
+// How many blocks to include in a snapshot, starting from the head of the chain.
+const SNAPSHOT_BLOCKS: u64 = 30000;


Can this be Engine specific? Different ones have different security requirements.

keorn · 2017-04-24T09:47:47Z

ethcore/src/client/client.rs

@@ -380,6 +390,12 @@ impl Client {
 			return Err(());
 		};

+		let verify_external_result = self.verifier.verify_block_external(header, &block.bytes, engine);
+		if let Err(e) = verify_external_result {
+			warn!(target: "client", "Stage 4 block verification failed for #{} ({})\nError: {:?}", header.number(), header.hash(), e);


Stage 4 is here.

(changed that to stage 5)

keorn · 2017-04-24T10:15:11Z

ethcore/src/engines/validator_set/multi.rs

+		self.correct_set_by_number(header.number()).1.epoch_proof(header, caller)
+	}
+
+	fn epoch_set(&self, header: &Header, proof: &[u8]) -> Result<(u64, super::SimpleList), ::error::Error> {


Maybe pass the previous block epoch nonce and return ValidatorSet instead of SimpleList? Then you can bump the nonce on recursive call.

main problem there is that it's hard to say what the previous block epoch nonce would be without recovering it; meaning you have to package two state proofs for each transition in warp/light rather than one.

right now, I think it's pretty fair to say that nested multi-sets don't make any sense -- we might just want to rewrite the validator set constructor to take a flag indicating whether we're at the top level (set false before recursing) and throw an error on nested multi set.

Makes sense, though this makes SimpleList a special fundamental ValidatorSet rather than just another trait implementation. But I guess that's fine since there are no other scenarios yet.

I'm fine with generalizing a bit later, but all current validator sets can express themselves as a SimpleList at any given block, and we need the property of fast verification of blocks out-of-order for the epoch verifier.

rphmeier · 2017-04-25T15:58:10Z

(tagging on behalf of @keorn)

rphmeier added 9 commits April 11, 2017 17:07

more useful Engine::verify_seal

a254b20

starting memoized validatorset

7723d62

engine changes

2d8a8bd

add validator contracts to native_contracts bin

2f5a774

use native contracts in ValidatorSet

0f80c57

supply optional call context to validator sets

ec922ee

skeleton for proof checking

34a1512

detect changes in safe_contract

b4f3e30

ChainVerifier for memoizing validator sets

715d5da

rphmeier added A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. M4-core ⛓ Core client code / Rust. labels Apr 13, 2017

rphmeier added 6 commits April 13, 2017 20:34

verify_external in verifier trait

e1fa4ab

epoch verifier rename

f6f9816

iterate DB by prefix

6da6c75

store epoch transition proofs in DB

a278dd5

ensure genesis validator set in DB

af868a7

iterate over all epochs

4d3f137

keorn reviewed Apr 19, 2017

View reviewed changes

ethcore/src/engines/authority_round.rs Outdated

#[test]

fn rejects_step_backwards() {

let tap = AccountProvider::transient_provider();

Copy link

keorn Apr 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent

rphmeier added 5 commits April 19, 2017 20:31

snapshot chunk and restore traits

2ec3397

Merge branch 'master' into aura-contract-warp

6652df0

fix indent

240c111

fix tests

a33b4cc

bloom check test

468a7a4

checkout correct tests submodule

023c45f

rphmeier changed the title ~~[WIP] Generalized warp sync~~ Groundwork for generalized warp sync Apr 20, 2017

rphmeier added A0-pleasereview 🤓 Pull request needs code review. and removed A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. labels Apr 20, 2017

rphmeier mentioned this pull request Apr 21, 2017

PoA warp implementation #5488

Merged

epoch generation proof fixes

7ab92f0

keorn suggested changes Apr 24, 2017

View reviewed changes

address grumbles

6a5702f

keorn approved these changes Apr 24, 2017

View reviewed changes

rphmeier added A8-looksgood 🦄 Pull request is reviewed well. and removed A0-pleasereview 🤓 Pull request needs code review. labels Apr 25, 2017

rphmeier merged commit 35958a0 into master Apr 25, 2017

rphmeier deleted the aura-contract-warp branch April 25, 2017 15:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Groundwork for generalized warp sync #5454

Groundwork for generalized warp sync #5454

rphmeier commented Apr 13, 2017 •

edited

Loading

keorn Apr 19, 2017

rphmeier commented Apr 20, 2017 •

edited

Loading

rphmeier commented Apr 20, 2017

keorn left a comment •

edited

Loading

keorn Apr 21, 2017

keorn Apr 24, 2017

keorn Apr 24, 2017

rphmeier Apr 25, 2017

keorn Apr 24, 2017

rphmeier Apr 24, 2017

rphmeier Apr 24, 2017

keorn Apr 24, 2017

rphmeier Apr 24, 2017

rphmeier commented Apr 25, 2017

Groundwork for generalized warp sync #5454

Groundwork for generalized warp sync #5454

Conversation

rphmeier commented Apr 13, 2017 • edited Loading

Choose a reason for hiding this comment

rphmeier commented Apr 20, 2017 • edited Loading

rphmeier commented Apr 20, 2017

keorn left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier commented Apr 25, 2017

rphmeier commented Apr 13, 2017 •

edited

Loading

rphmeier commented Apr 20, 2017 •

edited

Loading

keorn left a comment •

edited

Loading