[beta] Backports (#8624)

* Trace precompiled contracts when the transfer value is not zero (#8486)

* Trace precompiled contracts when the transfer value is not zero

* Add tests for precompiled CALL tracing

* Use byzantium test machine for the new test

* Add notes in comments on why we don't trace all precompileds

* Use is_transferred instead of transferred

* Return error if RLP size of transaction exceeds the limit (#8473)

* Return error if RLP size of transaction exceeds the limit

* Review comments fixed

* RLP check moved to verifier, corresponding pool test added

* Don't block sync when importing old blocks (#8530)

* Alter IO queueing.

* Don't require IoMessages to be Clone

* Ancient blocks imported via IoChannel.

* Get rid of private transactions io message.

* Get rid of deadlock and fix disconnected handler.

* Revert to old disconnect condition.

* Fix tests.

* Fix deadlock.

* Refactoring `ethcore-sync` - Fixing warp-sync barrier (#8543)

* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Address PR grumbles

* Retry failed CI job

* Fix tests

* PR Grumbles

* Handle socket address parsing errors (#8545)

Unpack errors and check for io::ErrorKind::InvalidInput and return our own AddressParse error. Remove the foreign link to std::net::AddrParseError and add an `impl From` for that error. Test parsing properly.

* Fix packet count when talking with PAR2 peers (#8555)

* Support diferent packet counts in different protocol versions.

* Fix light timeouts and eclipse protection.

* Fix devp2p tests.

* Fix whisper-cli compilation.

* Fix compilation.

* Fix ethcore-sync tests.

* Revert "Fix light timeouts and eclipse protection."

This reverts commit 06285ea.

* Increase timeouts.

* Add whisper CLI to the pipelines (#8578)

* Add whisper CLI to the pipelines

* Address todo, ref #8579

* Rename `whisper-cli binary` to `whisper` (#8579)

* rename whisper-cli binary to whisper

* fix tests

* Remove manually added text to the errors (#8595)

These messages were confusing for the users especially the help message.

* Fix account list double 0x display (#8596)

* Remove unused self import

* Fix account list double 0x display

* Fix BlockReward contract "arithmetic operation overflow" (#8611)

* Fix BlockReward contract "arithmetic operation overflow"

* Add docs on how execute_as_system works

* Fix typo

* Rlp decode returns Result (#8527)

rlp::decode returns Result

Make a best effort to handle decoding errors gracefully throughout the code, using `expect` where the value is guaranteed to be valid (and in other places where it makes sense).

* Remove expect (#8536)

* Remove expect and propagate rlp::DecoderErrors as TrieErrors

* Decoding headers can fail (#8570)

* rlp::decode returns Result

* Fix journaldb to handle rlp::decode Result

* Fix ethcore to work with rlp::decode returning Result

* Light client handles rlp::decode returning Result

* Fix tests in rlp_derive

* Fix tests

* Cleanup

* cleanup

* Allow panic rather than breaking out of iterator

* Let decoding failures when reading from disk blow up

* syntax

* Fix the trivial grumbles

* Fix failing tests

* Make Account::from_rlp return Result

* Syntx, sigh

* Temp-fix for decoding failures

* Header::decode returns Result

Handle new return type throughout the code base.

* Do not continue reading from the DB when a value could not be read

* Fix tests

* Handle header decoding in light_sync

* Handling header decoding errors

* Let the DecodeError bubble up unchanged

* Remove redundant error conversion

* fix compiler warning (#8590)

* Attempt to fix intermittent test failures (#8584)

Occasionally should_return_correct_nonces_when_dropped_because_of_limit fails, possibly because of multiple threads competing to finish. See CI logs here for an example: https://gitlab.parity.io/parity/parity/-/jobs/86738

* block_header can fail so return Result (#8581)

* block_header can fail so return Result

* Restore previous return type based on feedback

* Fix failing doc tests running on non-code

* Block::decode() returns Result (#8586)

* Gitlab test script fixes (#8573)

* Exclude /docs from modified files.

* Ensure all references in the working tree are available

* Remove duplicated line from test script

ethcore/light/src/client/header_chain.rs

@@ -228,15 +228,15 @@ impl HeaderChain {
 		let decoded_header = spec.genesis_header();
 
 		let chain = if let Some(current) = db.get(col, CURRENT_KEY)? {
-			let curr : BestAndLatest = ::rlp::decode(&current);
+			let curr : BestAndLatest = ::rlp::decode(&current).expect("decoding db value failed");
 
 			let mut cur_number = curr.latest_num;
 			let mut candidates = BTreeMap::new();
 
 			// load all era entries, referenced headers within them,
 			// and live epoch proofs.
 			while let Some(entry) = db.get(col, era_key(cur_number).as_bytes())? {
-				let entry: Entry = ::rlp::decode(&entry);
+				let entry: Entry = ::rlp::decode(&entry).expect("decoding db value failed");
 				trace!(target: "chain", "loaded header chain entry for era {} with {} candidates",
 					cur_number, entry.candidates.len());
 
@@ -305,7 +305,7 @@ impl HeaderChain {
 					batch.put(col, cht_key(cht_num as u64).as_bytes(), &::rlp::encode(cht_root));
 				}
 
-				let decoded_header = hardcoded_sync.header.decode();
+				let decoded_header = hardcoded_sync.header.decode()?;
 				let decoded_header_num = decoded_header.number();
 
 				// write the block in the DB.
@@ -524,7 +524,10 @@ impl HeaderChain {
 											None
 										}
 										Ok(None) => panic!("stored candidates always have corresponding headers; qed"),
-										Ok(Some(header)) => Some((epoch_transition, ::rlp::decode(&header))),
+										Ok(Some(header)) => Some((
+											epoch_transition,
+											::rlp::decode(&header).expect("decoding value from db failed")
+										)),
 									};
 								}
 							}
@@ -582,7 +585,7 @@ impl HeaderChain {
 						bail!(ErrorKind::Database(msg.into()));
 					};
 
-					let decoded = header.decode();
+					let decoded = header.decode().expect("decoding db value failed");
 
 					let entry: Entry = {
 						let bytes = self.db.get(self.col, era_key(h_num).as_bytes())?
@@ -591,7 +594,7 @@ impl HeaderChain {
 													in an inconsistent state", h_num);
 								ErrorKind::Database(msg.into())
 							})?;
-						::rlp::decode(&bytes)
+						::rlp::decode(&bytes).expect("decoding db value failed")
 					};
 
 					let total_difficulty = entry.candidates.iter()
@@ -604,9 +607,9 @@ impl HeaderChain {
 						.total_difficulty;
 
 					break Ok(Some(SpecHardcodedSync {
-						header: header,
-						total_difficulty: total_difficulty,
-						chts: chts,
+						header,
+						total_difficulty,
+						chts,
 					}));
 				},
 				None => {
@@ -742,7 +745,7 @@ impl HeaderChain {
 	/// so including it within a CHT would be redundant.
 	pub fn cht_root(&self, n: usize) -> Option<H256> {
 		match self.db.get(self.col, cht_key(n as u64).as_bytes()) {
-			Ok(val) => val.map(|x| ::rlp::decode(&x)),
+			Ok(db_fetch) => db_fetch.map(|bytes| ::rlp::decode(&bytes).expect("decoding value from db failed")),
 			Err(e) => {
 				warn!(target: "chain", "Error reading from database: {}", e);
 				None
@@ -793,7 +796,7 @@ impl HeaderChain {
 	pub fn pending_transition(&self, hash: H256) -> Option<PendingEpochTransition> {
 		let key = pending_transition_key(hash);
 		match self.db.get(self.col, &*key) {
-			Ok(val) => val.map(|x| ::rlp::decode(&x)),
+			Ok(db_fetch) => db_fetch.map(|bytes| ::rlp::decode(&bytes).expect("decoding value from db failed")),
 			Err(e) => {
 				warn!(target: "chain", "Error reading from database: {}", e);
 				None
@@ -812,7 +815,9 @@ impl HeaderChain {
 
 		for hdr in self.ancestry_iter(BlockId::Hash(parent_hash)) {
 			if let Some(transition) = live_proofs.get(&hdr.hash()).cloned() {
-				return Some((hdr.decode(), transition.proof))
+				return hdr.decode().map(|decoded_hdr| {
+					(decoded_hdr, transition.proof)
+				}).ok();
 			}
 		}
 
@@ -1192,7 +1197,7 @@ mod tests {
 
 		let cache = Arc::new(Mutex::new(Cache::new(Default::default(), Duration::from_secs(6 * 3600))));
 
-		let chain = HeaderChain::new(db.clone(), None, &spec, cache, HardcodedSync::Allow).unwrap();
+		let chain = HeaderChain::new(db.clone(), None, &spec, cache, HardcodedSync::Allow).expect("failed to instantiate a new HeaderChain");
 
 		let mut parent_hash = genesis_header.hash();
 		let mut rolling_timestamp = genesis_header.timestamp();
@@ -1211,17 +1216,17 @@ mod tests {
 			parent_hash = header.hash();
 
 			let mut tx = db.transaction();
-			let pending = chain.insert(&mut tx, header, None).unwrap();
+			let pending = chain.insert(&mut tx, header, None).expect("failed inserting a transaction");
 			db.write(tx).unwrap();
 			chain.apply_pending(pending);
 
 			rolling_timestamp += 10;
 		}
 
-		let hardcoded_sync = chain.read_hardcoded_sync().unwrap().unwrap();
+		let hardcoded_sync = chain.read_hardcoded_sync().expect("failed reading hardcoded sync").expect("failed unwrapping hardcoded sync");
 		assert_eq!(hardcoded_sync.chts.len(), 3);
 		assert_eq!(hardcoded_sync.total_difficulty, total_difficulty);
-		let decoded: Header = hardcoded_sync.header.decode();
+		let decoded: Header = hardcoded_sync.header.decode().expect("decoding failed");
 		assert_eq!(decoded.number(), h_num);
 	}
 }

ethcore/light/src/client/mod.rs

@@ -318,7 +318,7 @@ impl<T: ChainDataFetcher> Client<T> {
 
 			let epoch_proof =  self.engine.is_epoch_end(
 				&verified_header,
-				&|h| self.chain.block_header(BlockId::Hash(h)).map(|hdr| hdr.decode()),
+				&|h| self.chain.block_header(BlockId::Hash(h)).and_then(|hdr| hdr.decode().ok()),
 				&|h| self.chain.pending_transition(h),
 			);
 
@@ -426,7 +426,15 @@ impl<T: ChainDataFetcher> Client<T> {
 		};
 
 		// Verify Block Family
-		let verify_family_result = self.engine.verify_block_family(&verified_header, &parent_header.decode());
+
+		let verify_family_result = {
+			parent_header.decode()
+				.map_err(|dec_err| dec_err.into())
+				.and_then(|decoded| {
+					self.engine.verify_block_family(&verified_header, &decoded)
+				})
+
+		};
 		if let Err(e) = verify_family_result {
 			warn!(target: "client", "Stage 3 block verification failed for #{} ({})\nError: {:?}",
 				verified_header.number(), verified_header.hash(), e);

ethcore/light/src/net/mod.rs

@@ -75,14 +75,17 @@ const RECALCULATE_COSTS_INTERVAL: Duration = Duration::from_secs(60 * 60);
 // minimum interval between updates.
 const UPDATE_INTERVAL: Duration = Duration::from_millis(5000);
 
+/// Packet count for PIP.
+const PACKET_COUNT_V1: u8 = 9;
+
 /// Supported protocol versions.
-pub const PROTOCOL_VERSIONS: &'static [u8] = &[1];
+pub const PROTOCOL_VERSIONS: &'static [(u8, u8)] = &[
+	(1, PACKET_COUNT_V1),
+];
 
 /// Max protocol version.
 pub const MAX_PROTOCOL_VERSION: u8 = 1;
 
-/// Packet count for PIP.
-pub const PACKET_COUNT: u8 = 9;
 
 // packet ID definitions.
 mod packet {
@@ -111,9 +114,9 @@ mod packet {
 mod timeout {
 	use std::time::Duration;
 
-	pub const HANDSHAKE: Duration =  Duration::from_millis(2500);
-	pub const ACKNOWLEDGE_UPDATE: Duration = Duration::from_millis(5000);
-	pub const BASE: u64 = 1500; // base timeout for packet.
+	pub const HANDSHAKE: Duration = Duration::from_millis(4_000);
+	pub const ACKNOWLEDGE_UPDATE: Duration = Duration::from_millis(5_000);
+	pub const BASE: u64 = 2_500; // base timeout for packet.
 
 	// timeouts per request within packet.
 	pub const HEADERS: u64 = 250; // per header?
@@ -688,7 +691,7 @@ impl LightProtocol {
 			Err(e) => { punish(*peer, io, e); return }
 		};
 
-		if PROTOCOL_VERSIONS.iter().find(|x| **x == proto_version).is_none() {
+		if PROTOCOL_VERSIONS.iter().find(|x| x.0 == proto_version).is_none() {
 			punish(*peer, io, Error::UnsupportedProtocolVersion(proto_version));
 			return;
 		}

ethcore/light/src/net/request_credits.rs

@@ -407,7 +407,7 @@ mod tests {
 		let costs = CostTable::default();
 		let serialized = ::rlp::encode(&costs);
 
-		let new_costs: CostTable = ::rlp::decode(&*serialized);
+		let new_costs: CostTable = ::rlp::decode(&*serialized).unwrap();
 
 		assert_eq!(costs, new_costs);
 	}

ethcore/light/src/types/request/mod.rs

@@ -1642,7 +1642,7 @@ mod tests {
 	{
 		// check as single value.
 		let bytes = ::rlp::encode(&val);
-		let new_val: T = ::rlp::decode(&bytes);
+		let new_val: T = ::rlp::decode(&bytes).unwrap();
 		assert_eq!(val, new_val);
 
 		// check as list containing single value.