Fork choice and metadata framework for Engine #8401

sorpaas · 2018-04-13T14:52:58Z

This adds fork choice and metadata framework suitable for implementing Casper. Changes are discussed in sections below.

General comments on finalization, total difficulty and metadata

In this PR, we keep block reorg rules implemented in BlockChain, so as not to over-extend Engine. And we keep the total difficulty semantics unchanged, so as not to add complexity regarding wrap sync and light client. This requires BlockChain know about is_finalized and total_difficulty.

Metadata is implemented as a binary blob. This is because Engine currently is a trait object and we cannot specialize metadata types for each engine. The intention is that an engine will parse the metadata blob as RLP view.

`BlockDetails`

BlockDetails is extended with is_finalized and metadata, and is implemented in a backward compatible way. If is_finalized is false and metadata is empty, it is parsed in the old way, otherwise, in the new, extended way.

`LiveBlock`

I think it's suitable to consider metadata information as some extensions to a block's state. In most cases, we only allow setting metadata information when a block is live/open. This makes it so that we avoid the ambiguity about what will happen if block reorg happens. In OpenBlock, we allow setting of metadata and finalization (instant finality). For cases where info must be set on an ancestry block, we collect a list of AncestryAction and set it only when a new block is about to be committed. Currently AncestryAction only contains marking finality (post finality).

`parity_machine` traits

Several new traits based on Header and LiveBlock are added, allowing querying and setting metadata and finality information. Machine traits contain two new types:

ExtendedHeader, this is a normal header (so it's entirely possible that Machine::Header == Machine::ExtendedHeader), but extended with metadata information. In EthereumMachine, extended header has the total difficulty (total score) info and finality info.
AncestryAction (as explained above)

Engine lifecycle extension

on_new_block is extended with ancestry. This is an iterator of ExtendedHeader. The intention here is that it allows an engine to populate new block's metadata based on parent block. (But it's not needed by Casper).
ancestry_actions is called before the block is about to commit. This allows the block to set ancestry blocks finality info.
is_new_best is called with new and best as ExtendedHeader. The consideration here not to use an AncestryIterator is that an engine can easily get those info from on_new_block and ancestry_actions.

Casper

To implement Casper fork choice rules using this framework:

In on_close_block, set the score metadata and vote metadata information based on new.casper.
In ancestry_actions, check whether we are just in a new epoch, if so, check whether the last epoch as well as this epoch has enough votes. If so, emit an AncestryAction that sets the last epoch's block as finalized.
In is_new_best, set the fork choice rule to be based on score metadata information.

And a note on finalization: in the above I assume we use the "finalized" term as defined in the Casper FFG paper. That is, we consider a block to be finalized as long as a) there's a super majority link, and b) the link is about a direct child. The python implementation looks that to be using slightly different conditional branch, but I think there's a way to prove that they're equivalent (or doesn't matter).

rphmeier · 2018-04-13T15:06:51Z

ethcore/src/engines/mod.rs

@@ -338,6 +340,16 @@ pub trait Engine<M: Machine>: Sync + Send {
 	fn is_timestamp_valid(&self, header_timestamp: u64, parent_timestamp: u64) -> bool {
 		header_timestamp > parent_timestamp
 	}
+
+	/// Check whether a given block is the best block.
+	fn is_new_best(&self, bytes: &[u8], best_block_total_difficulty: U256, provider: &BlockProvider) -> bool {


can we abstract around total_difficulty and provider at the Machine level?

Will give it a try.

Here's an attempt: I added assoc type Machine::BlockMetadata and Machine::BlockProvider, and make Engine::is_new_best depend on them only. For EthEngine, BlockMetadata is currently just U256 which is the total difficulty (and we can extend it to contain Vec<(Bytes, Bytes)> in the future).

Client would first call Engine::generate_metadata to get the current set of metadata, and then call Engine::is_new_best before passing it to blockchain storage.

…a and Machine::BlockProvider

In this way they can use total difficulty fork choice check

rphmeier · 2018-04-14T12:30:57Z

ethcore/src/engines/mod.rs

+	let block = BlockView::new(bytes);
+	let header = block.header_view();
+	let parent_hash = header.parent_hash();
+	let parent_details = provider.block_details(&parent_hash).unwrap_or_else(|| panic!("Invalid parent hash: {:?}", parent_hash));


it would simplify things a lot just to provide the parent metadata. I am noticing BlockProvider and U256 leaking into a lot of engine implementations. Note that we want the engines module to be a separate crate at some point, and most of these engines don't need to be tied to Ethereum types specifically

I'm still trying to find out how to decouple Engine. The thing is, I don't think it's possible to completely make an engine (as currently it is) independent of all Ethereum types. For example, we now have logic in engine that needs to check a block's timestamp. If we want to generalize on this, we need to take account of some "blockchains" that don't have the concept of timestamp. I think parity_machine crate is trying to do similar things with ScoredHeader: Header, Transactions: LiveBlock, WithBalances: Machine, etc.

What I think might work better to decouple Engine is to split it into several different traits, and then use another struct to combine them together. For example, we might have InternalSealer, BlockVerifier, TimestampVerifier, ForkChooser, etc. In this way, we may be able to generalize some of the sub-traits to be independent of Ethereum types, while keeping those that need to depend on Ethereum types to be as it is.

we now have logic in engine that needs to check a block's timestamp

it probably should be abstracted at some point. we have verify_block/verify_header functions that can handle it AFAIK.

InternalSealer, BlockVerifier, TimestampVerifier, ForkChooser

The sub-traits approach would work if we weren't using Engine as a trait object everywhere. That's why there's the EthEngine specialization which is implementation for all Engine<EthereumMachine>.

I don't see anything in this PR that requires introduction of a dependency of Ethereum types.
The bigger issue is that it is not the state machine which should define this metadata type but the engine. Some engines will use total difficulty (all them them do now for historical reason, some when they shouldn't be), some will use other values.

Some engines will make distinctions between per-block metadata and "global" metadata. The most recently finalized block hash (in Aura or CasperFFG) shouldn't change depending on whether you look at block A or block B, but their total difficulty should. That's why I advocated for using a generic key-value store for metadata. We can prefix some keys by header-hash, and some would be global keys.

Then the only additional bound you need on total-difficulty using engines is

Machine::Header: ScoredHeader, <Machine::Header as ScoredHeader>::Score: Serialize + Deserialize + Add + Ord,

Engine::extra_info should also take a handle to that key-value metadata store.

rphmeier · 2018-04-16T12:48:25Z

machine/src/lib.rs

@@ -81,8 +81,12 @@ pub trait Machine: for<'a> LocalizedMachine<'a> {
 	type LiveBlock: LiveBlock<Header=Self::Header>;
 	/// A handle to a blockchain client for this machine.
 	type EngineClient: ?Sized;
+	/// A handle to the block provider for this machine.
+	type BlockProvider: ?Sized;


doesn't EngineClient do this already?

rphmeier · 2018-04-16T13:04:51Z

ethcore/src/engines/mod.rs

@@ -338,6 +340,27 @@ pub trait Engine<M: Machine>: Sync + Send {
 	fn is_timestamp_valid(&self, header_timestamp: u64, parent_timestamp: u64) -> bool {
 		header_timestamp > parent_timestamp
 	}
+
+	/// Generate metadata for a new block.
+	fn generate_metadata(&self, bytes: &[u8], provider: &M::BlockProvider) -> M::BlockMetadata;


I'm wary of providing the entire blockchain context here. We do it in a specialized way through Ancestry in some localized points. In most cases we won't need anything more than the parent_metadata.

I would recommend changing the signature to generate_metadata(header, ancestry). Then we can invoke the rolling finality tracker in authority_round from there and embed the most recently-finalized block hash in there and use it in the fork choice. That will pose a problem with strongly-typed metadata according to the machine because if we've defined the "machine" metadata to be only a TotalDifficulty then that's not enough for that modification to authority_round's fork choice.

The other option is to avoid strongly-typed metadata entirely. This is kind of necessary until we stop using trait-object engines.

…rk-choice-no-metadata

rphmeier

revisit reorganization through finalized blocks

If there's finalized blocks in from route, choose the old branch without calling `Engine::fork_choice`.

…rk-choice-no-metadata

sorpaas · 2018-05-09T12:41:56Z

Just some notes regarding now this ended up making the finalization rule works, in case the above discussion threads are too long:

The strict version of finalization is used -- No reorg can move pass any finalized blocks.
The Client will first check whether the finalization rule is held -- if there's finalized blocks preventing the reorg, Engine::fork_choice will not be called and the new block will directly become a branch. If no finalized block prevents the reorg, then it will call Engine::fork_choice to decide which branch to choose.

rphmeier · 2018-05-11T14:41:40Z

ethcore/src/blockchain/blockchain.rs

-			index: index
+			index: index,
+			is_from_route_finalized: is_from_route_finalized,
+			is_to_route_finalized: is_to_route_finalized,


still shouldn't have both of these

is_to_route_finalized doesn't make any sense:

we don't count the common ancestor when producing this value

thus there is some ancestor of the prospective block which is finalized

the current best block is either an ancestor of that block or on a fork starting from an ancestor of it.

best block implies that it is a maximization of the fork choice. when we finalize a block, we refuse to accept any blocks which would produce a maximization of the fork choice which does not include that finalized block.

but this current best block does not include the finalized block in its ancestry. contradiction

(the only place where this does make sense is if prospective.finalized = true). But this information is already available to the caller.

Yeah I think I got what you meant. If we use the finality definition settled down earlier, then is_to_route_finalized should always be false. I'm going to remove this variable.

rphmeier · 2018-05-11T14:41:53Z

ethcore/src/blockchain/blockchain.rs

@@ -769,11 +816,14 @@ impl BlockChain {
 				location: BlockLocation::CanonChain,
 			};

+			// TODO [sorpaas] support wrap sync insertion of finalization and metadata.


rphmeier · 2018-05-11T14:42:44Z

ethcore/src/blockchain/blockchain.rs

@@ -898,7 +948,22 @@ impl BlockChain {
 	/// Inserts the block into backing cache database.
 	/// Expects the block to be valid and already verified.
 	/// If the block is already known, does nothing.
-	pub fn insert_block(&self, batch: &mut DBTransaction, bytes: &[u8], receipts: Vec<Receipt>) -> ImportRoute {
+	pub fn insert_block(&self, batch: &mut DBTransaction, bytes: &[u8], receipts: Vec<Receipt>, extras:ExtrasInsert) -> ImportRoute {


space between extras and type

rphmeier · 2018-05-11T14:44:49Z

ethcore/src/blockchain/blockchain.rs

 		}
 	}

+	/// Mark a block to be considered finalized. Panic if the block hash is not found.


looks like it just returns None if it's not found. Option<()> for a function which can error is weird though

rphmeier · 2018-05-11T14:48:13Z

ethcore/src/client/client.rs

+			}
+		};
+
+		let route = chain.tree_route(best_hash, *parent).expect("blocks being imported always within recent history; qed");


this isn't an accurate reason for it not to panic. should refer to the fact that we don't keep forks without keeping a common ancestor

i think I saw another variant of this expectation somewhere else, should be fixed too if so

rphmeier · 2018-05-11T14:59:58Z

ethcore/src/client/client.rs

+
+		let route = chain.tree_route(best_hash, *parent).expect("blocks being imported always within recent history; qed");
+		let fork_choice = if route.is_from_route_finalized {
+			ForkChoice::Old


questionable whether to even bother executing the block in this case. but we can answer that later on

When no block reorg passes a finalized block, this variable is always false.

`blockchain` mod does not yet have an Error type, so we still temporarily use None here.

…rk-choice-no-metadata

andresilva · 2018-05-15T18:48:02Z

Just finished reviewing the latest changes, LGTM. I'll extend AuRa with ancestry_actions using RollingFinality to finalize ancestry.

* Add light client TODO item * Move existing total-difficulty-based fork choice check to Engine * Abstract total difficulty and block provider as Machine::BlockMetadata and Machine::BlockProvider * Decouple "generate_metadata" logic to Engine * Use fixed BlockMetadata and BlockProvider type for null and instantseal In this way they can use total difficulty fork choice check * Extend blockdetails with metadatas and finalized info * Extra data update: mark_finalized and update_metadatas * Check finalized block in Blockchain * Fix a test constructor in verification mod * Add total difficulty trait * Fix type import * Db migration to V13 with metadata column * Address grumbles * metadatas -> metadata * Use generic type for update_metadata to avoid passing HashMap all around * Remove metadata in blockdetails * [WIP] Implement a generic metadata architecture * [WIP] Metadata insertion logic in BlockChain * typo: Value -> Self::Value * [WIP] Temporarily remove Engine::is_new_best interface So that we don't have too many type errors. * [WIP] Fix more type errors * [WIP] ExtendedHeader::PartialEq * [WIP] Change metadata type Option<Vec<u8>> to Vec<u8> * [WIP] Remove Metadata Error * [WIP] Clean up error conversion * [WIP] finalized -> is_finalized * [WIP] Mark all fields in ExtrasInsert as pub * [WIP] Remove unused import * [WIP] Keep only local metadata info * Mark metadata as optional * [WIP] Revert metadata db change in BlockChain * [WIP] Put finalization in unclosed state * Use metadata interface in BlockDetail * [WIP] Fix current build failures * [WIP] Remove unused blockmetadata struct * Remove DB migration info * [WIP] Typo * Use ExtendedHeader to implement fork choice check * Implement is_new_best using Ancestry iterator * Use expect instead of panic * [WIP] Add ancestry Engine support via on_new_block * Fix tests * Emission of ancestry actions * use_short_version should take account of metadata * Engine::is_new_best -> Engine::fork_choice * Use proper expect format as defined in openethereum#1026 * panic -> expect * ancestry_header -> ancestry_with_metadata * Boxed iterator -> &mut iterator * Fix tests * is_new_best -> primitive_fork_choice * Document how fork_choice works * Engine::fork_choice -> Engine::primitive_fork_choice * comment: clarify types of finalization where Engine::primitive_fork_choice works * Expose FinalizationInfo to Engine * Fix tests due to merging * Remove TotalDifficulty trait * Do not pass FinalizationInfo to Engine If there's finalized blocks in from route, choose the old branch without calling `Engine::fork_choice`. * Fix compile * Fix unused import * Remove is_to_route_finalized When no block reorg passes a finalized block, this variable is always false. * Address format grumbles * Fix docs: mark_finalized returns None if block hash is not found `blockchain` mod does not yet have an Error type, so we still temporarily use None here. * Fix inaccurate tree_route None expect description

* master: (95 commits) remove empty file (openethereum#8705) Update mod.rs (openethereum#8695) Use impl Future in the light client RPC helpers (openethereum#8628) Fix cli signer (openethereum#8682) Allow making direct RPC queries from the C API (openethereum#8588) Remove the error when stopping the network (openethereum#8671) Move connection_filter to the network crate (openethereum#8674) Remove HostInfo::client_version() and secret() (openethereum#8677) Refactor EIP150, EIP160 and EIP161 forks to be specified in CommonParams (openethereum#8614) parity: improve cli help and logging (openethereum#8665) updated tiny-keccak to 1.4.2 (openethereum#8669) Remove the Keccak C library and use the pure Rust impl (openethereum#8657) Remove HostInfo::next_nonce (openethereum#8644) Fix not downloading old blocks (openethereum#8642) Resumable warp-sync / Seed downloaded snapshots (openethereum#8544) Don't open Browser post-install on Mac (openethereum#8641) Changelog for 1.10.4-stable and 1.11.1-beta (openethereum#8637) typo (openethereum#8640) Fork choice and metadata framework for Engine (openethereum#8401) Check that the Android build doesn't dep on c++_shared (openethereum#8538) ...

@andresilva

* Tx permission contract improvement * Use tuple for to address * Introduced ABI for deprecated tx permission contract * Improved ABI for tx permission contract with contract name, name hash and version * Introduced support for deprecated tx permission contract + fixed cache for the new version + introduced `target` for tx filter loging * Introduced test for the new tx permission contract version + old test renamed as deprecated * Removed empty lines * Introduced filter_only_sender return value in allowedTxTypes fn + improved caching * Introduced version checking for tx permission contract * Moved tx permission contract test genesis specs to separate files * handle queue import errors a bit more gracefully (#8385) * Some tweaks to main.rs for parity as a library (#8370) * Some tweaks to main.rs for parity as a library * Remove pub from PostExecutionAction * New Transaction Queue implementation (#8074) * Implementation of Verifier, Scoring and Ready. * Queue in progress. * TransactionPool. * Prepare for txpool release. * Miner refactor [WiP] * WiP reworking miner. * Make it compile. * Add some docs. * Split blockchain access to a separate file. * Work on miner API. * Fix ethcore tests. * Refactor miner interface for sealing/work packages. * Implement next nonce. * RPC compiles. * Implement couple of missing methdods for RPC. * Add transaction queue listeners. * Compiles! * Clean-up and parallelize. * Get rid of RefCell in header. * Revert "Get rid of RefCell in header." This reverts commit 0f2424c. * Override Sync requirement. * Fix status display. * Unify logging. * Extract some cheap checks. * Measurements and optimizations. * Fix scoring bug, heap size of bug and add cache * Disable tx queueing and parallel verification. * Make ethcore and ethcore-miner compile again. * Make RPC compile again. * Bunch of txpool tests. * Migrate transaction queue tests. * Nonce Cap * Nonce cap cache and tests. * Remove stale future transactions from the queue. * Optimize scoring and write some tests. * Simple penalization. * Clean up and support for different scoring algorithms. * Add CLI parameters for the new queue. * Remove banning queue. * Disable debug build. * Change per_sender limit to be 1% instead of 5% * Avoid cloning when propagating transactions. * Remove old todo. * Post-review fixes. * Fix miner options default. * Implement back ready transactions for light client. * Get rid of from_pending_block * Pass rejection reason. * Add more details to drop. * Rollback heap size of. * Avoid cloning hashes when propagating and include more details on rejection. * Fix tests. * Introduce nonces cache. * Remove uneccessary hashes allocation. * Lower the mem limit. * Re-enable parallel verification. * Add miner log. Don't check the type if not below min_gas_price. * Add more traces, fix disabling miner. * Fix creating pending blocks twice on AuRa authorities. * Fix tests. * re-use pending blocks in AuRa * Use reseal_min_period to prevent too frequent update_sealing. * Fix log to contain hash not sender. * Optimize local transactions. * Fix aura tests. * Update locks comments. * Get rid of unsafe Sync impl. * Review fixes. * Remove excessive matches. * Fix compilation errors. * Use new pool in private transactions. * Fix private-tx test. * Fix secret store tests. * Actually use gas_floor_target * Fix config tests. * Fix pool tests. * Address grumbles. * clarify that windows need perl and yasm (#8402) * Unify and limit rocksdb dependency places (#8371) * secret_store: remove kvdb_rocksdb dependency * cli: init db mod for open dispatch * cli: move db, client_db, restoration_db, secretstore_db to a separate mod * migration: rename to migration-rocksdb and remove ethcore-migrations * ethcore: re-move kvdb-rocksdb dep to test * mark test_helpers as test only and fix migration mod naming * Move restoration_db_handler to test_helpers_internal * Fix missing preambles in test_helpers_internal and rocksdb/helpers * Move test crates downward * Fix missing docs * cli, db::open_db: move each argument to a separate line * Use featuregate instead of dead code for `open_secretstore_db` * Move pathbuf import to open_secretstore_db Because it's only used there behind a feature gate * Use tokio::spawn in secret_store listener and fix Uri (#8373) * Directly wait for future to resolve in a threadpool * Ignore return value * Use path.starts_with instead of req_uri.is_absolute The later now means something else in hyper 0.11.. * Use tokio::spawn * typo: remove accidential unsafe impl * remove Tendermint extra_info due to seal inconsistencies (#8367) * More code refactoring to integrate Duration (#8322) * More code refactoring to integrate Duration * Fix typo * Fix tests * More test fix * tokio-core v0.1.16 -> v0.1.17 (#8408) * Replace legacy Rlp with UntrustedRlp and use in ethcore rlp views (#8316) * WIP * Replace Rlp with UntrustedRlp in views, explicity unwrap with expect First pass to get it to compile. Need to figure out whether to do this or to propogate Errors upstream, which would require many more changes to dependent code. If we do this way we are assuming that the views are always used in a context where the rlp is trusted to be valid e.g. when reading from our own DB. So need to fid out whether views are used with data received from an untrusted (e.g. extrernal peer). * Remove original Rlp impl, rename UntrustedRlp -> Rlp * Create rlp views with view! macro to record debug info Views are assumed to be over valid rlp, so if there is a decoding error we record where the view was created in the first place and report it in the expect * Use $crate in view! macro to avoid import, fix tests * Expect valid rlp in decode functions for now * Replace spaces with tabs in new file * Add doc tests for creating views with macro * Update rlp docs to reflect removing of UntrustedRlp * Replace UntrustedRlp usages in private-tx merge * Fix TODO comments (#8413) * update zip to 0.3 (#8381) * update zip to 0.3 * enable zip deflate feature * typo, docs parity_chainId: empty string -> None (#8434) * Fix receipts stripping. (#8414) * Changelogs for 1.9.6 and 1.10.1 (#8411) * Add changelog for 1.9.6 * Add Changelog for 1.10.1 * Move ethcore::Error to error_chain (#8386) * WIP * Convert Ethcore error to use error_chain * Use error_chain for ImportError and BlockImportError * Fix error pattern matches for error_chain in miner * Implement explicit From for AccountsError * Fix pattern matches for ErrorKinds * Handle ethcore error_chain in light client * Explicitly define Result type to avoid shadowing * Fix remaining Error pattern matches * Fix tab space formatting * Helps if the tests compile * Fix error chain matching after merge * remove From::from. (#8390) * Some tiny modifications. 1. fix some typo in the comment. 2. sort the order of methods in 'impl state::Backend for StateDB` * Remove the clone of code_cache, as it has been done in clone_basic. * remove From::from. It seems not necessary. * Use forked app_dirs crate for reverted Windows dir behavior (#8438) * Remove unused appdirs dependency in CLI * Use forked app_dirs crate for reverted Windows dir behavior * Permission fix (#8441) * Block reward contract (#8419) * engine: add block reward contract abi and helper client * aura: add support for block reward contract * engine: test block reward contract client * aura: test block reward contract * engine + aura: add missing docs * engine: share SystemCall type alias * aura: add transition for block reward contract * engine: fix example block reward contract source link and bytecode * Improve VM executor stack size estimation rules (#8439) * Improve VM executor stack size estimation rules * typo: docs add "(Debug build)" comment * Fix an off by one typo and set minimal stack size This avoids the case if `depth_threshold == max_depth`. Usually setting stack size to zero will just rebound it to platform minimal stack size, but we set it here just in case. * Use saturating_sub to avoid potential overflow * Private transactions processing error handling (#8431) * Integration test for private transaction returned * Do not interrupt verification in case of errors * Helpers use specified * Review comments fixed * Update Cargo hidapi-rs dependency (#8447) * Allow 32 bit pipelines to fail (#8454) * Disable 32bit tragets for gitlab * Rename linux pipelines * Update wasmi (#8452) * Return error in case eth_call returns VM errors (#8448) * Add VMError generator * Return executed exceptions in eth_call * ParityShell::open `Return result` (#8377) * start * add error handling for winapi * fix typo * fix warnings and windows errors * formatting * Address review comments * fix docker build (#8462) * Add changelog for 1.9.7 and 1.10.2 (#8460) * Add changelog for 1.9.7 * Add Changelog for 1.10.2 * Apply proper markdown * Run a spellchecker :) * Be pedantic about the 32-bit pipelines :) * fix typos in vm description comment (#8446) * Use rename_all for RichBlock and RichHeader serialization (#8471) * typo: fix a resolved TODO comment * Use rename_all instead of individual renames * Don't require write lock when fetching status. (#8481) * Bump master to 1.12 (#8477) * Bump master to 1.12 * Bump crates to 1.12 * Bump mac installer version to 1.12 * Update Gitlab scripts * Fix snap builds (#8483) * Update hardcodedSync for Ethereum, Kovan, and Ropsten (#8489) * Update wasmi and pwasm-utils (#8493) * Update wasmi to 0.2 New wasmi supports 32bit platforms and no longer requires a special feature to build for such platforms. * Update pwasm-utils to 0.1.5 * Remove three old warp boot nodes. (#8497) * Return error if RLP size of transaction exceeds the limit (#8473) * Return error if RLP size of transaction exceeds the limit * Review comments fixed * RLP check moved to verifier, corresponding pool test added * `duration_ns: u64 -> duration: Duration` (#8457) * duration_ns: u64 -> duration: Duration * format on millis {:.2} -> {} * Remove unused dependency `bigint` (#8505) * remove unused dependency bigint in * remove bigint in rpc_cli * Show imported messages for light client (#8517) * Directly return None if tracing is disabled (#8504) * Directly return None if tracing is disabled * Address gumbles: release read locks as fast as possible * Hardware Wallet trait (#8071) * getting started with replacing HardwareWalletManager * trezor and ledger impls the new trait with some drawbacks * Everything move to the new trait * It required lifetime annotations in the trait because [u8] in unsized * Lets now start moving entry point from HardwareWalletManager * rename trait to Wallet * move thread management to the actual wallets * Moved thread management to each respective Wallet * Cleaned up pub items that is needed to be pub * Wallet trait more or less finished * Cleaned up docs * fix tests * omit removed docs * fix spelling, naming och remove old comments * ledger test is broken, add correct logging format * So locally on my machine Linux Ubuntu 17.10 the test doesn't panic but on the CI server libusb::Context::new() fails which I don't understand because it has worked before * Additionally the ledger test is optional so I lean toward ignoring it the CI Server * ignore hardware tests by default * more verbose checking in ledger test * SecretStore: merge two types of errors into single one + Error::is_non_fatal (#8357) * SecretStore: error unify initial commit SecretStore: pass real error in error messages SecretStore: is_internal_error -> Error::is_non_fatal warnings SecretStore: ConsensusTemporaryUnreachable fix after merge removed comments removed comments SecretStore: updated HTTP error responses SecretStore: more ConsensusTemporaryUnreachable tests fix after rebase * fixed grumbles * use HashSet in tests * Enable WebAssembly and Byzantium for Ellaism (#8520) * Enable WebAssembly and Byzantium for Ellaism * Fix indentation * Remove empty lines * More changes for Android (#8421) * Transaction Pool improvements (#8470) * Don't use ethereum_types in transaction pool. * Hide internal insertion_id. * Fix tests. * Review grumbles. * Fetching logs by hash in blockchain database (#8463) * Fetch logs by hash in blockchain database * Fix tests * Add unit test for branch block logs fetching * Add docs that blocks must already be sorted * Handle branch block cases properly * typo: empty -> is_empty * Remove return_empty_if_none by using a closure * Use BTreeSet to avoid sorting again * Move is_canon to BlockChain * typo: pass value by reference * Use loop and wrap inside blocks to simplify the code Borrowed from #8463 (comment) * typo: missed a comment * Pass on storage keys tracing to handle the case when it is not modified (#8491) * Pass on storage keys even if it is not modified * typo: account and storage query `to_pod_diff` builds both `touched_addresses` merge and storage keys merge. * Fix tests * Use state query directly because of suicided accounts * Fix a RefCell borrow issue * Add tests for unmodified storage trace * Address grumbles * typo: remove unwanted empty line * ensure_cached compiles with the original signature * Don't panic in import_block if invalid rlp (#8522) * Don't panic in import_block if invalid rlp * Remove redundant type annotation * Replace RLP header view usage with safe decoding Using the view will panic with invalid RLP. Here we use Rlp decoding directly which will return a `Result<_, DecoderError>`. While this path currently should not have any invalid RLP - it makes it safer if ever called with invalid RLP from other code paths. * Remove expect (#8536) * Remove expect and propagate rlp::DecoderErrors as TrieErrors * EIP 145: Bitwise shifting instructions in EVM (#8451) * Add SHL, SHR, SAR opcodes * Add have_bitwise_shifting schedule flag * Add all EIP tests for SHL * Add SHR implementation and tests * Implement SAR and add tests * Add eip145transition config param * Change map_or to map_or_else when possible * Consolidate crypto functionality in `ethcore-crypto`. (#8432) * Consolidate crypto functionality in `ethcore-crypto`. - Move `ecdh`/`ecies` modules to `ethkey`. - Refactor `ethcore-crypto` to use file per module. - Replace `subtle` with `ethcore_crypto::is_equal`. - Add `aes_gcm` module to `ethcore-crypto`. * Rename `aes::{encrypt,decrypt,decrypt_cbc}` ... ... to `aes::{encrypt_128_ctr,decrypt_128_ctr,decrypt_128_cbc}`. * ethcore, rpc, machine: refactor block reward application and tracing (#8490) * Keep all enacted blocks notify in order (#8524) * Keep all enacted blocks notify in order * Collect is unnecessary * Update ChainNotify to use ChainRouteType * Fix all ethcore fn defs * Wrap the type within ChainRoute * Fix private-tx and sync api * Fix secret_store API * Fix updater API * Fix rpc api * Fix informant api * Eagerly cache enacted/retracted and remove contain_enacted/retracted * Fix indent * tests: should use full expr form for struct constructor * Use into_enacted_retracted to further avoid copy * typo: not a function * rpc/tests: ChainRoute -> ChainRoute::new * Node table sorting according to last contact data (#8541) * network-devp2p: sort nodes in node table using last contact data * network-devp2p: rename node contact types in node table json output * network-devp2p: fix node table tests * network-devp2p: note node failure when failed to establish connection * network-devp2p: handle UselessPeer error * network-devp2p: note failure when marking node as useless * Rlp decode returns Result (#8527) rlp::decode returns Result Make a best effort to handle decoding errors gracefully throughout the code, using `expect` where the value is guaranteed to be valid (and in other places where it makes sense). * Parity as a library (#8412) * Parity as a library * Fix concerns * Allow using a null on_client_restart_cb * Fix more concerns * Test the C library in test.sh * Reduce CMake version to 3.5 * Move the clib test before cargo test * Add println in test * Trace precompiled contracts when the transfer value is not zero (#8486) * Trace precompiled contracts when the transfer value is not zero * Add tests for precompiled CALL tracing * Use byzantium test machine for the new test * Add notes in comments on why we don't trace all precompileds * Use is_transferred instead of transferred * Don't block sync when importing old blocks (#8530) * Alter IO queueing. * Don't require IoMessages to be Clone * Ancient blocks imported via IoChannel. * Get rid of private transactions io message. * Get rid of deadlock and fix disconnected handler. * Revert to old disconnect condition. * Fix tests. * Fix deadlock. * Make trace-time publishable. (#8568) * Remove State::replace_backend (#8569) * Refactoring `ethcore-sync` - Fixing warp-sync barrier (#8543) * Start dividing sync chain : first supplier method * WIP - updated chain sync supplier * Finish refactoring the Chain Sync Supplier * Create Chain Sync Requester * Add Propagator for Chain Sync * Add the Chain Sync Handler * Move tests from mod -> handler * Move tests to propagator * Refactor SyncRequester arguments * Refactoring peer fork header handler * Fix wrong highest block number in snapshot sync * Small refactor... * Address PR grumbles * Retry failed CI job * Fix tests * PR Grumbles * Decoding headers can fail (#8570) * rlp::decode returns Result * Fix journaldb to handle rlp::decode Result * Fix ethcore to work with rlp::decode returning Result * Light client handles rlp::decode returning Result * Fix tests in rlp_derive * Fix tests * Cleanup * cleanup * Allow panic rather than breaking out of iterator * Let decoding failures when reading from disk blow up * syntax * Fix the trivial grumbles * Fix failing tests * Make Account::from_rlp return Result * Syntx, sigh * Temp-fix for decoding failures * Header::decode returns Result Handle new return type throughout the code base. * Do not continue reading from the DB when a value could not be read * Fix tests * Handle header decoding in light_sync * Handling header decoding errors * Let the DecodeError bubble up unchanged * Remove redundant error conversion * Update CHANGELOG for 1.9, 1.10, and 1.11 (#8556) * Move changelog for 1.10.x * Mark 1.9 EOL * Prepare changelog for 1.10.3 stable * Prepare changelog for 1.11.0 stable * Update changelogs * Update CHANGELOG for 1.10.3 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Format changelog * Handle socket address parsing errors (#8545) Unpack errors and check for io::ErrorKind::InvalidInput and return our own AddressParse error. Remove the foreign link to std::net::AddrParseError and add an `impl From` for that error. Test parsing properly. * Remove unnecessary cloning in overwrite_with (#8580) * Remove unnecessary cloning in overwrite_with * Remove into_iter * changelog nit (#8585) * Rename `whisper-cli binary` to `whisper` (#8579) * rename whisper-cli binary to whisper * fix tests * Add whisper CLI to the pipelines (#8578) * Add whisper CLI to the pipelines * Address todo, ref #8579 * Added Dockerfile for alpine linux by @andresilva, closes #3565 (#8587) * Changelog and Readme (#8591) * Move changelog for 1.10.x * Mark 1.9 EOL * Prepare changelog for 1.10.3 stable * Prepare changelog for 1.11.0 stable * Update changelogs * Update CHANGELOG for 1.10.3 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Format changelog * Update README for 1.11 * Fix typo * Attempt to fix intermittent test failures (#8584) Occasionally should_return_correct_nonces_when_dropped_because_of_limit fails, possibly because of multiple threads competing to finish. See CI logs here for an example: https://gitlab.parity.io/parity/parity/-/jobs/86738 * Make mio optional in ethcore-io (#8537) * Make mio optional in ethcore-io * Add some annotations, plus a check for features * Increase timer for test * Fix Parity UI link (#8600) Fix link #8599 * fix compiler warning (#8590) * Block::decode() returns Result (#8586) * block_header can fail so return Result (#8581) * block_header can fail so return Result * Restore previous return type based on feedback * Fix failing doc tests running on non-code * Remove inject.js server-side injection for dapps (#8539) * Remove inject.js server-side injection for dapps * Remove dapps test `should_inject_js` Parity doesn't inject a <script> tag inside the responses anymore * Fix the mio test again (#8602) * 2 tiny modification on snapshot (#8601) * Some tiny modifications. 1. fix some typo in the comment. 2. sort the order of methods in 'impl state::Backend for StateDB` * Remove the clone of code_cache, as it has been done in clone_basic. * remove From::from. It seems not necessary. * change mode: remove rust files' executable mode. * 2 tiny modifications on snapshot. * Use full qualified syntax for itertools::Itertools::flatten (#8606) * Fix packet count when talking with PAR2 peers (#8555) * Support diferent packet counts in different protocol versions. * Fix light timeouts and eclipse protection. * Fix devp2p tests. * Fix whisper-cli compilation. * Fix compilation. * Fix ethcore-sync tests. * Revert "Fix light timeouts and eclipse protection." This reverts commit 06285ea. * Increase timeouts. * typo: wrong indentation in kovan config (#8610) * Fix account list double 0x display (#8596) * Remove unused self import * Fix account list double 0x display * Remove manually added text to the errors (#8595) These messages were confusing for the users especially the help message. * Gitlab test script fixes (#8573) * Exclude /docs from modified files. * Ensure all references in the working tree are available * Remove duplicated line from test script * Fix BlockReward contract "arithmetic operation overflow" (#8611) * Fix BlockReward contract "arithmetic operation overflow" * Add docs on how execute_as_system works * Fix typo * ´main.rs´ typo (#8629) * Typo * Update main.rs * Store morden db and keys in "path/to/parity/data/Morden" (ropsten uses "test", like before) (#8621) * Store morden db and keys in "path/to/parity/data/morden" (ropsten uses "test", like before) * Fix light sync with initial validator-set contract (#8528) * Fix #8468 * Use U256::max_value() instead * Fix again * Also change initial transaction gas * Remove NetworkContext::io_channel() (#8625) * Remove io_channel() * Fix warning * Check that the Android build doesn't dep on c++_shared (#8538) * Fork choice and metadata framework for Engine (#8401) * Add light client TODO item * Move existing total-difficulty-based fork choice check to Engine * Abstract total difficulty and block provider as Machine::BlockMetadata and Machine::BlockProvider * Decouple "generate_metadata" logic to Engine * Use fixed BlockMetadata and BlockProvider type for null and instantseal In this way they can use total difficulty fork choice check * Extend blockdetails with metadatas and finalized info * Extra data update: mark_finalized and update_metadatas * Check finalized block in Blockchain * Fix a test constructor in verification mod * Add total difficulty trait * Fix type import * Db migration to V13 with metadata column * Address grumbles * metadatas -> metadata * Use generic type for update_metadata to avoid passing HashMap all around * Remove metadata in blockdetails * [WIP] Implement a generic metadata architecture * [WIP] Metadata insertion logic in BlockChain * typo: Value -> Self::Value * [WIP] Temporarily remove Engine::is_new_best interface So that we don't have too many type errors. * [WIP] Fix more type errors * [WIP] ExtendedHeader::PartialEq * [WIP] Change metadata type Option<Vec<u8>> to Vec<u8> * [WIP] Remove Metadata Error * [WIP] Clean up error conversion * [WIP] finalized -> is_finalized * [WIP] Mark all fields in ExtrasInsert as pub * [WIP] Remove unused import * [WIP] Keep only local metadata info * Mark metadata as optional * [WIP] Revert metadata db change in BlockChain * [WIP] Put finalization in unclosed state * Use metadata interface in BlockDetail * [WIP] Fix current build failures * [WIP] Remove unused blockmetadata struct * Remove DB migration info * [WIP] Typo * Use ExtendedHeader to implement fork choice check * Implement is_new_best using Ancestry iterator * Use expect instead of panic * [WIP] Add ancestry Engine support via on_new_block * Fix tests * Emission of ancestry actions * use_short_version should take account of metadata * Engine::is_new_best -> Engine::fork_choice * Use proper expect format as defined in #1026 * panic -> expect * ancestry_header -> ancestry_with_metadata * Boxed iterator -> &mut iterator * Fix tests * is_new_best -> primitive_fork_choice * Document how fork_choice works * Engine::fork_choice -> Engine::primitive_fork_choice * comment: clarify types of finalization where Engine::primitive_fork_choice works * Expose FinalizationInfo to Engine * Fix tests due to merging * Remove TotalDifficulty trait * Do not pass FinalizationInfo to Engine If there's finalized blocks in from route, choose the old branch without calling `Engine::fork_choice`. * Fix compile * Fix unused import * Remove is_to_route_finalized When no block reorg passes a finalized block, this variable is always false. * Address format grumbles * Fix docs: mark_finalized returns None if block hash is not found `blockchain` mod does not yet have an Error type, so we still temporarily use None here. * Fix inaccurate tree_route None expect description * typo (#8640) * Changelog for 1.10.4-stable and 1.11.1-beta (#8637) * Add changelog for 1.10.4 * Add changelog for 1.11.1 * Fix Typos * Don't open Browser post-install on Mac (#8641) Since we start parity with the UI disabled per default now, opening the browser post installation will show an annoying error message, confusing the user. This patch removes opening the browser to prevent that annoyance. fixes #8194 * Resumable warp-sync / Seed downloaded snapshots (#8544) * Start dividing sync chain : first supplier method * WIP - updated chain sync supplier * Finish refactoring the Chain Sync Supplier * Create Chain Sync Requester * Add Propagator for Chain Sync * Add the Chain Sync Handler * Move tests from mod -> handler * Move tests to propagator * Refactor SyncRequester arguments * Refactoring peer fork header handler * Fix wrong highest block number in snapshot sync * Small refactor... * Resume warp-sync downloaded chunks * Add comments * Refactoring the previous chunks import * Fix tests * Address PR grumbles * Fix not seeding current snapshot * Address PR Grumbles * Address PR grumble * Retry failed CI job * Update SnapshotService readiness check Fix restoration locking issue for previous chunks restoration * Fix tests * Fix tests * Fix test * Early abort importing previous chunks * PR Grumbles * Update Gitlab CI config * SyncState back to Waiting when Manifest peers disconnect * Move fix * Better fix * Revert GitLab CI changes * Fix Warning * Refactor resuming snapshots * Fix string construction * Revert "Refactor resuming snapshots" This reverts commit 75fd4b5. * Update informant log * Fix string construction * Refactor resuming snapshots * Fix informant * PR Grumbles * Update informant message : show chunks done * PR Grumbles * Fix * Fix Warning * PR Grumbles * Fix not downloading old blocks (#8642) * Remove HostInfo::next_nonce (#8644) * Remove the Keccak C library and use the pure Rust impl (#8657) * Add license and readme * Use pure rust implementation * Bump version to 0.1.1 * Don't use C, prefer the pure Rust implementation * Add test for `write_keccak` * Bump version * Add benchmarks * Add benchmarks * Add keccak_256, keccak_512, keccak_256_unchecked and keccak_512_unchecked – mostly for compatibility with ethash * Remove failed git merge attempt from external git repo Cargo.lock updates * whitespace * Mark unsafe function unsafe * Unsafe calls in unsafe block * Document unsafety invariants * Revert unintended changes to Cargo.lock * updated tiny-keccak to 1.4.2 (#8669) * parity: improve cli help and logging (#8665) * parity: indicate disabling ancient blocks is not recommended * parity: display decimals for stats in informant * Refactor EIP150, EIP160 and EIP161 forks to be specified in CommonParams (#8614) * Allow post-homestead forks to be specified in CommonParams * Fix all json configs * Fix test in json crate * Fix test in ethcore * Fix all chain configs to use tabs Given we use tabs in .editorconfig and the majority of chain configs. This change is done in Emacs using `mark-whole-buffer` and `indent-region`. * Remove HostInfo::client_version() and secret() (#8677) * Move connection_filter to the network crate (#8674) * Remove the error when stopping the network (#8671) * Allow making direct RPC queries from the C API (#8588) * Fix cli signer (#8682) * Update ethereum-types so `{:#x}` applies 0x prefix * Fix tx permission tests * Use impl Future in the light client RPC helpers (#8628) * Update mod.rs (#8695) - Update interfaces affected by unsafe-expose - replace `{{ }}` as this throws an error in Jekyll (wiki) * remove empty file (#8705) * Set the request index to that of the current request (#8683) * Set the request index to that of the current request When setting up the chain of (two) requests to look up a block by hash, the second need to refer to the first. This fixes an issue where the back ref was set to the subsequent request, not the current one. When the requests are executed we loop through them in order and ensure the requests that should produce headers all match up. We do this by index so they better be right. In other words: off by one. * parity: trim whitespace when parsing duration strings (#8692) * Implement recursive Debug for Nodes in patrica_trie::TrieDB (#8697) fixes #8184 * Remove unused imports (#8722) * Update dev chain (#8717) * Make dev chain more foundation-like and enable wasm. * Fix compilation warnings. * Add a test for decoding corrupt data (#8713) * Fix compilation error on nightly rust (#8707) On nightly rust passing `public_url` works but that breaks on stable. This works for both. * network-devp2p: handle UselessPeer disconnect (#8686) * Shutdown the Snapshot Service early (#8658) * Shutdown the Snapshot Service when shutting down the runner * Rename `service` to `client_service` * Fix tests * Fix local transactions policy. (#8691) * Add a deadlock detection thread (#8727) * Add a deadlock detection thread Expose it under a feature flag: `cargo build --features "deadlock_detection"` * Address Nicklas's comments * Remove unused function new_pow_test_spec (#8735) * Add 'interface' option to cli (#8699) Additionally as to the port, the new command line option now allows the user to specify the network interface the P2P-Parity listens, too. With support for 'all' and 'local' like in all other versions of this flag. Default is 'all' (aka ). * Fix some nits using clippy (#8731) * fix some nits using clippy * fix tests * Remove a couple of unnecessary `transmute()` (#8736) * bump tinykeccak to 1.4 (#8728) * ease tiny-keccak version requirements (1.4.1 -> 1.4) (#8726) * Remove -k/--insecure option from curl installer (#8719) Piping `curl` to `bash` while **disabling** certificate verification can lead to security problems. * Fix PoW blockchains sealing notifications in chain_new_blocks (#8656) * Print warnings when fetching pending blocks (#8711) * Lots of println to figure out what eth_getBlockByNumber does/should do * Remove debugging * Print warnings when fetching pending blocks When calling `eth_getBlockByNumber` with `pending`, we now print a deprecation warning and: * if a pending block is found, use it to respond * if no pending block is found, respond as if if was a request for `Latest` Addresses issue #8703 (not sure if it's enough to close it tbh) * Fix XOR distance calculation in discovery Kademlia impl (#8589) * network-devp2p: Test for discovery bucket insertion. All test values are randomly generated and the assertions are checked manually. Test fails because distance metric is implemented incorrectly. * network-devp2p: Fix discovery distance function. The Kademlia distance function (XOR) was implemented incorrectly as a population count. * network-devp2p: Refactor nearest_node_entries to be on instance. Optimizations are possible with more access to the discovery state. * network-devp2p: Fix loss of precision in nearest_node_entries. * network-devp2p: More efficient nearest node search. The discovery algorithm to identify the nearest k nodes does not need to scan all entries in all buckets. * Remove NetworkService::config() (#8653) * CI: Fixes for Android Pipeline (#8745) * ci: Remove check for shared libraries in gitlab script * ci: allow android arm build to fail * Custom Error Messages on ENFILE and EMFILE IO Errors (#8744) * Custom Error Messages on ENFILE and EMFILE IO Errors Add custom mapping of ENFILE and EMFILE IO Errors (Failure because of missing system resource) right when chaining ioError into ::util::Network::Error to improve Error Messages given to user Note: Adds libc as a dependency to util/network * Use assert-matches for more readable tests * Fix Wording and consistency

@andresilva

* Tx permission contract improvement * Use tuple for to address * Introduced ABI for deprecated tx permission contract * Improved ABI for tx permission contract with contract name, name hash and version * Introduced support for deprecated tx permission contract + fixed cache for the new version + introduced `target` for tx filter loging * Introduced test for the new tx permission contract version + old test renamed as deprecated * Removed empty lines * Introduced filter_only_sender return value in allowedTxTypes fn + improved caching * Introduced version checking for tx permission contract * Moved tx permission contract test genesis specs to separate files * handle queue import errors a bit more gracefully (#8385) * Some tweaks to main.rs for parity as a library (#8370) * Some tweaks to main.rs for parity as a library * Remove pub from PostExecutionAction * New Transaction Queue implementation (#8074) * Implementation of Verifier, Scoring and Ready. * Queue in progress. * TransactionPool. * Prepare for txpool release. * Miner refactor [WiP] * WiP reworking miner. * Make it compile. * Add some docs. * Split blockchain access to a separate file. * Work on miner API. * Fix ethcore tests. * Refactor miner interface for sealing/work packages. * Implement next nonce. * RPC compiles. * Implement couple of missing methdods for RPC. * Add transaction queue listeners. * Compiles! * Clean-up and parallelize. * Get rid of RefCell in header. * Revert "Get rid of RefCell in header." This reverts commit 0f2424c9b7319a786e1565ea2a8a6d801a21b4fb. * Override Sync requirement. * Fix status display. * Unify logging. * Extract some cheap checks. * Measurements and optimizations. * Fix scoring bug, heap size of bug and add cache * Disable tx queueing and parallel verification. * Make ethcore and ethcore-miner compile again. * Make RPC compile again. * Bunch of txpool tests. * Migrate transaction queue tests. * Nonce Cap * Nonce cap cache and tests. * Remove stale future transactions from the queue. * Optimize scoring and write some tests. * Simple penalization. * Clean up and support for different scoring algorithms. * Add CLI parameters for the new queue. * Remove banning queue. * Disable debug build. * Change per_sender limit to be 1% instead of 5% * Avoid cloning when propagating transactions. * Remove old todo. * Post-review fixes. * Fix miner options default. * Implement back ready transactions for light client. * Get rid of from_pending_block * Pass rejection reason. * Add more details to drop. * Rollback heap size of. * Avoid cloning hashes when propagating and include more details on rejection. * Fix tests. * Introduce nonces cache. * Remove uneccessary hashes allocation. * Lower the mem limit. * Re-enable parallel verification. * Add miner log. Don't check the type if not below min_gas_price. * Add more traces, fix disabling miner. * Fix creating pending blocks twice on AuRa authorities. * Fix tests. * re-use pending blocks in AuRa * Use reseal_min_period to prevent too frequent update_sealing. * Fix log to contain hash not sender. * Optimize local transactions. * Fix aura tests. * Update locks comments. * Get rid of unsafe Sync impl. * Review fixes. * Remove excessive matches. * Fix compilation errors. * Use new pool in private transactions. * Fix private-tx test. * Fix secret store tests. * Actually use gas_floor_target * Fix config tests. * Fix pool tests. * Address grumbles. * clarify that windows need perl and yasm (#8402) * Unify and limit rocksdb dependency places (#8371) * secret_store: remove kvdb_rocksdb dependency * cli: init db mod for open dispatch * cli: move db, client_db, restoration_db, secretstore_db to a separate mod * migration: rename to migration-rocksdb and remove ethcore-migrations * ethcore: re-move kvdb-rocksdb dep to test * mark test_helpers as test only and fix migration mod naming * Move restoration_db_handler to test_helpers_internal * Fix missing preambles in test_helpers_internal and rocksdb/helpers * Move test crates downward * Fix missing docs * cli, db::open_db: move each argument to a separate line * Use featuregate instead of dead code for `open_secretstore_db` * Move pathbuf import to open_secretstore_db Because it's only used there behind a feature gate * Use tokio::spawn in secret_store listener and fix Uri (#8373) * Directly wait for future to resolve in a threadpool * Ignore return value * Use path.starts_with instead of req_uri.is_absolute The later now means something else in hyper 0.11.. * Use tokio::spawn * typo: remove accidential unsafe impl * remove Tendermint extra_info due to seal inconsistencies (#8367) * More code refactoring to integrate Duration (#8322) * More code refactoring to integrate Duration * Fix typo * Fix tests * More test fix * tokio-core v0.1.16 -> v0.1.17 (#8408) * Replace legacy Rlp with UntrustedRlp and use in ethcore rlp views (#8316) * WIP * Replace Rlp with UntrustedRlp in views, explicity unwrap with expect First pass to get it to compile. Need to figure out whether to do this or to propogate Errors upstream, which would require many more changes to dependent code. If we do this way we are assuming that the views are always used in a context where the rlp is trusted to be valid e.g. when reading from our own DB. So need to fid out whether views are used with data received from an untrusted (e.g. extrernal peer). * Remove original Rlp impl, rename UntrustedRlp -> Rlp * Create rlp views with view! macro to record debug info Views are assumed to be over valid rlp, so if there is a decoding error we record where the view was created in the first place and report it in the expect * Use $crate in view! macro to avoid import, fix tests * Expect valid rlp in decode functions for now * Replace spaces with tabs in new file * Add doc tests for creating views with macro * Update rlp docs to reflect removing of UntrustedRlp * Replace UntrustedRlp usages in private-tx merge * Fix TODO comments (#8413) * update zip to 0.3 (#8381) * update zip to 0.3 * enable zip deflate feature * typo, docs parity_chainId: empty string -> None (#8434) * Fix receipts stripping. (#8414) * Changelogs for 1.9.6 and 1.10.1 (#8411) * Add changelog for 1.9.6 * Add Changelog for 1.10.1 * Move ethcore::Error to error_chain (#8386) * WIP * Convert Ethcore error to use error_chain * Use error_chain for ImportError and BlockImportError * Fix error pattern matches for error_chain in miner * Implement explicit From for AccountsError * Fix pattern matches for ErrorKinds * Handle ethcore error_chain in light client * Explicitly define Result type to avoid shadowing * Fix remaining Error pattern matches * Fix tab space formatting * Helps if the tests compile * Fix error chain matching after merge * remove From::from. (#8390) * Some tiny modifications. 1. fix some typo in the comment. 2. sort the order of methods in 'impl state::Backend for StateDB` * Remove the clone of code_cache, as it has been done in clone_basic. * remove From::from. It seems not necessary. * Use forked app_dirs crate for reverted Windows dir behavior (#8438) * Remove unused appdirs dependency in CLI * Use forked app_dirs crate for reverted Windows dir behavior * Permission fix (#8441) * Block reward contract (#8419) * engine: add block reward contract abi and helper client * aura: add support for block reward contract * engine: test block reward contract client * aura: test block reward contract * engine + aura: add missing docs * engine: share SystemCall type alias * aura: add transition for block reward contract * engine: fix example block reward contract source link and bytecode * Improve VM executor stack size estimation rules (#8439) * Improve VM executor stack size estimation rules * typo: docs add "(Debug build)" comment * Fix an off by one typo and set minimal stack size This avoids the case if `depth_threshold == max_depth`. Usually setting stack size to zero will just rebound it to platform minimal stack size, but we set it here just in case. * Use saturating_sub to avoid potential overflow * Private transactions processing error handling (#8431) * Integration test for private transaction returned * Do not interrupt verification in case of errors * Helpers use specified * Review comments fixed * Update Cargo hidapi-rs dependency (#8447) * Allow 32 bit pipelines to fail (#8454) * Disable 32bit tragets for gitlab * Rename linux pipelines * Update wasmi (#8452) * Return error in case eth_call returns VM errors (#8448) * Add VMError generator * Return executed exceptions in eth_call * ParityShell::open `Return result` (#8377) * start * add error handling for winapi * fix typo * fix warnings and windows errors * formatting * Address review comments * fix docker build (#8462) * Add changelog for 1.9.7 and 1.10.2 (#8460) * Add changelog for 1.9.7 * Add Changelog for 1.10.2 * Apply proper markdown * Run a spellchecker :) * Be pedantic about the 32-bit pipelines :) * fix typos in vm description comment (#8446) * Use rename_all for RichBlock and RichHeader serialization (#8471) * typo: fix a resolved TODO comment * Use rename_all instead of individual renames * Don't require write lock when fetching status. (#8481) * Bump master to 1.12 (#8477) * Bump master to 1.12 * Bump crates to 1.12 * Bump mac installer version to 1.12 * Update Gitlab scripts * Fix snap builds (#8483) * Update hardcodedSync for Ethereum, Kovan, and Ropsten (#8489) * Update wasmi and pwasm-utils (#8493) * Update wasmi to 0.2 New wasmi supports 32bit platforms and no longer requires a special feature to build for such platforms. * Update pwasm-utils to 0.1.5 * Remove three old warp boot nodes. (#8497) * Return error if RLP size of transaction exceeds the limit (#8473) * Return error if RLP size of transaction exceeds the limit * Review comments fixed * RLP check moved to verifier, corresponding pool test added * `duration_ns: u64 -> duration: Duration` (#8457) * duration_ns: u64 -> duration: Duration * format on millis {:.2} -> {} * Remove unused dependency `bigint` (#8505) * remove unused dependency bigint in * remove bigint in rpc_cli * Show imported messages for light client (#8517) * Directly return None if tracing is disabled (#8504) * Directly return None if tracing is disabled * Address gumbles: release read locks as fast as possible * Hardware Wallet trait (#8071) * getting started with replacing HardwareWalletManager * trezor and ledger impls the new trait with some drawbacks * Everything move to the new trait * It required lifetime annotations in the trait because [u8] in unsized * Lets now start moving entry point from HardwareWalletManager * rename trait to Wallet * move thread management to the actual wallets * Moved thread management to each respective Wallet * Cleaned up pub items that is needed to be pub * Wallet trait more or less finished * Cleaned up docs * fix tests * omit removed docs * fix spelling, naming och remove old comments * ledger test is broken, add correct logging format * So locally on my machine Linux Ubuntu 17.10 the test doesn't panic but on the CI server libusb::Context::new() fails which I don't understand because it has worked before * Additionally the ledger test is optional so I lean toward ignoring it the CI Server * ignore hardware tests by default * more verbose checking in ledger test * SecretStore: merge two types of errors into single one + Error::is_non_fatal (#8357) * SecretStore: error unify initial commit SecretStore: pass real error in error messages SecretStore: is_internal_error -> Error::is_non_fatal warnings SecretStore: ConsensusTemporaryUnreachable fix after merge removed comments removed comments SecretStore: updated HTTP error responses SecretStore: more ConsensusTemporaryUnreachable tests fix after rebase * fixed grumbles * use HashSet in tests * Enable WebAssembly and Byzantium for Ellaism (#8520) * Enable WebAssembly and Byzantium for Ellaism * Fix indentation * Remove empty lines * More changes for Android (#8421) * Transaction Pool improvements (#8470) * Don't use ethereum_types in transaction pool. * Hide internal insertion_id. * Fix tests. * Review grumbles. * Fetching logs by hash in blockchain database (#8463) * Fetch logs by hash in blockchain database * Fix tests * Add unit test for branch block logs fetching * Add docs that blocks must already be sorted * Handle branch block cases properly * typo: empty -> is_empty * Remove return_empty_if_none by using a closure * Use BTreeSet to avoid sorting again * Move is_canon to BlockChain * typo: pass value by reference * Use loop and wrap inside blocks to simplify the code Borrowed from openethereum/parity-ethereum#8463 (comment) * typo: missed a comment * Pass on storage keys tracing to handle the case when it is not modified (#8491) * Pass on storage keys even if it is not modified * typo: account and storage query `to_pod_diff` builds both `touched_addresses` merge and storage keys merge. * Fix tests * Use state query directly because of suicided accounts * Fix a RefCell borrow issue * Add tests for unmodified storage trace * Address grumbles * typo: remove unwanted empty line * ensure_cached compiles with the original signature * Don't panic in import_block if invalid rlp (#8522) * Don't panic in import_block if invalid rlp * Remove redundant type annotation * Replace RLP header view usage with safe decoding Using the view will panic with invalid RLP. Here we use Rlp decoding directly which will return a `Result<_, DecoderError>`. While this path currently should not have any invalid RLP - it makes it safer if ever called with invalid RLP from other code paths. * Remove expect (#8536) * Remove expect and propagate rlp::DecoderErrors as TrieErrors * EIP 145: Bitwise shifting instructions in EVM (#8451) * Add SHL, SHR, SAR opcodes * Add have_bitwise_shifting schedule flag * Add all EIP tests for SHL * Add SHR implementation and tests * Implement SAR and add tests * Add eip145transition config param * Change map_or to map_or_else when possible * Consolidate crypto functionality in `ethcore-crypto`. (#8432) * Consolidate crypto functionality in `ethcore-crypto`. - Move `ecdh`/`ecies` modules to `ethkey`. - Refactor `ethcore-crypto` to use file per module. - Replace `subtle` with `ethcore_crypto::is_equal`. - Add `aes_gcm` module to `ethcore-crypto`. * Rename `aes::{encrypt,decrypt,decrypt_cbc}` ... ... to `aes::{encrypt_128_ctr,decrypt_128_ctr,decrypt_128_cbc}`. * ethcore, rpc, machine: refactor block reward application and tracing (#8490) * Keep all enacted blocks notify in order (#8524) * Keep all enacted blocks notify in order * Collect is unnecessary * Update ChainNotify to use ChainRouteType * Fix all ethcore fn defs * Wrap the type within ChainRoute * Fix private-tx and sync api * Fix secret_store API * Fix updater API * Fix rpc api * Fix informant api * Eagerly cache enacted/retracted and remove contain_enacted/retracted * Fix indent * tests: should use full expr form for struct constructor * Use into_enacted_retracted to further avoid copy * typo: not a function * rpc/tests: ChainRoute -> ChainRoute::new * Node table sorting according to last contact data (#8541) * network-devp2p: sort nodes in node table using last contact data * network-devp2p: rename node contact types in node table json output * network-devp2p: fix node table tests * network-devp2p: note node failure when failed to establish connection * network-devp2p: handle UselessPeer error * network-devp2p: note failure when marking node as useless * Rlp decode returns Result (#8527) rlp::decode returns Result Make a best effort to handle decoding errors gracefully throughout the code, using `expect` where the value is guaranteed to be valid (and in other places where it makes sense). * Parity as a library (#8412) * Parity as a library * Fix concerns * Allow using a null on_client_restart_cb * Fix more concerns * Test the C library in test.sh * Reduce CMake version to 3.5 * Move the clib test before cargo test * Add println in test * Trace precompiled contracts when the transfer value is not zero (#8486) * Trace precompiled contracts when the transfer value is not zero * Add tests for precompiled CALL tracing * Use byzantium test machine for the new test * Add notes in comments on why we don't trace all precompileds * Use is_transferred instead of transferred * Don't block sync when importing old blocks (#8530) * Alter IO queueing. * Don't require IoMessages to be Clone * Ancient blocks imported via IoChannel. * Get rid of private transactions io message. * Get rid of deadlock and fix disconnected handler. * Revert to old disconnect condition. * Fix tests. * Fix deadlock. * Make trace-time publishable. (#8568) * Remove State::replace_backend (#8569) * Refactoring `ethcore-sync` - Fixing warp-sync barrier (#8543) * Start dividing sync chain : first supplier method * WIP - updated chain sync supplier * Finish refactoring the Chain Sync Supplier * Create Chain Sync Requester * Add Propagator for Chain Sync * Add the Chain Sync Handler * Move tests from mod -> handler * Move tests to propagator * Refactor SyncRequester arguments * Refactoring peer fork header handler * Fix wrong highest block number in snapshot sync * Small refactor... * Address PR grumbles * Retry failed CI job * Fix tests * PR Grumbles * Decoding headers can fail (#8570) * rlp::decode returns Result * Fix journaldb to handle rlp::decode Result * Fix ethcore to work with rlp::decode returning Result * Light client handles rlp::decode returning Result * Fix tests in rlp_derive * Fix tests * Cleanup * cleanup * Allow panic rather than breaking out of iterator * Let decoding failures when reading from disk blow up * syntax * Fix the trivial grumbles * Fix failing tests * Make Account::from_rlp return Result * Syntx, sigh * Temp-fix for decoding failures * Header::decode returns Result Handle new return type throughout the code base. * Do not continue reading from the DB when a value could not be read * Fix tests * Handle header decoding in light_sync * Handling header decoding errors * Let the DecodeError bubble up unchanged * Remove redundant error conversion * Update CHANGELOG for 1.9, 1.10, and 1.11 (#8556) * Move changelog for 1.10.x * Mark 1.9 EOL * Prepare changelog for 1.10.3 stable * Prepare changelog for 1.11.0 stable * Update changelogs * Update CHANGELOG for 1.10.3 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Format changelog * Handle socket address parsing errors (#8545) Unpack errors and check for io::ErrorKind::InvalidInput and return our own AddressParse error. Remove the foreign link to std::net::AddrParseError and add an `impl From` for that error. Test parsing properly. * Remove unnecessary cloning in overwrite_with (#8580) * Remove unnecessary cloning in overwrite_with * Remove into_iter * changelog nit (#8585) * Rename `whisper-cli binary` to `whisper` (#8579) * rename whisper-cli binary to whisper * fix tests * Add whisper CLI to the pipelines (#8578) * Add whisper CLI to the pipelines * Address todo, ref #8579 * Added Dockerfile for alpine linux by @andresilva, closes #3565 (#8587) * Changelog and Readme (#8591) * Move changelog for 1.10.x * Mark 1.9 EOL * Prepare changelog for 1.10.3 stable * Prepare changelog for 1.11.0 stable * Update changelogs * Update CHANGELOG for 1.10.3 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Update CHANGELOG for 1.11.0 beta * Format changelog * Update README for 1.11 * Fix typo * Attempt to fix intermittent test failures (#8584) Occasionally should_return_correct_nonces_when_dropped_because_of_limit fails, possibly because of multiple threads competing to finish. See CI logs here for an example: https://gitlab.parity.io/parity/parity/-/jobs/86738 * Make mio optional in ethcore-io (#8537) * Make mio optional in ethcore-io * Add some annotations, plus a check for features * Increase timer for test * Fix Parity UI link (#8600) Fix link openethereum/parity-ethereum#8599 * fix compiler warning (#8590) * Block::decode() returns Result (#8586) * block_header can fail so return Result (#8581) * block_header can fail so return Result * Restore previous return type based on feedback * Fix failing doc tests running on non-code * Remove inject.js server-side injection for dapps (#8539) * Remove inject.js server-side injection for dapps * Remove dapps test `should_inject_js` Parity doesn't inject a <script> tag inside the responses anymore * Fix the mio test again (#8602) * 2 tiny modification on snapshot (#8601) * Some tiny modifications. 1. fix some typo in the comment. 2. sort the order of methods in 'impl state::Backend for StateDB` * Remove the clone of code_cache, as it has been done in clone_basic. * remove From::from. It seems not necessary. * change mode: remove rust files' executable mode. * 2 tiny modifications on snapshot. * Use full qualified syntax for itertools::Itertools::flatten (#8606) * Fix packet count when talking with PAR2 peers (#8555) * Support diferent packet counts in different protocol versions. * Fix light timeouts and eclipse protection. * Fix devp2p tests. * Fix whisper-cli compilation. * Fix compilation. * Fix ethcore-sync tests. * Revert "Fix light timeouts and eclipse protection." This reverts commit 06285ea8c1d9d184d809f64b5507aece633da6cc. * Increase timeouts. * typo: wrong indentation in kovan config (#8610) * Fix account list double 0x display (#8596) * Remove unused self import * Fix account list double 0x display * Remove manually added text to the errors (#8595) These messages were confusing for the users especially the help message. * Gitlab test script fixes (#8573) * Exclude /docs from modified files. * Ensure all references in the working tree are available * Remove duplicated line from test script * Fix BlockReward contract "arithmetic operation overflow" (#8611) * Fix BlockReward contract "arithmetic operation overflow" * Add docs on how execute_as_system works * Fix typo * ´main.rs´ typo (#8629) * Typo * Update main.rs * Store morden db and keys in "path/to/parity/data/Morden" (ropsten uses "test", like before) (#8621) * Store morden db and keys in "path/to/parity/data/morden" (ropsten uses "test", like before) * Fix light sync with initial validator-set contract (#8528) * Fix #8468 * Use U256::max_value() instead * Fix again * Also change initial transaction gas * Remove NetworkContext::io_channel() (#8625) * Remove io_channel() * Fix warning * Check that the Android build doesn't dep on c++_shared (#8538) * Fork choice and metadata framework for Engine (#8401) * Add light client TODO item * Move existing total-difficulty-based fork choice check to Engine * Abstract total difficulty and block provider as Machine::BlockMetadata and Machine::BlockProvider * Decouple "generate_metadata" logic to Engine * Use fixed BlockMetadata and BlockProvider type for null and instantseal In this way they can use total difficulty fork choice check * Extend blockdetails with metadatas and finalized info * Extra data update: mark_finalized and update_metadatas * Check finalized block in Blockchain * Fix a test constructor in verification mod * Add total difficulty trait * Fix type import * Db migration to V13 with metadata column * Address grumbles * metadatas -> metadata * Use generic type for update_metadata to avoid passing HashMap all around * Remove metadata in blockdetails * [WIP] Implement a generic metadata architecture * [WIP] Metadata insertion logic in BlockChain * typo: Value -> Self::Value * [WIP] Temporarily remove Engine::is_new_best interface So that we don't have too many type errors. * [WIP] Fix more type errors * [WIP] ExtendedHeader::PartialEq * [WIP] Change metadata type Option<Vec<u8>> to Vec<u8> * [WIP] Remove Metadata Error * [WIP] Clean up error conversion * [WIP] finalized -> is_finalized * [WIP] Mark all fields in ExtrasInsert as pub * [WIP] Remove unused import * [WIP] Keep only local metadata info * Mark metadata as optional * [WIP] Revert metadata db change in BlockChain * [WIP] Put finalization in unclosed state * Use metadata interface in BlockDetail * [WIP] Fix current build failures * [WIP] Remove unused blockmetadata struct * Remove DB migration info * [WIP] Typo * Use ExtendedHeader to implement fork choice check * Implement is_new_best using Ancestry iterator * Use expect instead of panic * [WIP] Add ancestry Engine support via on_new_block * Fix tests * Emission of ancestry actions * use_short_version should take account of metadata * Engine::is_new_best -> Engine::fork_choice * Use proper expect format as defined in #1026 * panic -> expect * ancestry_header -> ancestry_with_metadata * Boxed iterator -> &mut iterator * Fix tests * is_new_best -> primitive_fork_choice * Document how fork_choice works * Engine::fork_choice -> Engine::primitive_fork_choice * comment: clarify types of finalization where Engine::primitive_fork_choice works * Expose FinalizationInfo to Engine * Fix tests due to merging * Remove TotalDifficulty trait * Do not pass FinalizationInfo to Engine If there's finalized blocks in from route, choose the old branch without calling `Engine::fork_choice`. * Fix compile * Fix unused import * Remove is_to_route_finalized When no block reorg passes a finalized block, this variable is always false. * Address format grumbles * Fix docs: mark_finalized returns None if block hash is not found `blockchain` mod does not yet have an Error type, so we still temporarily use None here. * Fix inaccurate tree_route None expect description * typo (#8640) * Changelog for 1.10.4-stable and 1.11.1-beta (#8637) * Add changelog for 1.10.4 * Add changelog for 1.11.1 * Fix Typos * Don't open Browser post-install on Mac (#8641) Since we start parity with the UI disabled per default now, opening the browser post installation will show an annoying error message, confusing the user. This patch removes opening the browser to prevent that annoyance. fixes #8194 * Resumable warp-sync / Seed downloaded snapshots (#8544) * Start dividing sync chain : first supplier method * WIP - updated chain sync supplier * Finish refactoring the Chain Sync Supplier * Create Chain Sync Requester * Add Propagator for Chain Sync * Add the Chain Sync Handler * Move tests from mod -> handler * Move tests to propagator * Refactor SyncRequester arguments * Refactoring peer fork header handler * Fix wrong highest block number in snapshot sync * Small refactor... * Resume warp-sync downloaded chunks * Add comments * Refactoring the previous chunks import * Fix tests * Address PR grumbles * Fix not seeding current snapshot * Address PR Grumbles * Address PR grumble * Retry failed CI job * Update SnapshotService readiness check Fix restoration locking issue for previous chunks restoration * Fix tests * Fix tests * Fix test * Early abort importing previous chunks * PR Grumbles * Update Gitlab CI config * SyncState back to Waiting when Manifest peers disconnect * Move fix * Better fix * Revert GitLab CI changes * Fix Warning * Refactor resuming snapshots * Fix string construction * Revert "Refactor resuming snapshots" This reverts commit 75fd4b553a38e4a49dc5d6a878c70e830ff382eb. * Update informant log * Fix string construction * Refactor resuming snapshots * Fix informant * PR Grumbles * Update informant message : show chunks done * PR Grumbles * Fix * Fix Warning * PR Grumbles * Fix not downloading old blocks (#8642) * Remove HostInfo::next_nonce (#8644) * Remove the Keccak C library and use the pure Rust impl (#8657) * Add license and readme * Use pure rust implementation * Bump version to 0.1.1 * Don't use C, prefer the pure Rust implementation * Add test for `write_keccak` * Bump version * Add benchmarks * Add benchmarks * Add keccak_256, keccak_512, keccak_256_unchecked and keccak_512_unchecked – mostly for compatibility with ethash * Remove failed git merge attempt from external git repo Cargo.lock updates * whitespace * Mark unsafe function unsafe * Unsafe calls in unsafe block * Document unsafety invariants * Revert unintended changes to Cargo.lock * updated tiny-keccak to 1.4.2 (#8669) * parity: improve cli help and logging (#8665) * parity: indicate disabling ancient blocks is not recommended * parity: display decimals for stats in informant * Refactor EIP150, EIP160 and EIP161 forks to be specified in CommonParams (#8614) * Allow post-homestead forks to be specified in CommonParams * Fix all json configs * Fix test in json crate * Fix test in ethcore * Fix all chain configs to use tabs Given we use tabs in .editorconfig and the majority of chain configs. This change is done in Emacs using `mark-whole-buffer` and `indent-region`. * Remove HostInfo::client_version() and secret() (#8677) * Move connection_filter to the network crate (#8674) * Remove the error when stopping the network (#8671) * Allow making direct RPC queries from the C API (#8588) * Fix cli signer (#8682) * Update ethereum-types so `{:#x}` applies 0x prefix * Fix tx permission tests * Use impl Future in the light client RPC helpers (#8628) * Update mod.rs (#8695) - Update interfaces affected by unsafe-expose - replace `{{ }}` as this throws an error in Jekyll (wiki) * remove empty file (#8705) * Set the request index to that of the current request (#8683) * Set the request index to that of the current request When setting up the chain of (two) requests to look up a block by hash, the second need to refer to the first. This fixes an issue where the back ref was set to the subsequent request, not the current one. When the requests are executed we loop through them in order and ensure the requests that should produce headers all match up. We do this by index so they better be right. In other words: off by one. * parity: trim whitespace when parsing duration strings (#8692) * Implement recursive Debug for Nodes in patrica_trie::TrieDB (#8697) fixes #8184 * Remove unused imports (#8722) * Update dev chain (#8717) * Make dev chain more foundation-like and enable wasm. * Fix compilation warnings. * Add a test for decoding corrupt data (#8713) * Fix compilation error on nightly rust (#8707) On nightly rust passing `public_url` works but that breaks on stable. This works for both. * network-devp2p: handle UselessPeer disconnect (#8686) * Shutdown the Snapshot Service early (#8658) * Shutdown the Snapshot Service when shutting down the runner * Rename `service` to `client_service` * Fix tests * Fix local transactions policy. (#8691) * Add a deadlock detection thread (#8727) * Add a deadlock detection thread Expose it under a feature flag: `cargo build --features "deadlock_detection"` * Address Nicklas's comments * Remove unused function new_pow_test_spec (#8735) * Add 'interface' option to cli (#8699) Additionally as to the port, the new command line option now allows the user to specify the network interface the P2P-Parity listens, too. With support for 'all' and 'local' like in all other versions of this flag. Default is 'all' (aka ). * Fix some nits using clippy (#8731) * fix some nits using clippy * fix tests * Remove a couple of unnecessary `transmute()` (#8736) * bump tinykeccak to 1.4 (#8728) * ease tiny-keccak version requirements (1.4.1 -> 1.4) (#8726) * Remove -k/--insecure option from curl installer (#8719) Piping `curl` to `bash` while **disabling** certificate verification can lead to security problems. * Fix PoW blockchains sealing notifications in chain_new_blocks (#8656) * Print warnings when fetching pending blocks (#8711) * Lots of println to figure out what eth_getBlockByNumber does/should do * Remove debugging * Print warnings when fetching pending blocks When calling `eth_getBlockByNumber` with `pending`, we now print a deprecation warning and: * if a pending block is found, use it to respond * if no pending block is found, respond as if if was a request for `Latest` Addresses issue #8703 (not sure if it's enough to close it tbh) * Fix XOR distance calculation in discovery Kademlia impl (#8589) * network-devp2p: Test for discovery bucket insertion. All test values are randomly generated and the assertions are checked manually. Test fails because distance metric is implemented incorrectly. * network-devp2p: Fix discovery distance function. The Kademlia distance function (XOR) was implemented incorrectly as a population count. * network-devp2p: Refactor nearest_node_entries to be on instance. Optimizations are possible with more access to the discovery state. * network-devp2p: Fix loss of precision in nearest_node_entries. * network-devp2p: More efficient nearest node search. The discovery algorithm to identify the nearest k nodes does not need to scan all entries in all buckets. * Remove NetworkService::config() (#8653) * CI: Fixes for Android Pipeline (#8745) * ci: Remove check for shared libraries in gitlab script * ci: allow android arm build to fail * Custom Error Messages on ENFILE and EMFILE IO Errors (#8744) * Custom Error Messages on ENFILE and EMFILE IO Errors Add custom mapping of ENFILE and EMFILE IO Errors (Failure because of missing system resource) right when chaining ioError into ::util::Network::Error to improve Error Messages given to user Note: Adds libc as a dependency to util/network * Use assert-matches for more readable tests * Fix Wording and consistency

sorpaas added 2 commits April 13, 2018 21:14

Add light client TODO item

b07ad5f

Move existing total-difficulty-based fork choice check to Engine

3c891b9

sorpaas added A0-pleasereview 🤓 Pull request needs code review. M4-core ⛓ Core client code / Rust. labels Apr 13, 2018

sorpaas added this to the 1.11 milestone Apr 13, 2018

rphmeier reviewed Apr 13, 2018

View reviewed changes

sorpaas and others added 4 commits April 14, 2018 17:37

Abstract total difficulty and block provider as Machine::BlockMetadat…

d272376

…a and Machine::BlockProvider

Decouple "generate_metadata" logic to Engine

c4d349c

Merge branch 'master' into fork-choice-no-metadata

e1c0252

Use fixed BlockMetadata and BlockProvider type for null and instantseal

1d8149a

In this way they can use total difficulty fork choice check

rphmeier reviewed Apr 14, 2018

View reviewed changes

sorpaas added 3 commits April 16, 2018 19:51

Extend blockdetails with metadatas and finalized info

d3ef877

Extra data update: mark_finalized and update_metadatas

f325298

Check finalized block in Blockchain

6756cef

rphmeier reviewed Apr 16, 2018

View reviewed changes

Fix a test constructor in verification mod

7c958b4

rphmeier reviewed Apr 16, 2018

View reviewed changes

sorpaas added 5 commits April 16, 2018 22:36

Add total difficulty trait

e491abf

Fix type import

5459695

Db migration to V13 with metadata column

7fe0d79

Address grumbles

ce105c6

metadatas -> metadata

45b8ce7

sorpaas added A3-inprogress ⏳ Pull request is in progress. No review needed at this stage. and removed A0-pleasereview 🤓 Pull request needs code review. labels Apr 16, 2018

Use generic type for update_metadata to avoid passing HashMap all around

f86d4cf

sorpaas mentioned this pull request Apr 17, 2018

Check finalized block in blockchain::Blockchain #8410

Closed

sorpaas added 4 commits April 17, 2018 16:30

Merge branch 'finalize' into fork-choice-no-metadata

24b6316

Merge branch 'master' of https://github.com/paritytech/parity into fo…

e797462

…rk-choice-no-metadata

Remove metadata in blockdetails

c6dd811

[WIP] Implement a generic metadata architecture

fb1cc21

Merge branch 'master' of https://github.com/paritytech/parity into fo…

4df05f0

…rk-choice-no-metadata

rphmeier suggested changes May 8, 2018

View reviewed changes

Do not pass FinalizationInfo to Engine

60b6def

If there's finalized blocks in from route, choose the old branch without calling `Engine::fork_choice`.

5chdn added the B7-releasenotes 📜 Changes should be mentioned in the release notes of the next minor version release. label May 9, 2018

Merge branch 'master' of https://github.com/paritytech/parity into fo…

fca4ca1

…rk-choice-no-metadata

sorpaas added 2 commits May 9, 2018 20:45

Fix compile

15a967b

Fix unused import

8d1d98b

rphmeier reviewed May 11, 2018

View reviewed changes

sorpaas added 5 commits May 11, 2018 23:30

Remove is_to_route_finalized

63a3f54

When no block reorg passes a finalized block, this variable is always false.

Address format grumbles

7b0cd66

Fix docs: mark_finalized returns None if block hash is not found

2c17071

`blockchain` mod does not yet have an Error type, so we still temporarily use None here.

Fix inaccurate tree_route None expect description

819b3d8

Merge branch 'master' of https://github.com/paritytech/parity into fo…

58756e4

…rk-choice-no-metadata

rphmeier approved these changes May 15, 2018

View reviewed changes

andresilva approved these changes May 15, 2018

View reviewed changes

sorpaas merged commit 0ecbb3e into openethereum:master May 16, 2018

5chdn added A8-looksgood 🦄 Pull request is reviewed well. and removed A0-pleasereview 🤓 Pull request needs code review. labels May 16, 2018

sorpaas mentioned this pull request Jul 16, 2018

block cleanup #9117

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fork choice and metadata framework for Engine #8401

Fork choice and metadata framework for Engine #8401

sorpaas commented Apr 13, 2018 •

edited

Loading

rphmeier Apr 13, 2018 •

edited

Loading

sorpaas Apr 13, 2018

sorpaas Apr 14, 2018

rphmeier Apr 14, 2018 •

edited

Loading

sorpaas Apr 16, 2018

rphmeier Apr 16, 2018 •

edited

Loading

rphmeier Apr 16, 2018

rphmeier Apr 16, 2018 •

edited

Loading

rphmeier left a comment

sorpaas commented May 9, 2018 •

edited

Loading

rphmeier May 11, 2018

rphmeier May 11, 2018 •

edited

Loading

rphmeier May 11, 2018

sorpaas May 11, 2018

rphmeier May 11, 2018

rphmeier May 11, 2018

rphmeier May 11, 2018

rphmeier May 11, 2018

rphmeier May 11, 2018 •

edited

Loading

rphmeier May 11, 2018

andresilva commented May 15, 2018

Fork choice and metadata framework for Engine #8401

Fork choice and metadata framework for Engine #8401

Conversation

sorpaas commented Apr 13, 2018 • edited Loading

General comments on finalization, total difficulty and metadata

BlockDetails

LiveBlock

parity_machine traits

Engine lifecycle extension

Casper

rphmeier Apr 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier Apr 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

rphmeier left a comment

Choose a reason for hiding this comment

sorpaas commented May 9, 2018 • edited Loading

Choose a reason for hiding this comment

rphmeier May 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier May 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andresilva commented May 15, 2018

sorpaas commented Apr 13, 2018 •

edited

Loading

`BlockDetails`

`LiveBlock`

`parity_machine` traits

rphmeier Apr 13, 2018 •

edited

Loading

rphmeier Apr 14, 2018 •

edited

Loading

rphmeier Apr 16, 2018 •

edited

Loading

rphmeier Apr 16, 2018 •

edited

Loading

sorpaas commented May 9, 2018 •

edited

Loading

rphmeier May 11, 2018 •

edited

Loading

rphmeier May 11, 2018 •

edited

Loading