chore: merge development into feature-dan (#4815)

* fix: batch rewind operations (#4752) Description --- Split rewind DbTx into smaller pieces. How Has This Been Tested? --- I did rewind on 20000+ (empty) blocks. * fix: fix config.toml bug (#4780) Description --- The base node errored when reading the `block_sync_trigger = 5` setting ``` ExitError { exit_code: ConfigError, details: Some("Invalid value for `base_node`: unknown field `block_sync_trigger`, expected one of `override_from`, `unconfirmed_pool`, `reorg_pool`, `service`") } ``` Motivation and Context --- Reading default config settings should not cause an error How Has This Been Tested? --- System level testing * fix(p2p/liveness): remove fallible unwrap (#4784) Description --- Removed stray unwrap in liveness service Motivation and Context --- Caused a base node to panic in stress test conditions. ``` thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: DhtOutboundError(RequesterReplyChannelClosed)', base_layer\p2p\src\services\liveness\service.rs:164:71 ``` How Has This Been Tested? --- Tests pass * fix(tari-script): use tari script encoding for execution stack serde de/serialization (#4791) Description --- - Uses tari script encoding (equivalent to consensus encoding) for `ExecutionStack` serde impl - Rename as_bytes to to_bytes as per rust convention. - adds migration to fix execution stack encoding in db Motivation and Context --- Resolves #4790 How Has This Been Tested? --- Added test to alert if breaking changes occur with serde serialization for execution stack. Manual testing in progress * feat: optimize transaction service queries (#4775) Description --- Transaction service sql db queries must handle `DieselError(DatabaseError(__Unknown, "database is locked"))`. This PR attempts to remove situations where that error may occur under highly busy async cirumstances, specifically: - Combine find and update/write type queries into one. - Add sql transactions around complex tasks. _**Note:** Partial resolution for #4731._ Motivation and Context --- See above. How Has This Been Tested? --- - Passed unit tests. - Passed cucumber tests. - ~~**TODO:**~~ System level tests under stress conditions. * feat: move nonce to first in sha hash (#4778) Description --- This moves the nonce to the front of the hashing order when hashing for the sha3 difficulty. This is done so that mining cannot cache part most the header and only load the nonce in. This forces the miner to hash the complete header each time the nonce chances. Motivation and Context --- Fixes: #4767 How Has This Been Tested? --- Unit tests all pass. * fix(dht): remove some invalid saf failure cases (#4787) Description --- - Ignores nanos for `stored_at` field in StoredMessages - Uses direct u32 <-> i32 conversion - Improve error message if attempting to store an expired message - Discard expired messages immediately - Debug log when remote client closes the connection in RPC server Motivation and Context --- - Nano conversion will fail when >= 2_000_000_000, nanos are not important to preserve so we ignore them (set to zero) - u32 to/from i32 conversion does not lose any data as both are 32-bit, only used as i32 in the database - 'The message was not valid for store and forward' occurs if the message has expired, this PR uses a more descriptive error message for this specific case. - Expired messages should be discarded immediately - Early close "errors" on the rpc server simply indicate that the client went away, which is expected and not something that the server controls, and so is logged at debug level How Has This Been Tested? --- Manually, * v0.38.6 * fix(core): only resize db if migration is required (#4792) Description --- Adds conditional to only increase database size if migration is required Motivation and Context --- A new database (cucumber, functional tests) has no inputs and so migration is not required. Ref #4791 How Has This Been Tested? --- * fix(miner): clippy error (#4793) Description --- Removes unused function in miner Motivation and Context --- Clippy How Has This Been Tested? --- No clippy error * test: remove cucumber tests, simplify others (#4794) Description --- * remove auto update tests from cucumber * rename some tests to be prefixed with `test_` * simplified two cucumber tests by removing steps Motivation and Context --- The auto update tests have an external dependency, which makes it hard to test reliably. They were marked as broken, so I rather removed them. There were two steps in the `list_height` and `list_headers` tests that created base nodes. Upon inspection of the logs, these base nodes never synced to the height of 5 and were not checked in the test, so were pretty useless and just slowed the test down How Has This Been Tested? --- npm test * v0.38.7 * feat: add deepsource config * fix(core): periodically commit large transaction in prune_to_height (#4805) * fix(comms/rpc): measures client-side latency to first message received (#4817) * fix(core): increase sync timeouts (#4800) Co-authored-by: Cayle Sharrock <CjS77@users.noreply.github.com> * feat: add multisig script that returns aggregate of signed public keys (#4742) Description --- Added an `m-of-n` multisig TariScript that returns the aggregate public key of the signatories if successful and fails otherwise. This is useful if the aggregate public key of the signatories is also the script public key, where signatories would work together to create an aggregate script signature using their individual script private keys. Motivation and Context --- To enhance the practicality of the `m-of-n` multisig TariScript. How Has This Been Tested? --- Unit tests Co-Authored-By: SW van Heerden swvheerden@gmail.com * feat(comms): adds periodic socket-level liveness checks (#4819) Description --- - adds socket-level liveness checks - adds configuration to enable liveness checks (currently enabled by default in base node, disabled in wallet) - update status line to display liveness status Motivation and Context --- Allows us to gain visibility on the base latency of the transport without including overhead of the noise socket and yamux How Has This Been Tested? --- Manually * fix(core): dont request full non-tip block if block is empty (#4802) Description --- - checks for edge-case which prevents an unnecessary full candidate block request when block is empty. Motivation and Context --- A full block request for empty block is not necessary as we already have all the information required to construct the candidate block. This check was missing from the branch where the candidate block is not the next tip block. How Has This Been Tested? --- Co-authored-by: Martin Stefcek <35243812+Cifko@users.noreply.github.com> Co-authored-by: Hansie Odendaal <39146854+hansieodendaal@users.noreply.github.com> Co-authored-by: SW van Heerden <swvheerden@gmail.com> Co-authored-by: stringhandler <mikethetike@tari.com> Co-authored-by: CjS77 <CjS77@users.noreply.github.com>
tari-project · Oct 19, 2022 · ce35b65 · ce35b65
1 parent 2dbceaa
commit ce35b65
Show file tree

Hide file tree

Showing 115 changed files with 1,881 additions and 1,317 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -31,6 +31,9 @@ commands:
       - run:
           name: Build miner
           command: cargo build --release --bin tari_miner
+      - run:
+          name: Build wallet FFI
+          command: cargo build --release --package tari_wallet_ffi
       - run:
           name: Run cucumber scenarios
           no_output_timeout: 20m

diff --git a/.deepsource.toml b/.deepsource.toml
@@ -0,0 +1,10 @@
+version = 1
+
+
+[[analyzers]]
+name = "rust"
+enabled = true
+
+  [analyzers.meta]
+  msrv = "stable"
+
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/applications/tari_app_grpc/Cargo.toml b/applications/tari_app_grpc/Cargo.toml
@@ -4,7 +4,7 @@ authors = ["The Tari Development Community"]
 description = "This crate is to provide a single source for all cross application grpc files and conversions to and from tari::core"
 repository = "https://github.com/tari-project/tari"
 license = "BSD-3-Clause"
-version = "0.38.5"
+version = "0.38.7"
 edition = "2018"
 
 [dependencies]

diff --git a/applications/tari_app_grpc/src/conversions/transaction_input.rs b/applications/tari_app_grpc/src/conversions/transaction_input.rs
@@ -119,8 +119,8 @@ impl TryFrom<TransactionInput> for grpc::TransactionInput {
                 script: input
                     .script()
                     .map_err(|_| "Non-compact Transaction input should contain script".to_string())?
-                    .as_bytes(),
-                input_data: input.input_data.as_bytes(),
+                    .to_bytes(),
+                input_data: input.input_data.to_bytes(),
                 script_signature,
                 sender_offset_public_key: input
                     .sender_offset_public_key()

diff --git a/applications/tari_app_grpc/src/conversions/transaction_output.rs b/applications/tari_app_grpc/src/conversions/transaction_output.rs
@@ -85,7 +85,7 @@ impl From<TransactionOutput> for grpc::TransactionOutput {
             features: Some(output.features.into()),
             commitment: Vec::from(output.commitment.as_bytes()),
             range_proof: Vec::from(output.proof.as_bytes()),
-            script: output.script.as_bytes(),
+            script: output.script.to_bytes(),
             sender_offset_public_key: output.sender_offset_public_key.as_bytes().to_vec(),
             metadata_signature: Some(grpc::ComSignature {
                 public_nonce_commitment: Vec::from(output.metadata_signature.public_nonce().as_bytes()),

diff --git a/applications/tari_app_grpc/src/conversions/unblinded_output.rs b/applications/tari_app_grpc/src/conversions/unblinded_output.rs
@@ -41,8 +41,8 @@ impl From<UnblindedOutput> for grpc::UnblindedOutput {
             value: u64::from(output.value),
             spending_key: output.spending_key.as_bytes().to_vec(),
             features: Some(output.features.into()),
-            script: output.script.as_bytes(),
-            input_data: output.input_data.as_bytes(),
+            script: output.script.to_bytes(),
+            input_data: output.input_data.to_bytes(),
             script_private_key: output.script_private_key.as_bytes().to_vec(),
             sender_offset_public_key: output.sender_offset_public_key.as_bytes().to_vec(),
             metadata_signature: Some(grpc::ComSignature {

diff --git a/applications/tari_app_utilities/Cargo.toml b/applications/tari_app_utilities/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "tari_app_utilities"
-version = "0.38.5"
+version = "0.38.7"
 authors = ["The Tari Development Community"]
 edition = "2018"
 license = "BSD-3-Clause"

diff --git a/applications/tari_base_node/Cargo.toml b/applications/tari_base_node/Cargo.toml
@@ -4,7 +4,7 @@ authors = ["The Tari Development Community"]
 description = "The tari full base node implementation"
 repository = "https://github.com/tari-project/tari"
 license = "BSD-3-Clause"
-version = "0.38.5"
+version = "0.38.7"
 edition = "2018"
 
 [dependencies]

diff --git a/applications/tari_base_node/src/bootstrap.rs b/applications/tari_base_node/src/bootstrap.rs
@@ -20,7 +20,7 @@
 //  WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
 //  USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-use std::{cmp, str::FromStr, sync::Arc};
+use std::{cmp, str::FromStr, sync::Arc, time::Duration};
 
 use log::*;
 use tari_app_utilities::{consts, identity_management, identity_management::load_from_json};
@@ -106,6 +106,12 @@ where B: BlockchainBackend + 'static
             .map_err(|e| ExitError::new(ExitCode::ConfigError, e))?;
         p2p_config.transport.tor.identity = tor_identity;
 
+        // TODO: This should probably be disabled in future and have it optionally set/unset in the config - this check
+        //       does allow MITM/ISP/tor router to connect this node's IP to a destination IP/onion address.
+        //       Specifically, "pingpong" text is periodically sent on an unencrypted socket allowing anyone observing
+        //       the traffic to recognise the sending IP address as almost certainly a tari node.
+        p2p_config.listener_liveness_check_interval = Some(Duration::from_secs(15));
+
         let mut handles = StackBuilder::new(self.interrupt_signal)
             .add_initializer(P2pInitializer::new(
                 p2p_config.clone(),

diff --git a/applications/tari_base_node/src/commands/command/add_peer.rs b/applications/tari_base_node/src/commands/command/add_peer.rs
@@ -44,7 +44,8 @@ pub struct ArgsAddPeer {
 impl HandleCommand<ArgsAddPeer> for CommandContext {
     async fn handle_command(&mut self, args: ArgsAddPeer) -> Result<(), Error> {
         let public_key = args.public_key.into();
-        if self.peer_manager.exists(&public_key).await {
+        let peer_manager = self.comms.peer_manager();
+        if peer_manager.exists(&public_key).await {
             return Err(anyhow!("Peer with public key '{}' already exists", public_key));
         }
         let node_id = NodeId::from_public_key(&public_key);
@@ -57,7 +58,7 @@ impl HandleCommand<ArgsAddPeer> for CommandContext {
             vec![],
             String::new(),
         );
-        self.peer_manager.add_peer(peer).await?;
+        peer_manager.add_peer(peer).await?;
         println!("Peer with node id '{}'was added to the base node.", node_id);
         Ok(())
     }

diff --git a/applications/tari_base_node/src/commands/command/ban_peer.rs b/applications/tari_base_node/src/commands/command/ban_peer.rs
@@ -80,13 +80,14 @@ impl CommandContext {
         if self.base_node_identity.node_id() == &node_id {
             Err(ArgsError::BanSelf.into())
         } else if must_ban {
-            self.connectivity
+            self.comms
+                .connectivity()
                 .ban_peer_until(node_id.clone(), duration, "UI manual ban".to_string())
                 .await?;
             println!("Peer was banned in base node.");
             Ok(())
         } else {
-            self.peer_manager.unban_peer(&node_id).await?;
+            self.comms.peer_manager().unban_peer(&node_id).await?;
             println!("Peer ban was removed from base node.");
             Ok(())
         }

diff --git a/applications/tari_base_node/src/commands/command/dial_peer.rs b/applications/tari_base_node/src/commands/command/dial_peer.rs
@@ -48,7 +48,7 @@ impl HandleCommand<Args> for CommandContext {
 impl CommandContext {
     /// Function to process the dial-peer command
     pub async fn dial_peer(&self, dest_node_id: NodeId) -> Result<(), Error> {
-        let connectivity = self.connectivity.clone();
+        let connectivity = self.comms.connectivity();
         task::spawn(async move {
             let start = Instant::now();
             println!("☎️  Dialing peer...");

diff --git a/applications/tari_base_node/src/commands/command/get_peer.rs b/applications/tari_base_node/src/commands/command/get_peer.rs
@@ -63,16 +63,16 @@ enum ArgsError {
 
 impl CommandContext {
     pub async fn get_peer(&self, partial: Vec<u8>, original_str: String) -> Result<(), Error> {
-        let peers = self.peer_manager.find_all_starts_with(&partial).await?;
+        let peer_manager = self.comms.peer_manager();
+        let peers = peer_manager.find_all_starts_with(&partial).await?;
         let peer = {
             if let Some(peer) = peers.into_iter().next() {
                 peer
             } else {
                 let pk = parse_emoji_id_or_public_key(&original_str).ok_or_else(|| ArgsError::NoPeerMatching {
                     original_str: original_str.clone(),
                 })?;
-                let peer = self
-                    .peer_manager
+                let peer = peer_manager
                     .find_by_public_key(&pk)
                     .await?
                     .ok_or(ArgsError::NoPeerMatching { original_str })?;

diff --git a/applications/tari_base_node/src/commands/command/list_connections.rs b/applications/tari_base_node/src/commands/command/list_connections.rs
@@ -53,9 +53,9 @@ impl CommandContext {
             "User Agent",
             "Info",
         ]);
+        let peer_manager = self.comms.peer_manager();
         for conn in conns {
-            let peer = self
-                .peer_manager
+            let peer = peer_manager
                 .find_by_node_id(conn.peer_node_id())
                 .await
                 .expect("Unexpected peer database error")
@@ -105,7 +105,7 @@ impl CommandContext {
 impl CommandContext {
     /// Function to process the list-connections command
     pub async fn list_connections(&mut self) -> Result<(), Error> {
-        let conns = self.connectivity.get_active_connections().await?;
+        let conns = self.comms.connectivity().get_active_connections().await?;
         let (mut nodes, mut clients) = conns
             .into_iter()
             .partition::<Vec<_>, _>(|a| a.peer_features().is_node());