chore: improve cucumber tags and run time speed (#3439)

Description
---

This PR improves the run time of certain tests to remove `waits` and make sure they have async functions to retest waitable events.
It adds a `@base-node` tag to all base node tests.
It adds a `@wallet` tag to all wallet tests.
It adds `@`  tags to all feature files.
Its adds and remove `@critical` tags to all required tests, adding where critical functionality is not tested and removing where it is double tested and or not critical to core functionality.

Motivation and Context
---
Made sure we can run all tests as short as possible. Tests are grouped with the following tags:
`@base-node` which can be used to run tests if verification of code base node functionality is desired
`@wallet` which can be used to run tests if verification of code wallet functionality is desired
`@critical` if a short test of all core functionality is desired.
`@long-running` if really detailed long-running tests are desired. Can take hours. 

Tests without a `@critical` and `@long-running` are tests that are required for passing CI, but will take longer to run.
Running test with `--tags @base-node AND @critical` or `--tags @wallet AND @critical` completes it run time below 10 minutes.

integration_tests/README.md

@@ -45,6 +45,10 @@
   npm test -- --tags "@critical"
   ./run-tests.sh --tags "@critical"
 
+    # Runs all critical tests base node tests
+  npm test -- --tags "@critical AND @base-node"
+  ./run-tests.sh --tags "@critical AND @base-node"
+
    # Runs all critical tests, but not @long-running and not @tbroken
   npm test -- --tags "@critical and not @long-running and not @broken"
   ```

integration_tests/features/BaseNodeAutoUpdate.feature

@@ -9,5 +9,4 @@ Feature: AutoUpdate
     @broken
     Scenario: Auto update ignores update with invalid signature on base node
         Given I have a node NODE_A with auto update configured with a bad signature
-        And I wait 10 seconds
         Then NODE_A does not have a new software update

integration_tests/features/BaseNodeConnectivity.feature

@@ -1,12 +1,14 @@
 @base-node-connectivity
 Feature: Base Node Connectivity
 
+    @critical @base-node
     Scenario: Basic connectivity between 2 nodes
         Given I have a seed node SEED_A
         And I have a base node NODE_A connected to all seed nodes
         When I wait for NODE_A to connect to SEED_A
         Then SEED_A is connected to NODE_A
 
+    @critical @base-node @wallet
     Scenario: Basic connectivity between nodes and wallet
         Given I have a seed node SEED_A
         And I have wallet WALLET_A connected to all seed nodes

integration_tests/features/BlockExplorerGRPC.feature

@@ -1,7 +1,6 @@
 @block-explorer
 Feature: Block Explorer GRPC
 
-  @critical
   Scenario: As a user I want to get the network difficulties
     Given I have a seed node NODE
     And I have wallet WALLET connected to all seed nodes

integration_tests/features/Mempool.feature

@@ -1,7 +1,7 @@
-@mempool
+@mempool @base-node
 Feature: Mempool
 
-  @flaky
+  @critical @flaky
   Scenario: Transactions are propagated through a network
     #
     # The randomness of the TX1 propagation can result in this test not passing.
@@ -18,7 +18,7 @@ Feature: Mempool
     Then TX1 is in the MEMPOOL of all nodes, where 1% can fail
 
 
-  @flaky
+  @critical @flaky
   Scenario: Transactions are synced
     Given I have 2 seed nodes
     And I have a base node SENDER connected to all seed nodes
@@ -35,7 +35,7 @@ Feature: Mempool
     Then SENDER has TX1 in MINED state
     Then TX1 is in the MINED of all nodes
 
-  @flaky @broken
+  @broken
   Scenario: Clear out mempool
     Given I have 1 seed nodes
     And I have a base node SENDER connected to all seed nodes
@@ -62,7 +62,7 @@ Feature: Mempool
     Then SENDER has TX3 in MINED state
 
 
-  @critical @long-running
+  @critical
   Scenario: Double spend
     Given I have 1 seed nodes
     And I have a base node SENDER connected to all seed nodes
@@ -75,70 +75,50 @@ Feature: Mempool
     Then SENDER has TX1 in MEMPOOL state
     Then SENDER has TX2 in MEMPOOL state
     When I mine 1 blocks on SENDER
-    Then SENDER has TX1 in NOT_STORED state
+    # a transaction that was removed from the pool will be reported as unknown as long as it is stored in the reorg pool for 5 mins
+    Then SENDER has TX1 in UNKNOWN state
     Then SENDER has TX2 in MINED state
+
+ @long-running
+  Scenario: Double spend eventually ends up as not stored
+    Given I have 1 seed nodes
+    And I have a base node SENDER connected to all seed nodes
+    When I mine a block on SENDER with coinbase CB1
+    When I mine 4 blocks on SENDER
+    When I create a custom fee transaction TX1 spending CB1 to UTX1 with fee 80
+    When I create a custom fee transaction TX2 spending CB1 to UTX2 with fee 100
+    When I submit transaction TX1 to SENDER
+    When I submit transaction TX2 to SENDER
+    Then SENDER has TX1 in MEMPOOL state
+    Then SENDER has TX2 in MEMPOOL state
     When I mine 1 blocks on SENDER
+    # a transaction that was removed from the pool will be reported as unknown as long as it is stored in the reorg pool for 5 mins
     Then SENDER has TX1 in NOT_STORED state
     Then SENDER has TX2 in MINED state
 
   @critical
   Scenario: Mempool clearing out invalid transactions after a reorg
-    #
-    # Chain 1:
-    #   Collects 7 coinbases into one wallet, send 7 transactions
-    #   Stronger chain
-    #
     Given I do not expect all automated transactions to succeed
     Given I have a seed node SEED_A
-    And I have a base node NODE_A1 connected to seed SEED_A
-    And I have wallet WALLET_A1 connected to seed node SEED_A
-    And I have wallet WALLET_A2 connected to seed node SEED_A
-    And I have mining node MINER_A1 connected to base node SEED_A and wallet WALLET_A1
-    When I wait 5 seconds
-    When mining node MINER_A1 mines 7 blocks with min difficulty 200 and max difficulty 100000
-    Then node SEED_A is at height 7
-    Then node NODE_A1 is at height 7
-    When I mine 3 blocks on SEED_A
-    Then wallet WALLET_A1 detects at least 7 coinbase transactions as Mined_Confirmed
-    Then node SEED_A is at height 10
-    Then node NODE_A1 is at height 10
-    And I multi-send 7 transactions of 1000000 uT from wallet WALLET_A1 to wallet WALLET_A2 at fee 100
-    Then wallet WALLET_A1 detects all transactions are at least Broadcast
-    When I wait 1 seconds
-    #
-    # Chain 2:
-    #   Collects 7 coinbases into one wallet, send 7 transactions
-    #   Weaker chain
-    #
-    And I have a seed node SEED_B
-    And I have a base node NODE_B1 connected to seed SEED_B
-    And I have wallet WALLET_B1 connected to seed node SEED_B
-    And I have wallet WALLET_B2 connected to seed node SEED_B
-    And I have mining node MINER_B1 connected to base node SEED_B and wallet WALLET_B1
-    When I wait 5 seconds
-    When mining node MINER_B1 mines 7 blocks with min difficulty 1 and max difficulty 100
-    Then node SEED_B is at height 7
-    Then node NODE_B1 is at height 7
-    When I mine 5 blocks on SEED_B
-    Then wallet WALLET_B1 detects at least 7 coinbase transactions as Mined_Confirmed
-    Then node SEED_B is at height 12
-    Then node NODE_B1 is at height 12
-    And I multi-send 7 transactions of 1000000 uT from wallet WALLET_B1 to wallet WALLET_B2 at fee 100
-    Then wallet WALLET_B1 detects all transactions are at least Broadcast
-    When I wait 1 seconds
-    #
-    # Connect Chain 1 and 2 in stages
-    #    New node connects to weaker chain, receives all broadcast (not mined) transactions into mempool
-    #    New node connects to stronger chain, then reorgs its complete chain
-    #    New node mines blocks; no invalid inputs from the weaker chain should be used in the block template
-    #
-    And I have a base node NODE_C connected to seed SEED_B
-    Then node NODE_C is at height 12
-    # Wait for the reorg to filter through
-    And I connect node SEED_A to node NODE_C and wait 30 seconds
-    Then all nodes are at height 10
-    When I mine 6 blocks on NODE_C
-    Then all nodes are at height 16
+    And I have a base node NODE_A connected to seed SEED_A
+    When I mine a block on NODE_A with coinbase CB_A
+    When I mine 3 blocks on NODE_A
+    Given I have a seed node SEED_B
+    And I have a base node NODE_B connected to seed SEED_B
+    When I mine a block on NODE_B with coinbase CB_B
+    When I mine 10 blocks on NODE_B
+    When I create a custom fee transaction TXA spending CB_A to UTX1 with fee 80
+    When I create a custom fee transaction TXB spending CB_B to UTX1 with fee 80
+    When I submit transaction TXA to NODE_A
+    When I submit transaction TXB to NODE_B
+    Then NODE_A has TXA in MEMPOOL state
+    Then NODE_B has TXB in MEMPOOL state
+    When I mine 1 blocks on NODE_A
+    When I mine 1 blocks on NODE_B
+    And I connect node NODE_A to node NODE_B and wait 1 seconds
+    Then all nodes are at height 12
+    Then NODE_A has TXA in NOT_STORED state
+    Then NODE_A has TXB in MINED state
 
   @critical
   Scenario: Zero-conf transactions
@@ -173,7 +153,6 @@ Feature: Mempool
     Then SENDER has TX12 in MINED state
     Then SENDER has TX13 in MINED state
 
-  @critical
   Scenario: Mempool unconfirmed transactions
     Given I have 1 seed nodes
     And I have a base node BN1 connected to all seed nodes
@@ -191,7 +170,6 @@ Feature: Mempool
     When I submit transaction TX5 to BN1
     Then I wait until base node BN1 has 5 unconfirmed transactions in its mempool
 
-  @critical
   Scenario: Mempool unconfirmed transaction to mined transaction
     Given I have 1 seed nodes
     And I have a base node BN1 connected to all seed nodes

integration_tests/features/MergeMining.feature

@@ -1,7 +1,6 @@
-@merge-mining
+@merge-mining @base-node
   Feature: Merge Mining
 
-    @critical
     Scenario: Merge Mining Functionality Test Without Submitting To Origin
       Given I have a seed node NODE
       And I have wallet WALLET connected to all seed nodes
@@ -13,7 +12,6 @@
       When I submit a block through proxy PROXY
       Then Proxy response block submission is valid without submitting to origin
 
-    @critical
     Scenario: Merge Mining Functionality Test With Submitting To Origin
       Given I have a seed node NODE
       And I have wallet WALLET connected to all seed nodes

integration_tests/features/Propagation.feature

@@ -1,4 +1,4 @@
-@propagation
+@propagation @base-node
 Feature: Block Propagation
 
   Scenario Outline: Blocks are propagated through the network
@@ -7,19 +7,16 @@ Feature: Block Propagation
     And I have a SHA3 miner MINER connected to all seed nodes
     And mining node MINER mines <NumBlocks> blocks
     Then all nodes are at height <NumBlocks>
-    @critical
+
     Examples:
       | NumSeeds | NumNonSeeds | NumBlocks |
       | 1        | 1           | 5         |
 
+    @long-running
     Examples:
       | NumSeeds | NumNonSeeds | NumBlocks |
       | 1        | 10          | 5         |
       | 4        | 10          | 5         |
-
-    @long-running
-    Examples:
-      | NumSeeds | NumNonSeeds | NumBlocks |
       | 8        | 40          | 10        |
 
   @critical
@@ -31,6 +28,7 @@ Feature: Block Propagation
     Then node MINER is at height 5
     Then all nodes are at height 5
 
+  @critical
   Scenario: Duplicate block is rejected
     Given I have 1 seed nodes
     And I have a base node MINER connected to all seed nodes
@@ -58,41 +56,39 @@ Feature: Block Propagation
     Then I receive an error containing 'Orphan block'
     And all nodes are at height 1
 
-  @non-sync-propagation @long-running
+  @non-sync-propagation
   Scenario: Nodes should never switch to block sync but stay synced via propagation
     Given I have 1 seed nodes
     Given I have a SHA3 miner MINER connected to all seed nodes
     And I have a lagging delayed node LAG1 connected to node MINER with blocks_behind_before_considered_lagging 10000
     Given I have a lagging delayed node LAG2 connected to node MINER with blocks_behind_before_considered_lagging 10000
     # Wait for node to so start and get into listening mode
-    When I wait 100 seconds
+    Then node LAG1 has reached initial sync
+    Then node LAG2 has reached initial sync
     When mining node MINER mines 5 blocks
     Then all nodes are at height 5
     Given mining node MINER mines 15 blocks
     Then all nodes are at height 20
 
-  @long-running
   Scenario: Node should lag for while before syncing
     Given I have 1 seed nodes
     Given I have a SHA3 miner MINER connected to all seed nodes
     And I have a lagging delayed node LAG1 connected to node MINER with blocks_behind_before_considered_lagging 6
     Given mining node MINER mines 1 blocks
-    When I wait 100 seconds
+    Then all nodes are at height 1
     When I stop node LAG1
-    When I wait 10 seconds
     And mining node MINER mines 5 blocks
-    When I wait 100 seconds
+    Then node MINER is at height 6
     When I start base node LAG1
     # Wait for node to so start and get into listening mode
-    When I wait 100 seconds
-    Then node MINER is at height 6
+    Then node LAG1 has reached initial sync
     #node was shutdown, so it never received the propagation messages
     Then node LAG1 is at height 1
     Given mining node MINER mines 1 blocks
     Then node MINER is at height 7
-    When I wait 20 seconds
     Then all nodes are at height 7
 
+  @critical
   Scenario: Pruned node should prune outputs
     Given I have 1 seed nodes
     And I have a base node SENDER connected to all seed nodes

integration_tests/features/Recovery.feature

@@ -13,10 +13,10 @@ Feature: Recovery
         Examples:
             | NumBlocks |
             | 10        |
-            | 25        |
 
         # Takes 1min+ on Circle CI
         @long-running
         Examples:
             | NumBlocks |
-            | 50        |
+            | 25        |
+            | 50        |