Feature/bos dpos pbft (EOSIO#33)

* dpos pbft

* bug fix: ignore empty certificate on view change validation

* bug fix: force switch to new schedule when there is new schedule in pending; remove unnecessary logs.

* open view change related logs.

* separate checkpoint db from pbft state

* update certificate validations.

* add logs for large scale debug.

* bug fix: generate prepare (commit) certificates with valid prepare (commits) msgs

* turn on view change related logs for testing.

* extend waiting time on BP schedule change.

* separate checkpoint db from state db; remove dpos confirm related logic.

* move checkpoints dir into blocks dir; open more logs for debug.

* no longer persist target view (default value will be current view +1); open more logs for debug.

* bug fix: avoid fetching empty block state.

* open prepare-adding related logs

* bug fix: inappropriate comparision may lead to crash during sorting.

* force activate schedule once gpo is set.

* remove unnecessary logs.

* force all bps, who belong to any schedule in fork db blocks, sending pbft related msgs. bug fix: cannot load empty pbft index upon restart.

* bp candidates are involved in pbft consensus.

* bug fix: proposed new view should be strictly equal to view changed cert.

* open new_view related logs.

* force new view check, even if I am the primary.

* bug fix: calculate primary key

* extend view change timeout to 8 sec.

* skip view change validation for test.

* skip new view validation for test.

* add prepare validation

* update prepared(committed) cert validation, add uuid info for view change msg.

* open view change validation related log.

* increase prepare (commit) timeout to 2s.

* remove unnecessary logs; optimise msg validation.

* remove unnecessary validations

* remove uuid from pbft msgs; only publish new msgs into chain plugin.

* reformat end itr validation.

* include chain_id into pbft msgs; extend msg cache ttl to very long.

* include chain_id into pbft msgs; extend msg cache ttl to very long.

* skip all signature validation for test.

* change to fetch from forkdb

* fix crash bug

* optimise certificate validation

* add uuid into pbft msgs

* replace committed cert with stable checkpoint; use stable checkpoints to skip view change state; not send any pbft msgs when syncing and replaying.

* send view change immediately on transition to veiw change state (do not wait for timer)

* increase view change waiting time.

* bug fix: fetch stable checkpoint block id.

* remove sig validation for profiling

* bug fix: accept prepare(commit) msgs from bp candidates; enqueue msg only when connection is active;

* skip cert check on view change msgs

* no pbft msgs if all connections are syncing

* add current view into view change msg.

* skip sig check for profiling; add timestamp & current view in view change msg

* force resend checkpoints; send msgs only to active connections

* optimise debug log.

* add constructor (destructor) log for pbft ctrl;

* add timestamp to all pbft msgs; skip all msgs that are generated before 10s ago; open sig check;

* add connection monitor log

* add more connection monitor info

* add view assertion

* remove debug logs; optimise some implementation.

* prevent bp set lib via checkpoints; force send prepare (commit) msg after transit to new state;

* enable net plugin queue log and restrict out queue size to 10

* dont send pbft untili  handshake finished

* use pbft_queue to send pbft message

* bug fix: treat an isolated node as synced one.

* add log

* revert Dockerfile

* fix network issue by adding pbft queue and timeout

* separate outqueue limit for write queue and pbft queue

* add debug log in handle pbft msg.

* check write queue size for safty

* add log

* update log

* disable 'syncing' & 'current' check; add set_view api in producer api plugin to enable manually view set (after set, node will transit to view change state immediately).

* remove write queue size limit, add log for local_txns

* clear outqueue when close conn

* resolve conflict

* do not bcast pbft msgs if I am not in sync; only bcast (forward) to current connections; force send checkpoints upon committed local; clean view change cache upon new state; bug fix: cal pending stable checkpoint via block info (instead of num)

* add timeout when conn is doing connecting

* clean up do_queue_write

* increase checkpoint interval to 100; increase block production sleep time to 100ms.

* skip switch fork if is pbft prepared, to ensure invariance

* add pbft test

* add pbft_supported into block_state, ensure prepared fork is current chain.

* bug fix: set prepared block as pbft_supported

* bug fix: set my prepare block as supported fork.

* correct conditions for prepare reserve; fix typo

* add prepared and my prepare priority to fork_db, to ensure fork_db head falls on the fork that the majority is prepared or that I used to prepare

* add log

* bug fix: add all prepared cert info pbft db, to ensure consistency

* clear pbft state during transit to new view

* prevent sending and receiving pbft msgs during syncing; reformat logs;

* switch connection stats monitor to debug level

* correct my prepare mark; remove unnecessary methods; change log level to debug in net plugin

Docker/Dockerfile

@@ -25,4 +25,4 @@ COPY --from=builder /bos/Docker/nodeosd.sh /opt/eosio/bin/nodeosd.sh
 ENV EOSIO_ROOT=/opt/eosio
 RUN chmod +x /opt/eosio/bin/nodeosd.sh
 ENV LD_LIBRARY_PATH /usr/local/lib
-ENV PATH /opt/eosio/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ENV PATH /opt/eosio/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

libraries/chain/CMakeLists.txt

@@ -14,6 +14,8 @@ add_library( eosio_chain
              block_header_state.cpp
              block_state.cpp
              fork_database.cpp
+             pbft_database.cpp
+             pbft.cpp
              controller.cpp
              authorization_manager.cpp
              resource_limits.cpp

libraries/chain/block_header_state.cpp

@@ -62,36 +62,38 @@ namespace eosio { namespace chain {
 
     result.active_schedule                       = active_schedule;
     result.pending_schedule                      = pending_schedule;
-    result.dpos_proposed_irreversible_blocknum   = dpos_proposed_irreversible_blocknum;
+//    result.dpos_proposed_irreversible_blocknum   = dpos_proposed_irreversible_blocknum;
     result.bft_irreversible_blocknum             = bft_irreversible_blocknum;
+    result.pbft_stable_checkpoint_blocknum       = pbft_stable_checkpoint_blocknum;
 
     result.producer_to_last_implied_irb[prokey.producer_name] = result.dpos_proposed_irreversible_blocknum;
-    result.dpos_irreversible_blocknum                         = result.calc_dpos_last_irreversible(); 
+//    result.dpos_irreversible_blocknum                         = result.calc_dpos_last_irreversible();
 
     /// grow the confirmed count
     static_assert(std::numeric_limits<uint8_t>::max() >= (config::max_producers * 2 / 3) + 1, "8bit confirmations may not be able to hold all of the needed confirmations");
 
     // This uses the previous block active_schedule because thats the "schedule" that signs and therefore confirms _this_ block
-    auto num_active_producers = active_schedule.producers.size();
-    uint32_t required_confs = (uint32_t)(num_active_producers * 2 / 3) + 1;
-
-    if( confirm_count.size() < config::maximum_tracked_dpos_confirmations ) {
-       result.confirm_count.reserve( confirm_count.size() + 1 );
-       result.confirm_count  = confirm_count;
-       result.confirm_count.resize( confirm_count.size() + 1 );
-       result.confirm_count.back() = (uint8_t)required_confs;
-    } else {
-       result.confirm_count.resize( confirm_count.size() );
-       memcpy( &result.confirm_count[0], &confirm_count[1], confirm_count.size() - 1 );
-       result.confirm_count.back() = (uint8_t)required_confs;
-    }
+//    auto num_active_producers = active_schedule.producers.size();
+//    uint32_t required_confs = (uint32_t)(num_active_producers * 2 / 3) + 1;
+
+//    if( confirm_count.size() < config::maximum_tracked_dpos_confirmations ) {
+//       result.confirm_count.reserve( confirm_count.size() + 1 );
+//       result.confirm_count  = confirm_count;
+//       result.confirm_count.resize( confirm_count.size() + 1 );
+//       result.confirm_count.back() = (uint8_t)required_confs;
+//    } else {
+//       result.confirm_count.resize( confirm_count.size() );
+//       memcpy( &result.confirm_count[0], &confirm_count[1], confirm_count.size() - 1 );
+//       result.confirm_count.back() = (uint8_t)required_confs;
+//    }
 
     return result;
   } /// generate_next
 
    bool block_header_state::maybe_promote_pending() {
-      if( pending_schedule.producers.size() &&
-          dpos_irreversible_blocknum >= pending_schedule_lib_num )
+      if (pending_schedule.producers.size())
+          //TODO: is this actually safe?
+//          bft_irreversible_blocknum >= pending_schedule_lib_num )
       {
          active_schedule = move( pending_schedule );
 
@@ -101,7 +103,7 @@ namespace eosio { namespace chain {
             if( existing != producer_to_last_produced.end() ) {
                new_producer_to_last_produced[pro.producer_name] = existing->second;
             } else {
-               new_producer_to_last_produced[pro.producer_name] = dpos_irreversible_blocknum;
+               new_producer_to_last_produced[pro.producer_name] = bft_irreversible_blocknum;
             }
          }
 
@@ -111,7 +113,7 @@ namespace eosio { namespace chain {
             if( existing != producer_to_last_implied_irb.end() ) {
                new_producer_to_last_implied_irb[pro.producer_name] = existing->second;
             } else {
-               new_producer_to_last_implied_irb[pro.producer_name] = dpos_irreversible_blocknum;
+               new_producer_to_last_implied_irb[pro.producer_name] = bft_irreversible_blocknum;
             }
          }
 
@@ -163,7 +165,7 @@ namespace eosio { namespace chain {
      /// below this point is state changes that cannot be validated with headers alone, but never-the-less,
      /// must result in header state changes
 
-    result.set_confirmed( h.confirmed );
+//    result.set_confirmed( h.confirmed );
 
     auto was_pending_promoted = result.maybe_promote_pending();