Include receipt evidence in snapshot

.daily_canary

@@ -1 +1 @@
-Run the daily CI please!
+Run the daily CI please :)

CHANGELOG.md

@@ -9,8 +9,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Added
 
-- Receipts now include the endorsed certificate of the node, as well as its node id, for convenience.
+- Receipts now include the endorsed certificate of the node, as well as its node id, for convenience (#2991).
 - `get_metrics_v1` API to `BaseEndpointRegistry` for applications that do not make use of builtins and want to version or customise metrics output.
+- Snapshot files now include receipt of evidence transaction. As such, nodes can now join or recover a service from a standalone snapshot file. 2.x nodes can still make use of snapshots created by a 1.x node, as long as the ledger suffix containing the proof of evidence is also specified at start-up (#2998).
 
 ## [2.0.0-dev4]
 

doc/operations/ledger_snapshot.rst

@@ -56,7 +56,7 @@ Snapshots are generated at regular intervals by the current primary node and sto
 
 To guarantee that the identity of the primary node that generated the snapshot can be verified offline, the SHA-256 digest of the snapshot (i.e. evidence) is recorded in the ``public:ccf.internal.snapshot_evidence`` table. The snapshot evidence will be signed by the primary node on the next signature transaction (see :ref:`operations/start_network:Signature Interval`).
 
-Committed snapshot files are named ``snapshot_<seqno>_<evidence_seqno>.commited_<evidence_commit_seqno>``, with ``<seqno>`` the sequence number of the state of the key-value store at which they were generated, ``<evidence_seqno>`` the sequence number at which the snapshot evidence was recorded and ``<evidence_commit_seqno>`` the sequence number at which the snapshot evidence was committed.
+Committed snapshot files are named ``snapshot_<seqno>_<evidence_seqno>.commited``, with ``<seqno>`` the sequence number of the state of the key-value store at which they were generated and ``<evidence_seqno>`` the sequence number at which the snapshot evidence was recorded.
 
 Uncommitted snapshot files, i.e. those whose evidence has not yet been committed, are named ``snapshot_<seqno>_<evidence_seqno>``. These files will be ignored by CCF when joining or recovering a service as no evidence can attest of their validity.
 
@@ -65,11 +65,9 @@ Join/Recover From Snapshot
 
 Once a snapshot has been generated by the primary, operators can copy or mount the snapshot directory to the new node directory before it is started. On start-up, the new node will automatically resume from the latest committed snapshot file in the ``--snapshot-dir`` directory. If no snapshot file is found, all historical transactions will be replicated to that node.
 
-To validate the snapshot a node is added from, the node first replays the transactions in the ledger following the snapshot until the proof that the snapshot was committed by the service to join is found. This process requires operators to copy the ledger suffix to the node's ledger directory. The validation procedure is generally quick and the node will automatically join the service once the snapshot has been validated. On recovery, the snapshot is automatically verified as part of the usual ledger recovery procedure.
+From 2.x releases (specifically, from `-dev5`), committed snapshot files embed the receipt of the evidence transaction. As such, nodes can join or recover a service from a standalone snapshot file. For 1.x releases, it is expected that operators also copy the ledger suffix containing the proof of commit of the evidence transaction to the node's ledger directory.
 
-For example, if a node is added using the ``snapshot_1000_1250.committed_1300`` snapshot file, operators should copy the ledger files containing all the sequence numbers between ``1000`` to ``1300`` to the directories specified by ``--ledger-dir`` (or ``--read-only ledger-dir``). This would involve copying the ledger files following the snapshot sequence number ``1000`` until the evidence commit sequence number ``1300``, e.g. ``ledger_1001-1200.committed`` and ``ledger_1201-1500.committed``, to the joining node's ledger directory.
-
-.. note:: If the snapshot to join/recover from is recent, it is likely that the evidence for that snapshot is included in the latest `uncommitted` ledger file. In this case, the corresponding ledger file(s) should be copied to the node's main ledger directory (as specified by ``--ledger-dir``) before start-up.
+.. note:: Snapshots emitted by 1.x nodes can be used by 2.x nodes to join or a recover a service.
 
 Historical Transactions
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -83,6 +81,6 @@ It is recommended for operators to backup the ledger and snapshot files as soon
 
 A low value for ``--ledger-chunk-bytes`` means that smaller ledger files are generated and can thus be backed up by operators more regularly, at the cost of having to manage a large number of ledger files.
 
-Similarly, a low value for ``--snapshot-tx-interval`` means that snapshots are generated often and that join/recovery time will be short, at the cost of additional workload of the primary node for snapshot generation.
+Similarly, a low value for ``--snapshot-tx-interval`` means that snapshots are generated often and that join/recovery time will be short, at the cost of additional workload on the primary node for snapshot generation.
 
 .. tip:: Uncommitted ledger files (which are likely to contain committed transactions) can also be used on join/recovery, as long as they are copied to the node's ``--ledger-dir`` directory.

include/ccf/historical_queries_interface.h

@@ -7,69 +7,13 @@
 #include "consensus/ledger_enclave_types.h"
 #include "kv/store.h"
 #include "node/history.h"
-#include "tls/base64.h"
+#include "node/tx_receipt.h"
 
 #include <chrono>
 #include <memory>
 
 namespace ccf::historical
 {
-  struct TxReceipt
-  {
-    std::vector<uint8_t> signature = {};
-    HistoryTree::Hash root = {};
-    std::shared_ptr<ccf::HistoryTree::Path> path = {};
-    ccf::NodeId node_id = {};
-    std::optional<crypto::Pem> cert = std::nullopt;
-
-    TxReceipt(
-      const std::vector<uint8_t>& s_,
-      const HistoryTree::Hash& r_,
-      std::shared_ptr<ccf::HistoryTree::Path> p_,
-      const NodeId& n_,
-      const std::optional<crypto::Pem>& c_) :
-      signature(s_),
-      root(r_),
-      path(p_),
-      node_id(n_),
-      cert(c_)
-    {}
-
-    void describe(ccf::Receipt& r, bool include_root = false)
-    {
-      r.signature = tls::b64_from_raw(signature);
-      if (include_root)
-      {
-        r.root = root.to_string();
-      }
-      if (path)
-      {
-        for (const auto& node : *path)
-        {
-          ccf::Receipt::Element n;
-          if (node.direction == ccf::HistoryTree::Path::Direction::PATH_LEFT)
-          {
-            n.left = node.hash.to_string();
-          }
-          else
-          {
-            n.right = node.hash.to_string();
-          }
-          r.proof.emplace_back(std::move(n));
-        }
-        r.leaf = path->leaf().to_string();
-      }
-      else
-      {
-        r.leaf = root.to_string();
-      }
-      r.node_id = node_id;
-      if (cert.has_value())
-        r.cert = cert->str();
-    }
-  };
-
-  using TxReceiptPtr = std::shared_ptr<TxReceipt>;
   using StorePtr = std::shared_ptr<kv::Store>;
 
   struct State

include/ccf/receipt.h

@@ -4,6 +4,7 @@
 #pragma once
 
 #include "ccf/entity_id.h"
+#include "crypto/hash.h"
 #include "ds/json.h"
 
 namespace ccf
@@ -30,4 +31,26 @@ namespace ccf
   DECLARE_JSON_TYPE_WITH_OPTIONAL_FIELDS(Receipt)
   DECLARE_JSON_REQUIRED_FIELDS(Receipt, signature, proof, leaf, node_id)
   DECLARE_JSON_OPTIONAL_FIELDS(Receipt, root, cert)
+
+  static crypto::Sha256Hash compute_root_from_receipt(const Receipt& receipt)
+  {
+    crypto::Sha256Hash current = receipt.leaf;
+    for (auto const& element : receipt.proof)
+    {
+      if (element.left.has_value())
+      {
+        assert(!element.right.has_value());
+        crypto::Sha256Hash left = element.left.value();
+        current = crypto::Sha256Hash(left, current);
+      }
+      else
+      {
+        assert(element.right.has_value());
+        crypto::Sha256Hash right = element.right.value();
+        current = crypto::Sha256Hash(current, right);
+      }
+    }
+
+    return current;
+  }
 }

python/ccf/ledger.py

@@ -20,6 +20,7 @@
 
 from ccf.merkletree import MerkleTree
 from ccf.tx_id import TxID
+import ccf.receipt
 
 GCM_SIZE_TAG = 16
 GCM_SIZE_IV = 12
@@ -32,6 +33,8 @@
 NODES_TABLE_NAME = "public:ccf.gov.nodes.info"
 ENDORSED_NODE_CERTIFICATES_TABLE_NAME = "public:ccf.gov.nodes.endorsed_certificates"
 
+COMMITTED_FILE_SUFFIX = ".committed"
+
 # Key used by CCF to record single-key tables
 WELL_KNOWN_SINGLETON_TABLE_KEY = bytes(bytearray(8))
 
@@ -49,7 +52,7 @@ def to_uint_64(buffer):
 
 
 def is_ledger_chunk_committed(file_name):
-    return file_name.endswith(".committed")
+    return file_name.endswith(COMMITTED_FILE_SUFFIX)
 
 
 def unpack(stream, fmt):
@@ -232,6 +235,24 @@ def _byte_read_safe(file, num_of_bytes):
     return ret
 
 
+def _peek(file, num_bytes, pos=None):
+    save_pos = file.tell()
+    if pos is not None:
+        file.seek(pos)
+    buffer = _byte_read_safe(file, num_bytes)
+    file.seek(save_pos)
+    return buffer
+
+
+def _peek_all(file, pos=None):
+    save_pos = file.tell()
+    if pos is not None:
+        file.seek(pos)
+    buffer = file.read()
+    file.seek(save_pos)
+    return buffer
+
+
 class TxBundleInfo(NamedTuple):
     """Bundle for transaction information required for validation"""
 
@@ -483,6 +504,7 @@ def _read_header(self):
         # read the transaction header
         buffer = _byte_read_safe(self._file, TransactionHeader.get_size())
         self._header = TransactionHeader(buffer)
+        entry_start_pos = self._file.tell()
 
         # read the AES GCM header
         buffer = _byte_read_safe(self._file, GcmHeader.size())
@@ -492,6 +514,8 @@ def _read_header(self):
         buffer = _byte_read_safe(self._file, LEDGER_DOMAIN_SIZE)
         self._public_domain_size = to_uint_64(buffer)
 
+        return entry_start_pos
+
     def get_public_domain(self) -> PublicDomain:
         """
         Retrieve the public (i.e. non-encrypted) domain for that entry.
@@ -551,15 +575,11 @@ def get_raw_tx(self) -> bytes:
         """
         assert self._file is not None
 
-        # remember where the pointer is in the file before we go back for the transaction bytes
-        save_pos = self._file.tell()
-        self._file.seek(self._tx_offset)
-        buffer = _byte_read_safe(
-            self._file, TransactionHeader.get_size() + self._header.size
+        return _peek(
+            self._file,
+            TransactionHeader.get_size() + self._header.size,
+            pos=self._tx_offset,
         )
-        # return to original filepointer and return the transaction bytes
-        self._file.seek(save_pos)
-        return buffer
 
     def _complete_read(self):
         self._file.seek(self._next_offset, 0)
@@ -595,14 +615,29 @@ def __init__(self, filename: str):
         super().__init__(filename)
         self._filename = filename
         self._file_size = os.path.getsize(filename)
-        super()._read_header()
 
-    def commit_seqno(self):
-        try:
-            return int(self._filename.split("committed_")[1])
-        except IndexError:
-            # Snapshot is not yet committed
-            return None
+        entry_start_pos = super()._read_header()
+
+        # Snapshots embed evidence receipt since 2.x
+        if self.is_committed() and not self.is_snapshot_file_1_x():
+            receipt_pos = entry_start_pos + self._header.size
+            receipt_bytes = _peek_all(self._file, pos=receipt_pos)
+
+            receipt = json.loads(receipt_bytes.decode("utf-8"))
+            root = ccf.receipt.root(receipt["leaf"], receipt["proof"])
+            node_cert = load_pem_x509_certificate(
+                receipt["cert"].encode(), default_backend()
+            )
+            ccf.receipt.verify(root, receipt["signature"], node_cert)
+
+    def is_committed(self):
+        # Note: Also valid for 1.x snapshots which end in ".committed_Z"
+        return COMMITTED_FILE_SUFFIX in self._filename
+
+    def is_snapshot_file_1_x(self):
+        if not self.is_committed():
+            raise ValueError(f"Snapshot file {self._filename} is not yet committed")
+        return len(self._filename.split(COMMITTED_FILE_SUFFIX)[1]) != 0
 
 
 class LedgerChunk:
@@ -656,7 +691,7 @@ def _reset_iterators(self):
     def _range_from_filename(cls, filename: str) -> Tuple[int, Optional[int]]:
         elements = (
             os.path.basename(filename)
-            .replace(".committed", "")
+            .replace(COMMITTED_FILE_SUFFIX, "")
             .replace("ledger_", "")
             .split("-")
         )
@@ -674,7 +709,7 @@ def __init__(self, directories: List[str], committed_only: bool = True):
         ledger_files = []
         for directory in directories:
             for path in os.listdir(directory):
-                if committed_only and not path.endswith(".committed"):
+                if committed_only and not path.endswith(COMMITTED_FILE_SUFFIX):
                     continue
                 chunk = os.path.join(directory, path)
                 if os.path.isfile(chunk):

python/ccf/read_ledger.py

@@ -150,7 +150,7 @@ def run(args_, tables_format_rules=None):
         snapshot_file = args.paths[0]
         with ccf.ledger.Snapshot(snapshot_file) as snapshot:
             LOG.info(
-                f"Reading snapshot from {snapshot_file} ({'' if snapshot.commit_seqno() else 'un'}committed)"
+                f"Reading snapshot from {snapshot_file} ({'' if snapshot.is_committed() else 'un'}committed)"
             )
             dump_entry(snapshot, table_filter, tables_format_rules)
     else:

src/apps/js_generic/js_generic_base.cpp

@@ -257,7 +257,7 @@ namespace ccfapp
       ccf::endpoints::EndpointContext& endpoint_ctx,
       kv::Tx& target_tx,
       const std::optional<ccf::TxID>& transaction_id,
-      ccf::historical::TxReceiptPtr receipt)
+      ccf::TxReceiptPtr receipt)
     {
       js::Runtime rt;
       rt.add_ccf_classdefs();

src/consensus/aft/raft.h

@@ -620,6 +620,21 @@ namespace aft
       create_and_remove_node_state();
     }
 
+    void record_signature(
+      kv::Version version,
+      const std::vector<uint8_t>& sig,
+      const ccf::NodeId& node_id,
+      const crypto::Pem& node_cert)
+    {
+      snapshotter->record_signature(version, sig, node_id, node_cert);
+    }
+
+    void record_serialised_tree(
+      kv::Version version, const std::vector<uint8_t>& tree)
+    {
+      snapshotter->record_serialised_tree(version, tree);
+    }
+
     void reconfigure(
       ccf::SeqNo seqno, const kv::NetworkConfiguration& netconfig)
     {

src/consensus/aft/raft_consensus.h

@@ -135,6 +135,21 @@ namespace aft
       aft->add_configuration(seqno, conf, learners);
     }
 
+    void record_signature(
+      kv::Version version,
+      const std::vector<uint8_t>& sig,
+      const ccf::NodeId& node_id,
+      const crypto::Pem& node_cert) override
+    {
+      aft->record_signature(version, sig, node_id, node_cert);
+    }
+
+    void record_serialised_tree(
+      kv::Version version, const std::vector<uint8_t>& tree) override
+    {
+      aft->record_serialised_tree(version, tree);
+    }
+
     bool orc(kv::ReconfigurationId rid, const ccf::NodeId& node_id) override
     {
       return aft->orc(rid, node_id);