Support for historical queries after recovery from snapshot

[jumaffre]

When resuming from a snapshot or after recovery, a node doesn't have the history of the ledger secrets as ledger secrets are stored at the same key in the ccf.shares table. Assuming that the historical ledger is later moved to that node (see #1527), if this node has to serve historical queries, it won't be able to decrypt or verify the integrity of these historical entries.

To be able to support historical queries, the node should process the re-sharing entries in the ledger backwards, from the last available re-sharing entry (available in the snapshot). Caveat here: on recovery, the re-sharing entry is written at a version later than the one at which the associated ledger secret is applicable from so this mechanism should account for that.

Ledger secrets can be decrypted one-by-one using the previously restored ledger secret. There is no need to restore ledger secret prior to the latest re-sharing entry prior to the start index of the historical query and no need to keep in memory ledger secrets more recent than the first one after the end index of the historical query (included).

[achamayou]

A slightly tangent point that came up in discussion but may be helpful in implementation is that reading through historical secrets going back can be quite quick (each one contains the version of its predecessor) and does not require ledger integrity validation because ledger secrets are encrypted and integrity protected with AES GCM.

A possible optimisation, should this turn out to be a significant cost, may be to include the last X ledger secrets in snapshots (similarly to what's being done for the term history).

[jumaffre]

Removing checkpoint milestone and re-phrasing issue slightly as this is not snapshot-specific but also applies to recoveries.

[jumaffre]

FYI, the minimal test suite to reproduce the issue is:

suite_rekey_recovery = [
    e2e_logging.test_historical_query,
    rekey.test,
    rekey.test,
    recovery.test, # Assuming this starts from snapshot
]

The recovery test will fail verifying historical queries emitted before the first rekey.