Commit log bootstrapper (placed before peers bootstrapper) is not reliable post topology change UNTIL a snapshot has taken place

[richardartoul]

Imagine an M3DB cluster running with the bootstrapper configuration (filesystem,commitlog,peers,uninitialized) and the following topology (RF=1 for simplicity):

Host 1: Shard 1
Host 2: Shard 2
Host 3: Shard 3
Host 4: shard 4

Topology change removes host 4, making the new topology:

Host 1: Shard 1, Shard 4
Host 2: Shard 2
Host 3: Shard 3

For non-active blocks, Host 1 will have incrementally flushed all the data for shard 4. However, for the active block of Shard 4 the data will still be in-memory and not yet flushed to disk. The data will also not be in any snapshot files or commit log files. If the node goes down at this point, the commit log bootstrapper will succeed the bootstrap but data will have been lost.

Potential solutions:

1. Require a successful snapshot post-bootstrap during topology changes before marking shards as "Available" on the new host
2. Incremental flush the active block to the snapshots directory during peer bootstrapping

[richardartoul]

Discussed offline with @prateek and settled on 2 probably being the best solution

[robskillington]

Agreed, second approach sounds the most pragmatic.

[richardartoul]

Resolved by #903