Merge pull request #110 from EventStore/DEV-271/make-checkpoint-path-…

…configurable

DEV-271 - Improve documentation for configuring Checkpoints

content/docs/Features/Checkpoints/_index.md

@@ -3,7 +3,43 @@ title: "Checkpoints"
 linkTitle: "Checkpoints"
 weight: 5
 description: >
-  Supported checkpoint stores
+  Replication checkpoints
 ---
 
-The location of the last processed event in the source is known as _checkpoint_. Replicator supports storing checkpoints in different stores, but nly one store can be used at a time. If you want to run the replication again, from the same source, using the same Replicator instance, you need to delete the checkpoint from the store.
+Replicator stores a checkpoint that represents the current position of replicated events.
+
+This allows Replicator to resume processing after a restart, instead of starting from the beginning.
+
+This is controlled via the `checkpointAfter` setting.
+
+### Configuring `checkpointAfter`
+
+The `checkpointAfter` setting can be used to control the threshold number of events that must be replicated before a checkpoint is stored, like so:
+
+```yaml
+replicator:
+  checkpoint:
+    checkpointAfter: 1000
+```
+
+By default, `checkpointAfter` is configured to store a checkpoint after every `1000` events replicated.
+
+A **lower** `checkpointAfter` would mean the replication process has stronger data consistency/duplication guarantees, at the cost of performance.
+
+For example, configuring `checkpointAfter: 1` would result in a checkpoint being stored after every replicated event, which would achieve **exactly-once** processing.
+
+This means in the event of a crash/restart, Replicator is guaranteed to not duplicate any events in the sink database, but comes at the cost of greatly reduced write performance to the sink database.
+
+A **higher** `checkpointAfter` would mean writes to the sink database are more performant, at the cost of data consistency/duplication guarantees.
+
+Configuring a higher `checkpointAfter` improves write performance by ensuring Replicator is not spending so much time saving checkpoints, but introduces a risk of events being duplicated during replication to the sink database in the event of a crash and restart, where a crash ocurred inside of the checkpoint window.
+
+Configure the `checkpointAfter` to align with your data consistency and performance requirements.
+
+### Checkpoint stores
+
+Replicator supports storing checkpoints in different stores. Only one store can be configured per Replicator instance. 
+
+If you want to run the replication again, from the same source, using the same Replicator instance and settings, you need to delete the checkpoint from the store.
+
+See the currently supported checkpoint stores below:

content/docs/Features/Checkpoints/file-store.md

@@ -6,16 +6,16 @@ description: >
     File system checkpoint store
 ---
 
-By default, Replicator stores the checkpoint in a local file. For that purpose, for example, the Helm chart has a persistent volume claim template, which is mapped to the `/data` path. The file name is `checkpoint` by default.
+By default, Replicator stores checkpoints in a local file.
 
-Here is the default configuration used by the Helm chart:
+The default configuration is:
 
 ```yaml
 replicator:
   checkpoint:
     type: file
-    path: "/data/checkpoint"
+    path: ./checkpoint
     checkpointAfter: 1000
 ```
 
-The `checkpointAfter` option defines the frequency of the stored checkpoint update. Smaller number increases the level of idempotency for the sink. For the highest level of idempotency this option should be set to `1`, so the checkpoint will be stored after writing each event. However, it also slows down the replication process, as storing the checkpoint to the file takes time.
+The `path` can be configured to store the checkpoint file at a different location, which can be useful for deployments to Kubernetes that may use a custom PVC configuration, for example.

content/docs/Features/Checkpoints/mongo-store.md

@@ -6,20 +6,36 @@ description: >
     MongoDB checkpoint store
 ---
 
-Although in many cases the file-based checkpoint works fine, storing the checkpoint outside the Replicator deployment is more secure. For that purpose you can use the MongoDB checkpoint store. This store writes the checkpoint as a MongoDB document to the specified database. It will unconditionally use the `checkpoint` collection, and will store the document using the unique id that must be provided in the settings. The uniqueness of the setting (`instanceId`) value is requires if you run multiple Replicator instances, and store checkpoints in the same Mongo database.
+Although in many cases the file-based checkpoint works fine, storing the checkpoint outside the Replicator deployment is more secure. For that purpose you can use the MongoDB checkpoint store. This store writes the checkpoint as a MongoDB document to the specified database. It will unconditionally use the `checkpoint` collection.
 
-Here are the settings for the MongoDB checkpoint store:
+Here are the minimum required settings for storing checkpoints in MongoDB:
 
 ```yaml
 replicator:
   checkpoint:
-    type: "mongo"
+    type: mongo
     path: "mongodb://mongoadmin:secret@localhost:27017"
-    database: "replicator"
-    instanceId: "test"
     checkpointAfter: 1000
 ```
 
-Here, the `path` setting must contain a pre-authenticated connection string.
+The `path` setting must contain a pre-authenticated connection string.
 
-The `checkpointAfter` option defines the frequency of the stored checkpoint update. Smaller number increases the level of idempotency for the sink. For the highest level of idempotency this option should be set to `1`, so the checkpoint will be stored after writing each event. However, it also slows down the replication process, as storing the checkpoint to the database takes time, and increases the load on the MongoDB instance.
+By default, Replicator will use the `replicator` database, but can be configured to use another database, like so:
+
+```yaml
+replicator:
+  checkpoint:
+    database: someOtherDatabase
+```
+
+If you run miltiple Replicator instances that store checkpoints in the same Mongo database, you can configure the `instanceId` setting to isolate checkpoints stored by other Replicator instances.
+
+By default, Replicator will use `default` as the instance identifier under the assumption it is the **only** instance storing checkpoints in the database.
+
+You may configure `instanceId` like so:
+
+```yaml
+replicator:
+  checkpoint:
+    instanceId: someUniqueIdentifier
+```