Add instructions on how to copy chunks between data nodes (github#169)

* Add instructions on how to copy chunks between data nodes

This change updates the section on native replication for distributed
hypertables with information on how to copy chunks between data nodes
in order to re-replicate chunks after, e.g., a data node failure.

* Update timescaledb/how-to-guides/distributed-hypertables/enable-native-replication.md

Co-authored-by: Lana Brindley <github@lanabrindley.com>

* Update timescaledb/how-to-guides/distributed-hypertables/enable-native-replication.md

Co-authored-by: Lana Brindley <github@lanabrindley.com>

* Update timescaledb/how-to-guides/distributed-hypertables/enable-native-replication.md

Co-authored-by: Lana Brindley <github@lanabrindley.com>

* Update timescaledb/how-to-guides/distributed-hypertables/enable-native-replication.md

Co-authored-by: Ryan Booz <ryan@ryanbooz.com>

Co-authored-by: Lana Brindley <github@lanabrindley.com>
Co-authored-by: Nuno Santos <nunofilipesantos@gmail.com>
Co-authored-by: Ryan Booz <ryan@ryanbooz.com>
Co-authored-by: Ryan Booz <ryan@timescale.com>

timescaledb/how-to-guides/distributed-hypertables/enable-native-replication.md

@@ -4,7 +4,8 @@ A distributed hypertable can be configured to write each chunk to
 multiple data nodes in order to replicate data at the chunk
 level. This *native replication* ensures that a distributed hypertable
 is protected against data node failures and provides an alternative to
-fully replicating each data node using streaming replication.
+fully replicating each data node using streaming replication in order
+to provide high availability.
 
 While data nodes require no additional setup to use native
 replication, the access node continues to rely on streaming
@@ -25,45 +26,80 @@ Alternatively, the function
 [`set_replication_factor`][set_replication_factor] can be used to
 enable native replication on an existing distributed hypertable.
 
-<highlight type="warning">
+<highlight type="warning"> 
 Native replication is currently under development and lacks
-functionality for a complete high-availability solution. Some
-functionality described in this section is not yet implemented or
-might have only a partial feature set. For instance, there is
-currently no implementation for (re-)replicating existing chunks in
-the background after a data node failure or increase in replication
-factor. We therefore recommend keeping the replication factor set at
-the default value of 1, and instead use streaming replication on each
-data node.
+functionality for a complete high-availability solution. For instance,
+if a node fails one might need to (re-)replicate the chunks that had a
+replica on the failed node.  While it is possible to copy chunks from
+one node to another, this functionality is experimental and not yet
+automated. For production environments, we therefore recommend keeping
+the replication factor set at the default value of 1, and instead use
+streaming replication on each data node.
 </highlight>
 
-Once enabled, native replication happens as part of normal inserts, by
+
+Once enabled, native replication happens as part of normal inserts by
 writing each row to multiple data nodes, and therefore requires no
-additional mechanism for replicating newly inserted data. Existing
-chunks, which are not replicated according to the configured
-replication factor, needs to be re-replicated by a background job in
-order to achieve the set replication factor.
+additional mechanism for replicating newly inserted data. When
+querying, the query planner knows how to include only one replica of
+each chunk in the query plan.
+
+
+## Handling node failures
+
+When a data node fails, queries and inserts that involve the failed
+node will fail in order to ensure data consistency until the data node
+is fully available again. If the data node cannot be recovered, native
+replication allows the node to be deleted from the multi-node cluster
+without losing data:
+
+```sql
+SELECT delete_data_node('data_node_2', force => true);
+WARNING:  distributed hypertable "conditions" is under-replicated
+```
+
+Note that it is not possible to force the deletion of a data node if
+it would mean that a distributed hypertable would permanently lose
+data.
+
+Once the failed data node has been removed, some data chunks will lack
+replicas, but queries and inserts should work as normal
+again. However, the multi-node cluster remains in a vulnerable state
+until all chunks that lack replicas are fully replicated again.
+
+To view the chunks that need to be replicated use this query:
+
+```sql
+SELECT chunk_schema, chunk_name, replica_nodes, non_replica_nodes 
+FROM timescaledb_experimental.chunk_replication_status 
+WHERE hypertable_name = 'conditions' AND num_replicas < desired_num_replicas;
+     chunk_schema      |      chunk_name       | replica_nodes |     non_replica_nodes
+-----------------------+-----------------------+---------------+---------------------------
+ _timescaledb_internal | _dist_hyper_1_1_chunk | {data_node_3} | {data_node_1,data_node_2}
+ _timescaledb_internal | _dist_hyper_1_3_chunk | {data_node_1} | {data_node_2,data_node_3}
+ _timescaledb_internal | _dist_hyper_1_4_chunk | {data_node_3} | {data_node_1,data_node_2}
+(3 rows)
+```
+
+With the information from the chunk replication status view, an
+under-replicated chunk can be copied to a new node to ensure the chunk
+has sufficient number of replicas:
+
+```sql
+CALL timescaledb_experimental.copy_chunk('_timescaledb_internal._dist_hyper_1_1_chunk', 'data_node_3', 'data_node_2');
+```
 
-To view the data nodes each chunk is replicated to, the following
-query can be used:
+The chunk is copied over several transactions and cannot be
+rolled back automatically. If the copy operation is aborted or
+terminated prematurely by the user, an operation ID for the aborted
+copy is logged. This operation ID can later be used to clean up
+any state left by the aborted operation:
 
 ```sql
-SELECT chunk_name, data_nodes
-FROM timescaledb_information.chunks
-WHERE hypertable_name = 'conditions';
-
-	   chunk_name       |              data_nodes
-------------------------+---------------------------------------
- _dist_hyper_1_1_chunk  | {data_node_1,data_node_2,data_node_3}
- _dist_hyper_1_2_chunk  | {data_node_1,data_node_2,data_node_3}
- _dist_hyper_1_3_chunk  | {data_node_1,data_node_2,data_node_3}
+CALL timescaledb_experimental.cleanup_copy_chunk_operation('ts_copy_1_31');
 ```
 
-When querying a distributed hypertable using native replication, the
-query planner knows how to include only one replica of each chunk in
-the query plan. The planner can employ different strategies to pick
-the set of chunk replicas in order to, e.g., evenly spread the query
-load across the data nodes.
+## Comparing native and streaming replication
 
 Compared to streaming replication, native replication provides several
 advantages: