[docs] Update instructions on cluster deletion (#181)

We've changed the way cluster deletion works due to the introduction of
the etcd cleanup finalizer. This PR updates the docs to reflect that.

docs/getting_started/create_cluster.md

@@ -168,24 +168,6 @@ $ kubectl exec simple-cluster-rep2-0 -- curl -sSf localhost:9002/health
 {"ok":true,"status":"up","bootstrapped":true}
 ```
 
-## Deleting a Cluster
-
-Delete your M3DB cluster with `kubectl`:
-```
-kubectl delete m3dbcluster simple-cluster
-```
-
-After deleting a cluster you'll have to delete the metadata in etcd if you want to reuse the same etcd cluster for a new
-M3DB cluster. On our roadmap is to add a finalizer to `m3dbcluster` API objects to allow the operator to clean up their
-data in etcd. Until then, you'll have to do so manually:
-
-**WARNING**: This will delete all data in your etcd cluster. If you are using this cluster for anything other than M3DB,
-you will want to delete only the relevant keys (generally anything containing the string `m3db`).
-
-```
-kubectl exec etcd-0 -- env ETCDCTL_API=3 etcdctl del --prefix ""
-```
-
 [pod-identity]: ../configuration/pod_identity
 [local-volumes]: https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/
 [node-affinity]: ../configuration/node_affinity

docs/getting_started/delete_cluster.md

@@ -0,0 +1,41 @@
+# Deleting a Cluster
+
+Delete your M3DB cluster with `kubectl`:
+```
+kubectl delete m3dbcluster simple-cluster
+```
+
+By default, the operator will delete the placement and namespaces associated with a cluster before the CRD resource
+deleted. If you do **not** want this behavior, set `keepEtcdDataOnDelete` to `true` on your cluster spec.
+
+Under the hood, the operator uses Kubernetes [finalizers] to ensure the cluster CRD is not deleted until the operator
+has had a chance to do cleanup.
+
+## Debugging Stuck Cluster Deletion
+
+If for some reason the operator is unable to delete the placement and namespace for the cluster, the cluster CRD itself
+will be stuck in a state where it can not be deleted, due to the way finalizers work in Kubernetes. The operator might
+be unable to clean up the data for many reasons, for example if the M3DB cluster itself is not available to serve the
+APIs for cleanup or if etcd is down and cannot fulfill the deleted.
+
+To allow the CRD to be deleted, you can `kubectl edit m3dbcluster $CLUSTER` and remove the
+`operator.m3db.io/etcd-deletion` finalizer. For example, in the following cluster you'd remove the finalizer from `metadata.finalizers`:
+
+```yaml
+apiVersion: operator.m3db.io/v1alpha1
+kind: M3DBCluster
+metadata:
+  ...
+  finalizers:
+  - operator.m3db.io/etcd-deletion
+  name: m3db-cluster
+...
+```
+
+Note that if you do this, you'll have to manually remove the relevant data in etcd. For a cluster in namespace `$NS`
+with name `$CLUSTER`, the keys are:
+
+- `_sd.placement/$NS/$CLUSTER/m3db`
+- `_kv/$NS/$CLUSTER/m3db.node.namespaces`
+
+[finalizers]: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#finalizers

mkdocs.yml

@@ -58,6 +58,7 @@ pages:
     - "Requirements": "getting_started/requirements.md"
     - "Installation": "getting_started/installation.md"
     - "Creating a Cluster": "getting_started/create_cluster.md"
+    - "Deleting a Cluster": "getting_started/delete_cluster.md"
     - "Monitoring": "getting_started/monitoring.md"
   - "Configuration":
     - "Configuring M3DB": "configuration/configuring_m3db.md"