diff --git a/docs/getting_started/create_cluster.md b/docs/getting_started/create_cluster.md index 029b4bad..6d1888fc 100644 --- a/docs/getting_started/create_cluster.md +++ b/docs/getting_started/create_cluster.md @@ -168,24 +168,6 @@ $ kubectl exec simple-cluster-rep2-0 -- curl -sSf localhost:9002/health {"ok":true,"status":"up","bootstrapped":true} ``` -## Deleting a Cluster - -Delete your M3DB cluster with `kubectl`: -``` -kubectl delete m3dbcluster simple-cluster -``` - -After deleting a cluster you'll have to delete the metadata in etcd if you want to reuse the same etcd cluster for a new -M3DB cluster. On our roadmap is to add a finalizer to `m3dbcluster` API objects to allow the operator to clean up their -data in etcd. Until then, you'll have to do so manually: - -**WARNING**: This will delete all data in your etcd cluster. If you are using this cluster for anything other than M3DB, -you will want to delete only the relevant keys (generally anything containing the string `m3db`). - -``` -kubectl exec etcd-0 -- env ETCDCTL_API=3 etcdctl del --prefix "" -``` - [pod-identity]: ../configuration/pod_identity [local-volumes]: https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/ [node-affinity]: ../configuration/node_affinity diff --git a/docs/getting_started/delete_cluster.md b/docs/getting_started/delete_cluster.md new file mode 100644 index 00000000..06decd82 --- /dev/null +++ b/docs/getting_started/delete_cluster.md @@ -0,0 +1,41 @@ +# Deleting a Cluster + +Delete your M3DB cluster with `kubectl`: +``` +kubectl delete m3dbcluster simple-cluster +``` + +By default, the operator will delete the placement and namespaces associated with a cluster before the CRD resource +deleted. If you do **not** want this behavior, set `keepEtcdDataOnDelete` to `true` on your cluster spec. + +Under the hood, the operator uses Kubernetes [finalizers] to ensure the cluster CRD is not deleted until the operator +has had a chance to do cleanup. + +## Debugging Stuck Cluster Deletion + +If for some reason the operator is unable to delete the placement and namespace for the cluster, the cluster CRD itself +will be stuck in a state where it can not be deleted, due to the way finalizers work in Kubernetes. The operator might +be unable to clean up the data for many reasons, for example if the M3DB cluster itself is not available to serve the +APIs for cleanup or if etcd is down and cannot fulfill the deleted. + +To allow the CRD to be deleted, you can `kubectl edit m3dbcluster $CLUSTER` and remove the +`operator.m3db.io/etcd-deletion` finalizer. For example, in the following cluster you'd remove the finalizer from `metadata.finalizers`: + +```yaml +apiVersion: operator.m3db.io/v1alpha1 +kind: M3DBCluster +metadata: + ... + finalizers: + - operator.m3db.io/etcd-deletion + name: m3db-cluster +... +``` + +Note that if you do this, you'll have to manually remove the relevant data in etcd. For a cluster in namespace `$NS` +with name `$CLUSTER`, the keys are: + +- `_sd.placement/$NS/$CLUSTER/m3db` +- `_kv/$NS/$CLUSTER/m3db.node.namespaces` + +[finalizers]: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#finalizers diff --git a/mkdocs.yml b/mkdocs.yml index fd281655..a6dd45bd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -58,6 +58,7 @@ pages: - "Requirements": "getting_started/requirements.md" - "Installation": "getting_started/installation.md" - "Creating a Cluster": "getting_started/create_cluster.md" + - "Deleting a Cluster": "getting_started/delete_cluster.md" - "Monitoring": "getting_started/monitoring.md" - "Configuration": - "Configuring M3DB": "configuration/configuring_m3db.md"