Merge pull request #13852 from serathius/recommend

changelog: Update and deduplicate production recommendations
etcd-io · Mar 29, 2022 · 0e83f62 · 0e83f62
2 parents 27e222e + 88a39d7
commit 0e83f62
Show file tree

Hide file tree

Showing 6 changed files with 21 additions and 14 deletions.
diff --git a/CHANGELOG/CHANGELOG-3.1.md b/CHANGELOG/CHANGELOG-3.1.md
@@ -2,10 +2,6 @@
 
 Previous change logs can be found at [CHANGELOG-3.0](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.0.md).
 
-
-The minimum recommended etcd versions to run in **production** are 3.3.18+, 3.4.2+, and 3.5.2+.
-
-
 <hr>
 
 ## [v3.1.21](https://github.com/etcd-io/etcd/releases/tag/v3.1.21) (2019-TBD)

diff --git a/CHANGELOG/CHANGELOG-3.2.md b/CHANGELOG/CHANGELOG-3.2.md
@@ -3,8 +3,6 @@
 Previous change logs can be found at [CHANGELOG-3.1](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.1.md).
 
 
-The minimum recommended etcd versions to run in **production** are 3.3.18+, 3.4.2+, and 3.5.2+.
-
 ## v3.2.33 (TBD)
 
 <hr>

diff --git a/CHANGELOG/CHANGELOG-3.3.md b/CHANGELOG/CHANGELOG-3.3.md
@@ -2,9 +2,6 @@
 
 Previous change logs can be found at [CHANGELOG-3.2](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.2.md).
 
-
-The minimum recommended etcd versions to run in **production** are 3.3.18+, 3.4.2+, and 3.5.2+. Refer to the [versioning policy](https://etcd.io/docs/v3.3/op-guide/versioning/) for more details.
-
 <hr>
 
 ## v3.3.27 (2021-10-15)

diff --git a/CHANGELOG/CHANGELOG-3.4.md b/CHANGELOG/CHANGELOG-3.4.md
@@ -2,9 +2,6 @@
 
 Previous change logs can be found at [CHANGELOG-3.3](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.3.md).
 
-
-The minimum recommended etcd versions to run in **production** are 3.3.18+, 3.4.2+, and 3.5.2+. Refer to the [versioning policy](https://etcd.io/docs/v3.4/op-guide/versioning/) for more details.
-
 <hr>
 
 ## v3.4.19 (TODO)

diff --git a/CHANGELOG/CHANGELOG-3.5.md b/CHANGELOG/CHANGELOG-3.5.md
@@ -3,8 +3,6 @@
 Previous change logs can be found at [CHANGELOG-3.4](https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.4.md).
 
 
-The minimum recommended etcd versions to run in **production** are 3.3.18+, 3.4.2+, and 3.5.2+. Refer to the [versioning policy](https://etcd.io/docs/v3.5/op-guide/versioning/) for more details.
-
 <hr>
 
 ## v3.5.3 (TBD)

diff --git a/CHANGELOG/README.md b/CHANGELOG/README.md
@@ -0,0 +1,21 @@
+# Production recommendation
+
+The minimum recommended etcd versions to run in **production** are 3.3.18+, 3.4.2+. Refer to the [versioning policy](https://etcd.io/docs/v3.5/op-guide/versioning/) for more details.
+
+Etcd v3.5.[0-2] versions are no longer recommended for production due to data corruption issue.
+
+### v3.5 data corruption issue 
+
+Running etcd v3.5.2, v3.5.1 and v3.5.0 under high load can cause a data corruption issue.
+If etcd process is killed, occasionally some committed transactions are not reflected on all the members.
+
+Recommendations if you are running v3.4.X:
+* **Don't upgrade your etcd clusters to v3.5** until the problem is fixed in the upcoming v3.5.3 release.
+* There are no breaking changes in API, meaning **it’s safe to let v3.5 clients (e.g. the latest Kubernetes releases) talk to v3.4 servers**.
+
+Recommendations if you are running v3.5.0, v3.5.1, or v3.5.2:
+* **Enable data corruption check** with `--experimental-initial-corrupt-check` flag. The flag is the only reliable automated way of detecting an inconsistency. This mode has seen significant usage in production and is going to be promoted as default in etcd v3.6.
+* **Ensure etcd cluster is not memory pressured or sigkill interrupted**, which could lead to processes being disrupted in the middle of business logic and trigger the issue.
+* **Etcd downgrade should be avoided** as they are not officially supported and clusters can be safely recovered as long as data corruption check is enabled.
+
+If you have encountered data corruption, please follow instructions on https://etcd.io/docs/v3.5/op-guide/data_corruption/.