Merge pull request #3312 from neolit123/1.25-kubeadm-patches-for-kubelet

sig-cl/kubeadm/1739: update KEP for support of patching kubelet config

keps/sig-cluster-lifecycle/kubeadm/1739-customization-with-patches/README.md

@@ -87,12 +87,16 @@ tags, and then generate with `hack/update-toc.sh`.
     - [Story 1](#story-1)
     - [Story 2](#story-2)
     - [Story 3](#story-3)
+    - [Story 4](#story-4)
   - [Notes/Constraints/Caveats (optional)](#notesconstraintscaveats-optional)
   - [Risks and Mitigations](#risks-and-mitigations)
 - [Design Details](#design-details)
   - [Code organization](#code-organization)
   - [Patch formats](#patch-formats)
   - [Patch file naming format](#patch-file-naming-format)
+  - [Patch targets](#patch-targets)
+    - [Control plane static Pods](#control-plane-static-pods)
+    - [Kubelet ComponentConfig](#kubelet-componentconfig)
   - [Support for multiple patches per file](#support-for-multiple-patches-per-file)
   - [Test Plan](#test-plan)
   - [Graduation Criteria](#graduation-criteria)
@@ -168,11 +172,12 @@ A good summary is probably at least a paragraph in length.
 This KEP proposes a replacement for the feature introduced by the kubeadm KEP
 "20190722-Advanced-configurations-with-kubeadm-(Kustomize)".
 
-It has the same scope and goals, allowing the users to amend manifests
-generated by kubeadm using patches, but not using Kustomize as the backend
-for applying the patches.
+It has similar scope and goals, allowing the users to amend manifests
+and configuration files generated by kubeadm using patches, but not using
+Kustomize as the backend for applying the patches.
 
-The existing Kustomize implementation in kubeadm is in Alpha state.
+The Kustomize implementation in kubeadm was in an Alpha state and was
+removed.
 
 ## Motivation
 
@@ -198,10 +203,10 @@ know that this has succeeded?
 -->
 
 - Implement a solution for applying patches over kubeadm generated manifests
-during the common kubeadm commands "init", "join" and "upgrade" and their phases.
+and configuration during the common kubeadm commands "init", "join" and "upgrade"
+and their phases.
 - Deprecate and remove the existing Kustomize solution after a grace period
 of at least one release.
-- To initially allow patching of static Pod manifests only.
 
 ### Non-Goals
 
@@ -214,8 +219,6 @@ and make progress.
 sane security values.
 - Allow patching of core addon configuration generated by kubeadm - e.g. CoreDNS,
 kube-proxy (until further notice).
-- Allowing patches to be added using the kubeadm ComponentConfig (until
-further notice).
 - Replace the kubeadm ComponentConfig. The sane defaults would still be
 generated by kubeadm.
 
@@ -265,6 +268,11 @@ liveness probes for edge clusters.
 As a cluster administrator, I want to upgrade my cluster preserving all the
 patches that were applied during "init"/"join".
 
+#### Story 4
+
+As a cluster administrator, I want to be able to apply instance specific
+configuration to kubelets on different nodes.
+
 ### Notes/Constraints/Caveats (optional)
 
 <!--
@@ -304,7 +312,7 @@ and release notes what is the scope of the new feature.
 - Users are allowed to pass configuration for components using the
 kubeadm ComponentConfiguration.
 - kubeadm applies sane / secure defaults.
-- The users can patch the generated manifests to their liking.
+- The users can patch the generated configuration to their liking.
 
 _Misleading expectations on the level of flexibility_
 
@@ -315,7 +323,7 @@ will be examined before moving forward in graduating the feature to beta.
 
 _Breaking changes post upgrade_
 
-A change in a kubeadm manifest can make a patch apply to fail.
+A change in a kubeadm generated configuration can make a patch apply to fail.
 
 The kubeadm maintainers will work on release notes to make potential
 breaking changes more visible. Additionally, upgrade instructions will be
@@ -388,16 +396,17 @@ Conversion from YAML to JSON will be performed using the
 
 Once the user passes a folder with patches to a kubeadm command,
 kubeadm must be able to determine what format each file is and what
-manifest it is targeting.
+is its target.
 
 Parsing the patch contents to determine the patch format is not possible,
 because the formats `strategic` and `merge` are the same, while they do
 have different apply mechanisms.
 
 The proposal is to have the following naming format:
-`componentname[suffix][+patchtype].{yaml|json}`
+`target[suffix][+patchtype].{yaml|json}`
 
-- `componentname` must be a known component name - e.g. `kube-apiserver`.
+- `target` must be a known component name - e.g. `kube-apiserver` or
+the name of generated configuration.
 - `suffix` is optional and can be used to ensure an order when applying
 patches from multiple files for the same component.
 - `+patchtype` is optional and can be used to pass one of the supported
@@ -424,6 +433,28 @@ but also allow for advanced usage with patch ordering and the `json` type.
 - Would allow having arbitrary files that are ignored in the patch folder -
 e.g. having a `README.md`.
 
+### Patch targets
+
+The mechanism allows patching the following targets via the `target` naming property:
+
+#### Control plane static Pods
+
+Static Pod manifests generated on control plane nodes can be patched with the following
+targets: `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and `etcd`.
+Their API GroupVersionKind is `core/v1.Pod`.
+
+Patches for the control plane static Pods are applied over the generated manifests
+in memory during `init/join/upgrade`.
+
+#### Kubelet ComponentConfig
+
+The generated per-node kubelet ComponentConfig can be patched with the `kubeletconfiguration`
+target. Its API GroupVersionKind is of type `kubelet.config.k8s.io/v1beta1.KubeletConfiguration`.
+
+During `init` patches for the kubelet are applied over the user provided and kubeadm defaulted
+`KubeletConfiguration`. During `join/upgrade` patches are applied over the downloaded
+from the cluster global `KubeletConfiguration`.
+
 ### Support for multiple patches per file
 
 To support multiple patches per file, the proposal is to implement
@@ -537,6 +568,7 @@ ComponentConfig version increment, so could land in GA graduation.
 - The feature is widely used and at least 2 cycles have passed since Beta.
 - The `--patches` flag is removed (optional). Depends on user feedback
 and the wider kubeadm plan for removing flags in favor of ComponentConfig.
+- More patch targets are considered (optional).
 
 ### Upgrade / Downgrade Strategy
 
@@ -575,6 +607,8 @@ enhancement:
   CRI or CNI may require updating that component before the kubelet.
 -->
 
+This section covers the `corev1.Pod` as a example target for version skew.
+
 Once introduced, the feature will support patching at minimum the `corev1.Pod`
 object. If at some point the Kubernetes core API graduates to `v2`, kubeadm would
 have to support patching both `corev2` and `corev1` for at least one release
@@ -584,8 +618,7 @@ This is due to the fact kubeadm supports deploying Kubernetes `v1.YY` and `v1.YY
 And if `corev2` is added in `v1.YY`, kubeadm must be able to handle `corev1`
 that is supported in `v1.YY-1`.
 
-If the patching mechanic one day supports patching different API types,
-it must also apply similar skew strategy to those types.
+A similar skew strategy must be applied for other supported patch target API types.
 
 ## Production Readiness Review Questionnaire
 
@@ -624,7 +657,7 @@ _This section must be completed when targeting alpha to a release._
   - [x] Other
     - Describe the mechanism:
       When a new kubeadm control-plane Node joins the cluster it can optionally
-      apply custom patches to the static Pods that kubeadm configures.
+      apply custom patches to kubeadm generated configuration.
     - Will enabling / disabling the feature require downtime of the control
       plane?
       No, unless the user created patches result in bad configuration and
@@ -646,7 +679,7 @@ _This section must be completed when targeting alpha to a release._
   feature, can it break the existing applications?).
 Yes. Once the patches are applied for a `kubeadm init|join|upgrade` command,
 the user can decide to roll-back the changes which will result in a restart
-of the control-plane component static Pods that were patches on this Node.
+of component that was customized by the patches.
 
 * **What happens if we reenable the feature if it was previously rolled back?**
 This is supported by invoking kubeadm phases. The patches will re-apply
@@ -669,7 +702,7 @@ _This section must be completed when targeting beta graduation to a release._
   Try to be as paranoid as possible - e.g. what if some components will restart
   in the middle of rollout?
 The feature can result in malfunctioning control-plane components, due to the
-fact that it targets patching of static Pod manifests with user values.
+fact that it targets patching of configuration with user values.
 The user is responsible for configuring the control-plane correctly.
 
 * **What specific metrics should inform a rollback?**
@@ -744,7 +777,7 @@ _This section must be completed when targeting beta graduation to a release._
     - Usage description:
       - Impact of its outage on the feature:
       - Impact of its degraded performance or high error rates on the feature:
-No, this feature customizes static Pod manifests for the control-plane.
+No, this feature customizes on top of kubeadm generated configuration.
 
 ### Scalability
 
@@ -793,8 +826,8 @@ No.
   operations covered by [existing SLIs/SLOs][]?**
   Think about adding additional work or introducing new steps in between
   (e.g. need to do X to start a container), etc. Please describe the details.
-No, unless the user patches the control-plane static Pods with undesired
-configuration.
+No, unless the user patches kubeadm generated configuration with undesired
+settings.
 
 * **Will enabling / using this feature result in non-negligible increase of
   resource usage (CPU, RAM, disk, IO, ...) in any components?**
@@ -803,8 +836,8 @@ configuration.
   volume), significant amount of data send and/or received over network, etc.
   This through this both in small and large cases, again with respect to the
   [supported limits][].
-No, unless the user patches the control-plane static Pods with undesired
-configuration.
+No, unless the user patches kubeadm generated configuration with undesired
+settings.
 
 ### Troubleshooting
 
@@ -859,6 +892,9 @@ Major milestones might include
 - 2020-05-14: addressed feedback, filled PRR questionnaire, KEP marked as implementable
 - 2021-09-09: marked the feature as graduated to Beta. This was done as part of
 the kubeadm v1beta3 API work. The actions under "Alpha -> Beta" are completed.
+- 2022-05-16: adapted the KEP to be more generic, allowing support for patching
+any kubeadm generated configuration in the future and not only control plane component
+manifests. Added enumeration of supported patch targets.
 
 ## Drawbacks