Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sig-cl/kubeadm/1739: update KEP for support of patching kubelet config #3312

Merged
merged 1 commit into from
Jun 6, 2022
Merged

sig-cl/kubeadm/1739: update KEP for support of patching kubelet config #3312

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 59 additions & 23 deletions keps/sig-cluster-lifecycle/kubeadm/1739-customization-with-patches/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,12 +87,16 @@ tags, and then generate with `hack/update-toc.sh`.
- [Story 1](#story-1)
- [Story 2](#story-2)
- [Story 3](#story-3)
- [Story 4](#story-4)
- [Notes/Constraints/Caveats (optional)](#notesconstraintscaveats-optional)
- [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
- [Code organization](#code-organization)
- [Patch formats](#patch-formats)
- [Patch file naming format](#patch-file-naming-format)
- [Patch targets](#patch-targets)
- [Control plane static Pods](#control-plane-static-pods)
- [Kubelet ComponentConfig](#kubelet-componentconfig)
- [Support for multiple patches per file](#support-for-multiple-patches-per-file)
- [Test Plan](#test-plan)
- [Graduation Criteria](#graduation-criteria)
Expand Down Expand Up @@ -168,11 +172,12 @@ A good summary is probably at least a paragraph in length.
This KEP proposes a replacement for the feature introduced by the kubeadm KEP
"20190722-Advanced-configurations-with-kubeadm-(Kustomize)".

It has the same scope and goals, allowing the users to amend manifests
generated by kubeadm using patches, but not using Kustomize as the backend
for applying the patches.
It has similar scope and goals, allowing the users to amend manifests
and configuration files generated by kubeadm using patches, but not using
Kustomize as the backend for applying the patches.

The existing Kustomize implementation in kubeadm is in Alpha state.
The Kustomize implementation in kubeadm was in an Alpha state and was
removed.

## Motivation

Expand All @@ -198,10 +203,10 @@ know that this has succeeded?
-->

- Implement a solution for applying patches over kubeadm generated manifests
during the common kubeadm commands "init", "join" and "upgrade" and their phases.
and configuration during the common kubeadm commands "init", "join" and "upgrade"
and their phases.
- Deprecate and remove the existing Kustomize solution after a grace period
of at least one release.
- To initially allow patching of static Pod manifests only.

### Non-Goals

Expand All @@ -214,8 +219,6 @@ and make progress.
sane security values.
- Allow patching of core addon configuration generated by kubeadm - e.g. CoreDNS,
kube-proxy (until further notice).
- Allowing patches to be added using the kubeadm ComponentConfig (until
further notice).
- Replace the kubeadm ComponentConfig. The sane defaults would still be
generated by kubeadm.

Expand Down Expand Up @@ -265,6 +268,11 @@ liveness probes for edge clusters.
As a cluster administrator, I want to upgrade my cluster preserving all the
patches that were applied during "init"/"join".

#### Story 4

As a cluster administrator, I want to be able to apply instance specific
configuration to kubelets on different nodes.

### Notes/Constraints/Caveats (optional)

<!--
Expand Down Expand Up @@ -304,7 +312,7 @@ and release notes what is the scope of the new feature.
- Users are allowed to pass configuration for components using the
kubeadm ComponentConfiguration.
- kubeadm applies sane / secure defaults.
- The users can patch the generated manifests to their liking.
- The users can patch the generated configuration to their liking.

_Misleading expectations on the level of flexibility_

Expand All @@ -315,7 +323,7 @@ will be examined before moving forward in graduating the feature to beta.

_Breaking changes post upgrade_

A change in a kubeadm manifest can make a patch apply to fail.
A change in a kubeadm generated configuration can make a patch apply to fail.

The kubeadm maintainers will work on release notes to make potential
breaking changes more visible. Additionally, upgrade instructions will be
Expand Down Expand Up @@ -388,16 +396,17 @@ Conversion from YAML to JSON will be performed using the

Once the user passes a folder with patches to a kubeadm command,
kubeadm must be able to determine what format each file is and what
manifest it is targeting.
is its target.

Parsing the patch contents to determine the patch format is not possible,
because the formats `strategic` and `merge` are the same, while they do
have different apply mechanisms.

The proposal is to have the following naming format:
`componentname[suffix][+patchtype].{yaml|json}`
`target[suffix][+patchtype].{yaml|json}`

- `componentname` must be a known component name - e.g. `kube-apiserver`.
- `target` must be a known component name - e.g. `kube-apiserver` or
the name of generated configuration.
- `suffix` is optional and can be used to ensure an order when applying
patches from multiple files for the same component.
- `+patchtype` is optional and can be used to pass one of the supported
Expand All @@ -424,6 +433,28 @@ but also allow for advanced usage with patch ordering and the `json` type.
- Would allow having arbitrary files that are ignored in the patch folder -
e.g. having a `README.md`.

### Patch targets

The mechanism allows patching the following targets via the `target` naming property:

#### Control plane static Pods

Static Pod manifests generated on control plane nodes can be patched with the following
targets: `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and `etcd`.
Their API GroupVersionKind is `core/v1.Pod`.

Patches for the control plane static Pods are applied over the generated manifests
in memory during `init/join/upgrade`.

#### Kubelet ComponentConfig

The generated per-node kubelet ComponentConfig can be patched with the `kubeletconfiguration`
target. Its API GroupVersionKind is of type `kubelet.config.k8s.io/v1beta1.KubeletConfiguration`.

During `init` patches for the kubelet are applied over the user provided and kubeadm defaulted
`KubeletConfiguration`. During `join/upgrade` patches are applied over the downloaded
from the cluster global `KubeletConfiguration`.
Copy link
Member

@pacoxu pacoxu May 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update LGTM.
My only concern is about how to make a new user notice this option and there is a patch for some components.

  • the documentation
  • maybe we can add some tips in the configmap/static-pod annotation like "kubeadm.kubernetes.io/patched.component=kube-apiserver".
  • some logs during kubeadm init/join/upgrade process.

Copy link
Member Author

@neolit123 neolit123 May 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the presence of a --patches flag or the option in config is the main indicator that components are being patched. And the user technically owns / maintains these flags and the config, so they must know what they are passing to kubeadm.

the documentation

I plan to update this page:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/

maybe we can add some tips in the configmap annotation like "kubeadm.kubernetes.io/patched.component=kube-apiserver".

Today we do not annotate static pods whether they are modified from kubeadm defaults. There is this extra customization layer due to extraArgs, volumes and mounts that applies over the defaults. We assume the config file is the source of truth and users will be able to trace back changes to it. Patches are also a customization layer and we assume that new users should look at the node config and see there is a patches directory.

I think such an annotation is doable, but its not clear to me whether it's going to be a net positive addition for us and the users.

some logs during kubeadm init/join/upgrade process.

Yes, logs already print that a patch folder is being processed and what patches were found there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm


### Support for multiple patches per file

To support multiple patches per file, the proposal is to implement
Expand Down Expand Up @@ -537,6 +568,7 @@ ComponentConfig version increment, so could land in GA graduation.
- The feature is widely used and at least 2 cycles have passed since Beta.
- The `--patches` flag is removed (optional). Depends on user feedback
and the wider kubeadm plan for removing flags in favor of ComponentConfig.
- More patch targets are considered (optional).

### Upgrade / Downgrade Strategy

Expand Down Expand Up @@ -575,6 +607,8 @@ enhancement:
CRI or CNI may require updating that component before the kubelet.
-->

This section covers the `corev1.Pod` as a example target for version skew.

Once introduced, the feature will support patching at minimum the `corev1.Pod`
object. If at some point the Kubernetes core API graduates to `v2`, kubeadm would
have to support patching both `corev2` and `corev1` for at least one release
Expand All @@ -584,8 +618,7 @@ This is due to the fact kubeadm supports deploying Kubernetes `v1.YY` and `v1.YY
And if `corev2` is added in `v1.YY`, kubeadm must be able to handle `corev1`
that is supported in `v1.YY-1`.

If the patching mechanic one day supports patching different API types,
it must also apply similar skew strategy to those types.
A similar skew strategy must be applied for other supported patch target API types.

## Production Readiness Review Questionnaire

Expand Down Expand Up @@ -624,7 +657,7 @@ _This section must be completed when targeting alpha to a release._
- [x] Other
- Describe the mechanism:
When a new kubeadm control-plane Node joins the cluster it can optionally
apply custom patches to the static Pods that kubeadm configures.
apply custom patches to kubeadm generated configuration.
- Will enabling / disabling the feature require downtime of the control
plane?
No, unless the user created patches result in bad configuration and
Expand All @@ -646,7 +679,7 @@ _This section must be completed when targeting alpha to a release._
feature, can it break the existing applications?).
Yes. Once the patches are applied for a `kubeadm init|join|upgrade` command,
the user can decide to roll-back the changes which will result in a restart
of the control-plane component static Pods that were patches on this Node.
of component that was customized by the patches.

* **What happens if we reenable the feature if it was previously rolled back?**
This is supported by invoking kubeadm phases. The patches will re-apply
Expand All @@ -669,7 +702,7 @@ _This section must be completed when targeting beta graduation to a release._
Try to be as paranoid as possible - e.g. what if some components will restart
in the middle of rollout?
The feature can result in malfunctioning control-plane components, due to the
fact that it targets patching of static Pod manifests with user values.
fact that it targets patching of configuration with user values.
The user is responsible for configuring the control-plane correctly.

* **What specific metrics should inform a rollback?**
Expand Down Expand Up @@ -744,7 +777,7 @@ _This section must be completed when targeting beta graduation to a release._
- Usage description:
- Impact of its outage on the feature:
- Impact of its degraded performance or high error rates on the feature:
No, this feature customizes static Pod manifests for the control-plane.
No, this feature customizes on top of kubeadm generated configuration.

### Scalability

Expand Down Expand Up @@ -793,8 +826,8 @@ No.
operations covered by [existing SLIs/SLOs][]?**
Think about adding additional work or introducing new steps in between
(e.g. need to do X to start a container), etc. Please describe the details.
No, unless the user patches the control-plane static Pods with undesired
configuration.
No, unless the user patches kubeadm generated configuration with undesired
settings.

* **Will enabling / using this feature result in non-negligible increase of
resource usage (CPU, RAM, disk, IO, ...) in any components?**
Expand All @@ -803,8 +836,8 @@ configuration.
volume), significant amount of data send and/or received over network, etc.
This through this both in small and large cases, again with respect to the
[supported limits][].
No, unless the user patches the control-plane static Pods with undesired
configuration.
No, unless the user patches kubeadm generated configuration with undesired
settings.

### Troubleshooting

Expand Down Expand Up @@ -859,6 +892,9 @@ Major milestones might include
- 2020-05-14: addressed feedback, filled PRR questionnaire, KEP marked as implementable
- 2021-09-09: marked the feature as graduated to Beta. This was done as part of
the kubeadm v1beta3 API work. The actions under "Alpha -> Beta" are completed.
- 2022-05-16: adapted the KEP to be more generic, allowing support for patching
any kubeadm generated configuration in the future and not only control plane component
manifests. Added enumeration of supported patch targets.

## Drawbacks

Expand Down