kubeadm: Improve the kubelet default configuration security-wise #64187

luxas · 2018-05-23T08:02:06Z

What this PR does / why we need it:

Disables the readonly port for the kubelets in the cluster
Enables delegated SA token authentication for the secure kubelet port (GCE also did this ref: enable token authentication for kubelets in GCE #58178)
Follows up kubelet: Move RotateCertificates to the KubeletConfiguration struct #63912 to move the last flag from the system dropin to the ComponentConfig

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes kubernetes/kubeadm#732
Fixes kubernetes/kubeadm#650
Replaces #57997

Special notes for your reviewer:
In order to make sure this actually works, or that clusters actually are secure, we're adding e2e tests for this: kubernetes/kubeadm#838 & #64140
Depends on #63912

Release note:

[action required] kubeadm: kubelets in kubeadm clusters now disable the readonly port (10255). If you're relying on unauthenticated access to the readonly port, please switch to using the secure port (10250). Instead, you can now use ServiceAccount tokens when talking to the secure port, which will make it easier to get access to e.g. the `/metrics` endpoint of the kubelet securely.

@kubernetes/sig-cluster-lifecycle-pr-reviews
@kubernetes/sig-auth-pr-reviews FYI

dixudx · 2018-05-23T12:50:22Z

/retest

mtaufen · 2018-05-23T17:55:29Z

cmd/kubeadm/app/apis/kubeadm/v1alpha1/defaults.go


 	// Disable the readonly port of the kubelet, in order to not expose unnecessary information
-	// TODO: Enable in a future PR
-	// obj.KubeletConfiguration.BaseConfig.ReadOnlyPort = 0
+	obj.KubeletConfiguration.BaseConfig.ReadOnlyPort = 0


Note that ReadOnlyPort is already 0 by default when you load a config from a file, or use dynamic config.
Similar for a few others too, like Webhook auth. The v1beta1 defaulter is the canonical source of defaults, and we maintain the old defaults for flags via applyLegacyDefaults in cmd/kubelet/app/options/options.go.

So if you're just generating the default Kubelet config, then you'll get the file/dynamic config defaults if you do it like NewKubeletConfiguration in options.go.

Ah, noticed above you're doing this:

if obj.KubeletConfiguration.BaseConfig == nil { obj.KubeletConfiguration.BaseConfig = &kubeletconfigv1beta1.KubeletConfiguration{} }

You might want to run it through the defaulter, like NewKubeletConfiguration in options.go.

We do run these through the defaulter, see later down the func where we construct the scheme.
But I like to enforce this here as well (double-checking basically to be safe), and be explicit with what we require. Does that make sense? Yeah I saw these are way more secure by default when not using the flags 👍

Yup, that's totally reasonable.

luxas · 2018-05-23T18:06:59Z

/retest

timothysc

/lgtm
/approve

timothysc · 2018-05-23T18:13:14Z

cmd/kubeadm/app/apis/kubeadm/v1alpha2/defaults.go

+	obj.KubeletConfiguration.BaseConfig.ReadOnlyPort = 0
+
+	// Enables client certificate rotation for the kubelet
+	obj.KubeletConfiguration.BaseConfig.RotateCertificates = true


For my own edification what's the default policy for kubelet cert rotation?

false, don't rotate an expired cert. It's beta, not GA yet. kubernetes/enhancements#266

k8s-ci-robot · 2018-05-23T18:13:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: luxas, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [luxas,timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2018-05-23T20:35:06Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-ci-robot · 2018-05-23T20:52:09Z

New changes are detected. LGTM label has been removed.

luxas · 2018-05-23T20:52:38Z

just fixed a fuzzing/roundtrip test

tallclair · 2018-05-23T21:06:35Z

cmd/kubeadm/app/apis/kubeadm/fuzzer/fuzzer.go

@@ -79,6 +79,7 @@ func Funcs(codecs runtimeserializer.CodecFactory) []interface{} {
 							Enabled: utilpointer.BoolPtr(false),
 						},
 					},
+					RotateCertificates: true,


I'm confused by this function. It doesn't look like it's fuzzing anything, just hardcoding the values?

I think it's making sure these values from the fuzzer / these defaults are preserved between conversions using the different API versions.

I was looking at this a while back, and this overrides the default fuzzing for the attributes and hardcodes specific values. We should most likely only be setting values for attributes here if we need to handle cases where randomized fuzzing would generate invalid configs (attributes that are mutually exclusive, dependent on other attributes, etc).

@detiber can you open an issue for improving that? I guess we're setting this for a bit too many items then...

tallclair · 2018-05-23T21:09:31Z

cmd/kubeadm/app/apis/kubeadm/v1alpha1/defaults.go


 	// Disable the readonly port of the kubelet, in order to not expose unnecessary information
-	// TODO: Enable in a future PR
-	// obj.KubeletConfiguration.BaseConfig.ReadOnlyPort = 0
+	obj.KubeletConfiguration.BaseConfig.ReadOnlyPort = 0


I assume you already configure heapster / metrics server to scrape from the 10255 port? (Can you point me to the config, to satisfy my curiosity?)

We actually don't configure heapster / the metrics server in our default configs. We do the bare minimum needed for conformance which doesn't require that kind of monitoring at the moment.

luxas · 2018-05-24T06:39:11Z

/retest

luxas · 2018-05-24T08:49:58Z

/retest

k8s-github-robot · 2018-05-24T16:41:11Z

Automatic merge from submit-queue (batch tested with PRs 64174, 64187, 64216, 63265, 64223). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot requested review from detiber and xiangpengzhao May 23, 2018 08:02

luxas assigned timothysc May 23, 2018

luxas mentioned this pull request May 23, 2018

Enable support for authenticating tokens in the Kubelet API #57997

Closed

luxas assigned mtaufen May 23, 2018

mtaufen reviewed May 23, 2018

View reviewed changes

luxas force-pushed the kubeadm_kubelet_improve_security branch from d80fc95 to 935835b Compare May 23, 2018 18:01

timothysc approved these changes May 23, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2018

kubeadm: Improve the kubelet default configuration security-wise

efc4089

luxas force-pushed the kubeadm_kubelet_improve_security branch from 935835b to efc4089 Compare May 23, 2018 20:52

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2018

luxas added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2018

tallclair reviewed May 23, 2018

View reviewed changes

k8s-github-robot merged commit 698ac55 into kubernetes:master May 24, 2018

bboreham mentioned this pull request Jun 28, 2018

Stop talking to the kubelet unsecured port weaveworks/scope#3242

Open

soolaugust mentioned this pull request Jan 23, 2019

"arene top job" couldn't detect metrics. kubeflow/arena#108

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: Improve the kubelet default configuration security-wise #64187

kubeadm: Improve the kubelet default configuration security-wise #64187

luxas commented May 23, 2018 •

edited

Loading

dixudx commented May 23, 2018

mtaufen May 23, 2018

luxas May 23, 2018

mtaufen May 23, 2018

luxas commented May 23, 2018

timothysc left a comment

timothysc May 23, 2018

luxas May 23, 2018

k8s-ci-robot commented May 23, 2018

fejta-bot commented May 23, 2018

k8s-ci-robot commented May 23, 2018

luxas commented May 23, 2018

tallclair May 23, 2018

luxas May 23, 2018

detiber Jun 5, 2018

luxas Jun 5, 2018

tallclair May 23, 2018

luxas May 23, 2018

luxas commented May 24, 2018

luxas commented May 24, 2018

k8s-github-robot commented May 24, 2018

kubeadm: Improve the kubelet default configuration security-wise #64187

kubeadm: Improve the kubelet default configuration security-wise #64187

Conversation

luxas commented May 23, 2018 • edited Loading

dixudx commented May 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented May 23, 2018

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 23, 2018

fejta-bot commented May 23, 2018

k8s-ci-robot commented May 23, 2018

luxas commented May 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented May 24, 2018

luxas commented May 24, 2018

k8s-github-robot commented May 24, 2018

luxas commented May 23, 2018 •

edited

Loading