cloud provider configuration will be deprecated for kubelet prameters

[jia-jerry]

How to categorize this issue?

/area control-plane
/kind enhancement
/priority normal
/platform alicloud

What would you like to be added:
We need to move provider-id, cloud-provider and enable-controller-attach-detach from parameter to configuration file for kubelet
Why is this needed:

Dec 13 21:32:32 iZgw87uepwv66jorfgr1cmZ kubelet[2841]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 13 21:32:32 iZgw87uepwv66jorfgr1cmZ kubelet[2841]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Dec 13 21:32:32 iZgw87uepwv66jorfgr1cmZ kubelet[2841]: Flag --enable-controller-attach-detach has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.```

[rfranzke]

/assign

[rfranzke]

The --cloud-provider and --enable-controller-attach-detach will no longer be used for Shoots >= 1.23 as part of #426.

However, I was not able to eliminate the --provider-id flag due to some special handling that is going on. There is already a TODO statement:

gardener-extension-provider-alicloud/pkg/webhook/controlplane/ensurer.go

Lines 130 to 135 in c5bc4e7

	// TODO: Figure out how to provide the provider-id via the kubelet config file (as of Kubernetes 1.19 the kubelet config
	// offers a new `providerID` field which can be used, and it's expected that `--provider-id` will be deprecated eventually).
	// Today, the problem is that the provider ID is determined dynamically using the script above, but the kubelet config cannot
	// reference environment variables like it's possible today with the CLI parameters.
	// See https://github.com/kubernetes/kubernetes/pull/90494
	command = extensionswebhook.EnsureStringWithPrefix(command, "--provider-id=", "${PROVIDER_ID}")

It probably makes sense to follow up on this with the Alicloud Kubernetes community as I have no idea how to migrate this to the kubelet's configuration field.

/unassign

@jia-jerry @EmoinLanyu Please follow up with this.

[shaoyongfeng]

With regards to provider-id flag, I did some investigation

1. providerid in node spec will be set by kubelet when initnode and joining cluster, if --provider-id flag is set as kubelet command line parameter
   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go#L374-L376

2. how it works in cloud provider aws?
   2.1 cloud provider aws implements the interface defined in https://github.com/kubernetes/cloud-provider
   2.2 when cloud provider aws reconcile nodes, it will try to get providerid first https://github.com/kubernetes/cloud-provider/blob/master/controllers/node/node_controller.go#L394
   2.3 if providerid is not set kubelet (this is the case for us), it will try to get providerid https://github.com/kubernetes/cloud-provider/blob/0429a85a45b2424c1508ea289fea6d1e8f15d30f/controllers/node/node_controller.go#L540
   2.4 Finally, the call goes to aws provider and aws provider by the provider id by node name
   https://github.com/kubernetes/cloud-provider-aws/blob/master/pkg/providers/v1/aws.go#L1724

3. Why ali doesn't work?
   Unfortunately, Ali provider doesn't follow the interface defined in https://github.com/kubernetes/cloud-provider, but implement by their own.
   At the beginning of reconcile process, ali provider just assumes the providerid is already set https://github.com/kubernetes/cloud-provider-alibaba-cloud/blob/master/pkg/controller/node/node_controller.go#L135-L138

We need to discuss with Ali how to fix this issue in ali provider, otherwise, we need to do some tricky as currently we do in order to remove --provider-id flag
issue reported to Alibaba provider: kubernetes/cloud-provider-alibaba-cloud#278