Add apiServerExtraArgs as optional configuration to docs #7853
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,118 @@ | ||
| # Allow users to configure kube-apiserver flags | ||
|
|
||
| ## Problem Statement | ||
|
|
||
| A customer is currently using OIDC for authenticating the kubernetes service accounts(KSA) and they need some mechanism to configure the kube-apiserver flags for their usecase. The main issue that we are addressing in this document is how we want to allow users to be able to configure these flags. | ||
|
|
||
| ## Overview of Solution | ||
|
|
||
| Allow users to configure the flags by exposing a map in the cluster spec yaml | ||
|
|
||
| **Schema:** | ||
|
|
||
| ```yaml | ||
| apiVersion: anywhere.eks.amazonaws.com/v1alpha1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: mgmt-cluster | ||
| spec: | ||
| ... | ||
| controlPlaneConfiguration: | ||
| ... | ||
| # More control plane components can be added here in the future | ||
| apiServerExtraArgs: | ||
| ... | ||
| "service-account-issuer": "https://{my-service-account-issuer-url}" | ||
| "service-account-jwks-uri": "https://{my-service-account-issuer-url}/openid/v1/jwks" | ||
| "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key" | ||
| "service-account-key-file": "/etc/kubernetes/pki/sa.pub" | ||
| ``` | ||
|
|
||
| **Validations:** | ||
|
|
||
| * Validate that oidc flags are not configured in apiServerExtraArgs if OIDCConfig identity provider is already configured in the spec | ||
| * Validate that the feature flag is enabled for configuring apiServerExtraArgs | ||
|
|
||
| **Pros:** | ||
|
|
||
| * Creates a standard way of exposing any flag for the control plane components | ||
| * Gives more flexibility to the users in terms of validating the flag values for the api-server | ||
|
|
||
| **Cons:** | ||
|
|
||
| * Does not enforce OIDC compliance or any other validations on the allowed values for the flags | ||
|
|
||
| ## Alternate Solutions | ||
|
|
||
| Allow users to configure the flags as a struct field in the cluster spec yaml | ||
|
|
||
| **Schema:** | ||
|
|
||
| ```yaml | ||
| apiVersion: anywhere.eks.amazonaws.com/v1alpha1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: mgmt-cluster | ||
| spec: | ||
| ... | ||
| controlPlaneConfiguration: | ||
| ... | ||
| apiServerConfiguration: | ||
| ... | ||
| serviceAccountIssuer: | ||
| - "https://{my-service-account-issuer-url}" | ||
| serviceAccountJwksUri: "https://{my-service-account-issuer-url}/openid/v1/jwks" | ||
| serviceAccountSigningKeyFile: "/etc/kubernetes/pki/sa.key" | ||
| serviceAccountKeyFile: "/etc/kubernetes/pki/sa.pub" | ||
| ``` | ||
|
|
||
| **Validations:** | ||
|
|
||
| * Validate that both serviceAccountIssuer and serviceAccountJwksUri have same domain and use https scheme | ||
| * Additional set of validations specific to each of the flags | ||
|
|
||
| **Pros:** | ||
|
|
||
| * Fails fast if any of the flags are misconfigured with invalid values | ||
| * Allows enforcing OIDC compliance for the service account flags of the api-server | ||
|
|
||
| **Cons:** | ||
|
|
||
| * Gives less flexibility to the users for configuring the flags in terms of number of validations | ||
| * Does not provide a standard way to configure the flags | ||
| * Difficult to validate each and every flag and debug any issues with apiserver | ||
|
|
||
| ## Implementation Details | ||
|
|
||
| ``` | ||
| apiServerExtraArgs: | ||
| "service-account-issuer": "https://{my-service-account-issuer-url}" | ||
| "service-account-jwks-uri": "https://{my-service-account-issuer-url}/openid/v1/jwks" | ||
| ``` | ||
|
|
||
| https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#apiserver-flags | ||
| These flags will be fetched from the cluster spec and added to the apiServerExtraArgs in the ClusterConfiguration object during create and upgrade operations for generating control plane CAPI spec | ||
|
|
||
| Users need to enable the feature flag `API_SERVER_EXTRA_ARGS_ENABLED=true` to configure the api server flags in the cluster spec. If it's not enabled, then it will throw an error when validating the cluster spec. This is done in order to expose this functionality for now before we determine to support it officially with some more robust validations. | ||
|
|
||
| The `service-account-issuer` flag can be configured for both podIamConfig as well as controlPlaneConfiguration to enable both features. If both are configured, the podIamConfig url will be appended to the controlPlaneConfiguration url. | ||
|
|
||
| If OIDCConfig is specified in the identityProviderRefs within the spec, then oidc flags cannot be configured in the apiServerExtraArgs and the CLI will throw an error. | ||
|
|
||
| ## Documentation | ||
|
|
||
| We would have to add `controlPlaneConfiguration.apiServerExtraArgs` as an optional configuration for the cluster spec in our EKS-A docs | ||
|
|
||
| ## Migration plan for existing flags | ||
|
|
||
| * Phase 1: We can add more flags to the above options and have validations for the existing flags configured in some other fields to make sure that there is no conflict between them and allow only one of them to be configured | ||
| * Phase 2: We can decide on the priority among the existing conflicting fields and if the flags are configured in multiple fields, the one with higher priority will have precedence and will be used in the cluster | ||
| * Phase 3: We can deprecate all the lower priority conflicting fields for the existing flags and have only one standardized way of configuring all the flags | ||
|
|
||
| ## References | ||
|
|
||
| * https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/1393-oidc-discovery | ||
| * https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery | ||
| * https://openid.net/developers/how-connect-works/ | ||
| * https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| --- | ||
| title: "API Server Extra Args" | ||
| linkTitle: "API Server Extra Args" | ||
| weight: 60 | ||
| description: > | ||
| EKS Anywhere cluster yaml specification for Kubernetes API Server Extra Args reference | ||
| --- | ||
|
|
||
sp1999 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## API Server Extra Args support (optional) | ||
|
|
||
| As of EKS Anywhere version v0.20.0, you can pass additional flags to configure the Kubernetes API server in your EKS Anywhere clusters. | ||
|
|
||
| #### Provider support details | ||
| | | vSphere | Bare Metal | Nutanix | CloudStack | Snow | | ||
| |:--------------:|:-------:|:----------:|:-------:|:----------:|:----:| | ||
| | **Supported?** | ✓ | ✓ | ✓ | ✓ | ✓ | | ||
|
|
||
| In order to configure a cluster with API Server extra args, you need to configure your cluster by updating the cluster configuration file to include the details below. The feature flag `API_SERVER_EXTRA_ARGS_ENABLED=true` needs to be set as an environment variable. | ||
|
|
||
| This is a generic template with some example API Server extra args configuration below for reference: | ||
| ```yaml | ||
| apiVersion: anywhere.eks.amazonaws.com/v1alpha1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: my-cluster-name | ||
| spec: | ||
| ... | ||
| controlPlaneConfiguration: | ||
| apiServerExtraArgs: | ||
|
There was a problem hiding this comment. Can these be updated for existing clusters on all of the providers? (ie. by running an |
||
| ... | ||
| disable-admission-plugins: "DefaultStorageClass,DefaultTolerationSeconds" | ||
| enable-admission-plugins: "NamespaceAutoProvision,NamespaceExists" | ||
| ``` | ||
|
|
||
|
There was a problem hiding this comment. It would be nice to show a example here that reflects a real-world customer use case. If this example is useful, I would suggest adding a couple of paragraphs, describing what the example achieves and why someone would do it. In particular, the OIDC use that you describe in the design doc might be a good thing to explain. There was a problem hiding this comment. Updated the template with other API Server options and described some example usecase below |
||
| The above example configures the `disable-admission-plugins` and `enable-admission-plugins` options of the API Server to enable additional admission plugins or disable some of the default ones. You can configure any of the API Server options using the above template. | ||
|
|
||
| ### controlPlaneConfiguration.apiServerExtraArgs (optional) | ||
| Reference the [Kubernetes documentation](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#options) for the list of flags that can be configured for the Kubernetes API server in EKS Anywhere | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to update each of these pages to say that these additional settings can be configured
https://anywhere.eks.amazonaws.com/docs/getting-started/vsphere/vsphere-spec/
https://anywhere.eks.amazonaws.com/docs/getting-started/baremetal/bare-spec/
https://anywhere.eks.amazonaws.com/docs/getting-started/snow/snow-spec/
https://anywhere.eks.amazonaws.com/docs/getting-started/cloudstack/cloud-spec/
https://anywhere.eks.amazonaws.com/docs/getting-started/nutanix/nutanix-spec/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated