Skip to content
This repository has been archived by the owner on Oct 24, 2023. It is now read-only.

Commit

Permalink
fix addon spec and tests
Browse files Browse the repository at this point in the history
  • Loading branch information
aramase committed Mar 19, 2020
1 parent 4bb8eb4 commit ce2c837
Show file tree
Hide file tree
Showing 15 changed files with 404 additions and 63 deletions.
3 changes: 2 additions & 1 deletion docs/topics/clusterdefinitions.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ $ aks-engine get-versions
| container-monitoring | false | 1 | Delivers the Kubernetes container monitoring component |
| [blobfuse-flexvolume](https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/blobfuse) | true | as many as linux agent nodes | Access virtual filesystem backed by the Azure Blob storage |
| [smb-flexvolume](https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/smb) | false | as many as linux agent nodes | Access SMB server by using CIFS/SMB protocol |
| [keyvault-flexvolume](../../examples/addons/keyvault-flexvolume/README.md) | true | as many as linux agent nodes | Access secrets, keys, and certs in Azure Key Vault from pods |
| [keyvault-flexvolume](../../examples/addons/keyvault-flexvolume/README.md) | true | as many as linux agent nodes | Access secrets, keys, and certs in Azure Key Vault from pods. This solution will be **DEPRECATED** in favor of csi-secrets-store addon |
| [aad-pod-identity](../../examples/addons/aad-pod-identity/README.md) | false | 1 + 1 on each linux agent nodes | Assign Azure Active Directory Identities to Kubernetes applications |
| [scheduled-maintenance](https://github.com/awesomenix/drainsafe) | false | 1 + 1 on each linux agent nodes | Cordon and drain node during planned/unplanned [azure maintenance](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/scheduled-events) |
| [azuredisk-csi-driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver) | true if using a Kubernetes cluster (v1.13+) with `useCloudControllerManager` enabled | 1 + 1 on each linux agent nodes | Allows Kubernetes to use [Azure Disk](https://azure.microsoft.com/en-us/services/storage/disks/) volume |
Expand All @@ -121,6 +121,7 @@ $ aks-engine get-versions
| aad | true if adminGroupID is specified in the aadProfile configuration | 0 | ClusterRoleBinding specification that adds an admin group matching the adminGroupID |
| [cilium](https://docs.cilium.io/en/v1.4/kubernetes/policy/#ciliumnetworkpolicy) | true if networkPolicy is "cilium"; currently validated against Kubernetes v1.13, v1.14, and v1.15 | 0 | A NetworkPolicy CRD implementation by the Cilium project (currently supports v1.4) |
| [flannel](https://coreos.com/flannel/docs/0.8.0/index.html) | false | 0 | An addon that delivers flannel: a virtual network that gives a subnet to each host for use with container runtimes. If `networkPlugin` is set to `"flannel"` this addon will be enabled automatically. Not compatible with any other `networkPlugin` or `networkPolicy`. |
| [csi-secrets-store](../../examples/addons/csi-secrets-store/README.md) | true | as many as linux agent nodes | Integrates secrets stores (Azure keyvault) via a [Container Storage Interface (CSI)](https://kubernetes-csi.github.io/docs/) volume. |

To give a bit more info on the `addons` property: We've tried to expose the basic bits of data that allow useful configuration of these cluster features. Here are some example usage patterns that will unpack what `addons` provide:

Expand Down
84 changes: 84 additions & 0 deletions examples/addons/csi-secrets-store/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# CSI Secrets Store Add-on

[The CSI Secrets Store](https://github.com/kubernetes-sigs/secrets-store-csi-driver) integrates secrets stores with Kubernetes via a [Container Storage Interface (CSI)](https://kubernetes-csi.github.io/docs/) volume.

With the CSI secrets store and the azure provider installed, developers can access application-specific secrets, keys, and certs stored in Azure Key Vault directly from their pods.

This addon is available for 1.15+ clusters

**Note** This addon is enabled by default for 1.16+ clusters

To enable on 1.15 cluster, add this addon to the apimodel and also set the feature gate `CSIInlineVolume=true` for API server and kubelet config

```json
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.15",
"kubernetesConfig": {
"addons": [
{
"name": "csi-secrets-store",
"enabled": true
}
],
"apiServerConfig": {
"--feature-gates": "CSIInlineVolume=true"
},
"kubeletConfig": {
"--feature-gates": "CSIInlineVolume=true"
}
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_DS2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool",
"count": 3,
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}
}
}

```

To validate the add-on is running as expected, run the following commands:

You should see the csi-secrets-store driver pods and azure provider pods running on each agent node:

```bash
kubectl get pods -n kube-system

csi-secrets-store-4vmbw 3/3 Running 0 43m
csi-secrets-store-kmjcr 3/3 Running 0 50m
csi-secrets-store-provider-azure-7ldqq 1/1 Running 0 43m
csi-secrets-store-provider-azure-h5xmh 1/1 Running 0 50m
```

Follow the README at https://github.com/Azure/secrets-store-csi-driver-provider-azure for get started steps.

## Supported Orchestrators

Kubernetes
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.16",
"kubernetesConfig": {
"addons": [
{
"name": "csi-secrets-store",
"enabled": true
}
]
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_DS2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool",
"count": 3,
"vmSize": "Standard_DS2_v2",
"availabilityProfile": "VirtualMachineScaleSets"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
},
"servicePrincipalProfile": {
"clientId": "",
"secret": ""
}
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,45 @@ apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: secrets-store.csi.k8s.io
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
podInfoOnMount: true
attachRequired: false
{{- if ContainerConfig "volume-lifecycle-mode"}}
volumeLifecycleModes:
- Ephemeral
{{ end }}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: secrets-store-csi-driver
namespace: default
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: secretproviderclasses-rolebinding
labels:
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: secretproviderclasses-role
subjects:
- kind: ServiceAccount
name: secrets-store-csi-driver
namespace: default
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secretproviderclasses-role
labels:
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
- secrets-store.csi.x-k8s.io
Expand All @@ -44,6 +54,8 @@ apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: secretproviderclasses.secrets-store.csi.x-k8s.io
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
group: secrets-store.csi.x-k8s.io
names:
Expand Down Expand Up @@ -101,6 +113,9 @@ kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-secrets-store
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
Expand Down Expand Up @@ -140,8 +155,15 @@ spec:
mountPath: /csi
- name: registration-dir
mountPath: /registration
resources:
limits:
cpu: {{ContainerCPULimits "csi-node-driver-registrar"}}
memory: {{ContainerMemLimits "csi-node-driver-registrar"}}
requests:
cpu: {{ContainerCPUReqs "csi-node-driver-registrar"}}
memory: {{ContainerMemReqs "csi-node-driver-registrar"}}
- name: secrets-store
image: {{ContainerImage "csi-secrets-store"}}
image: {{ContainerImage "secrets-store"}}
args:
- "--debug=true"
- "--endpoint=$(CSI_ENDPOINT)"
Expand Down Expand Up @@ -178,6 +200,13 @@ spec:
mountPropagation: Bidirectional
- name: providers-dir
mountPath: /etc/kubernetes/secrets-store-csi-providers
resources:
limits:
cpu: {{ContainerCPULimits "secrets-store"}}
memory: {{ContainerMemLimits "secrets-store"}}
requests:
cpu: {{ContainerCPUReqs "secrets-store"}}
memory: {{ContainerMemReqs "secrets-store"}}
- name: liveness-probe
image: {{ContainerImage "livenessprobe"}}
imagePullPolicy: Always
Expand All @@ -188,6 +217,13 @@ spec:
volumeMounts:
- name: plugin-dir
mountPath: /csi
resources:
limits:
cpu: {{ContainerCPULimits "livenessprobe"}}
memory: {{ContainerMemLimits "livenessprobe"}}
requests:
cpu: {{ContainerCPUReqs "livenessprobe"}}
memory: {{ContainerMemReqs "livenessprobe"}}
volumes:
- name: mountpoint-dir
hostPath:
Expand All @@ -209,9 +245,11 @@ spec:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: csi-secrets-store-provider-azure
namespace: kube-system
labels:
app: csi-secrets-store-provider-azure
name: csi-secrets-store-provider-azure
addonmanager.kubernetes.io/mode: Reconcile
spec:
updateStrategy:
type: RollingUpdate
Expand All @@ -226,22 +264,21 @@ spec:
tolerations:
containers:
- name: provider-azure-installer
image: {{ContainerImage "csi-secrets-store-provider-azure"}}
image: {{ContainerImage "provider-azure-installer"}}
imagePullPolicy: Always
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
cpu: 50m
memory: 100Mi
env:
# set TARGET_DIR env var and mount the same directory to to the container
- name: TARGET_DIR
value: "/etc/kubernetes/secrets-store-csi-providers"
volumeMounts:
- mountPath: "/etc/kubernetes/secrets-store-csi-providers"
name: providervol
resources:
limits:
cpu: {{ContainerCPULimits "provider-azure-installer"}}
memory: {{ContainerMemLimits "provider-azure-installer"}}
requests:
cpu: {{ContainerCPUReqs "provider-azure-installer"}}
memory: {{ContainerMemReqs "provider-azure-installer"}}
volumes:
- name: providervol
hostPath:
Expand Down
48 changes: 38 additions & 10 deletions pkg/api/addons.go
Original file line number Diff line number Diff line change
Expand Up @@ -773,28 +773,49 @@ func (cs *ContainerService) setAddonsConfig(isUpgrade bool) {
}

defaultSecretsStoreCSIDriverAddonsConfig := KubernetesAddon{
Name: common.SecretStoreCSIDriverAddonName,
Enabled: to.BoolPtr(DefaultSecretStoreCSIDriverAddonEnabled && common.IsKubernetesVersionGe(o.OrchestratorVersion, "1.15.0")),
Name: common.SecretsStoreCSIDriverAddonName,
Enabled: to.BoolPtr(DefaultSecretStoreCSIDriverAddonEnabled && common.IsKubernetesVersionGe(o.OrchestratorVersion, "1.16.0")),
Containers: []KubernetesContainerSpec{
{
Name: "livenessprobe",
Image: "quay.io/k8scsi/livenessprobe:v1.1.0",
Name: common.CSILivenessProbeContainerName,
Image: specConfig.MCRKubernetesImageBase + k8sComponents[common.CSILivenessProbeContainerName],
CPURequests: "10m",
MemoryRequests: "20Mi",
CPULimits: "200m",
MemoryLimits: "200Mi",
},
{
Name: "csi-node-driver-registrar",
Image: "quay.io/k8scsi/csi-node-driver-registrar:v1.2.0",
Name: common.CSINodeDriverRegistrarContainerName,
Image: specConfig.MCRKubernetesImageBase + k8sComponents[common.CSINodeDriverRegistrarContainerName],
CPURequests: "10m",
MemoryRequests: "20Mi",
CPULimits: "200m",
MemoryLimits: "200Mi",
},
{
Name: "csi-secrets-store",
Image: "docker.io/deislabs/secrets-store-csi:v0.0.8",
Name: common.CSISecretsStoreDriverContainerName,
Image: "docker.io/deislabs/secrets-store-csi:v0.0.8",
CPURequests: "50m",
MemoryRequests: "100Mi",
CPULimits: "200m",
MemoryLimits: "200Mi",
},
{
Name: "csi-secrets-store-provider-azure",
Image: "mcr.microsoft.com/k8s/csi/secrets-store/provider-azure:0.0.3",
Name: common.CSISecretsStoreProviderAzureContainerName,
Image: specConfig.MCRKubernetesImageBase + k8sComponents[common.CSISecretsStoreProviderAzureContainerName],
CPURequests: "50m",
MemoryRequests: "100Mi",
CPULimits: "200m",
MemoryLimits: "200Mi",
},
},
}

// volume lifecycle mode needs to be explicitly set for 1.16+
if common.IsKubernetesVersionGe(o.OrchestratorVersion, "1.16.0") {
defaultSecretsStoreCSIDriverAddonsConfig.Config = map[string]string{"volume-lifecycle-mode": "Ephemeral"}
}

// Allow folks to simply enable kube-dns at cluster creation time without also requiring that coredns be explicitly disabled
if !isUpgrade && o.KubernetesConfig.IsAddonEnabled(common.KubeDNSAddonName) {
defaultCorednsAddonsConfig.Enabled = to.BoolPtr(false)
Expand Down Expand Up @@ -988,6 +1009,13 @@ func (cs *ContainerService) setAddonsConfig(isUpgrade bool) {
o.KubernetesConfig.Addons[i].Enabled = to.BoolPtr(true)
}
}

// Enable keyvault flexvolume addon during upgrade for 1.16 or greater scenarios, unless explicitly disabled
if isUpgrade && common.IsKubernetesVersionGe(o.OrchestratorVersion, "1.16.0") && !o.KubernetesConfig.IsAddonDisabled(common.KeyVaultFlexVolumeAddonName) {
if i := getAddonsIndexByName(o.KubernetesConfig.Addons, common.KeyVaultFlexVolumeAddonName); i > -1 {
o.KubernetesConfig.Addons[i].Enabled = to.BoolPtr(true)
}
}
}

func appendAddonIfNotPresent(addons []KubernetesAddon, addon KubernetesAddon) []KubernetesAddon {
Expand Down
Loading

0 comments on commit ce2c837

Please sign in to comment.