AKS using system assigned identities on VMSS to access other Azure resources

[jmos5156]

Hello.

In deploying an AKS cluster in private mode I see that three managed identities are created in my case.

1. MI for the kube dashboard
2. MI for the OMS agent
3. MI for the agent pool

I cannot find any documentation surrounding the usage of these MI's. Is there any available that will describe what the AKS cluster deployment does when adding profiles/nodes to a deployment?

Further to this, we want to grant access to Keyvault from the pods. Under normal IaaS processes, we create a system-assigned identity to Azure VMSS and this is enough to grant implicit access to the pods running on the worker nodes. Is this the correct way of granting this level of permission? If so how can we enable the system assigned identity at deploy time? There doesn't seem to be anything directly related in ARM resource Microsoft.ContainerService/managedCluster.

On deploying the AKS cluster, a managed identity is created (see point 3 above) yet granting this access to other Azure resources does not seem to work for the pods, hence using the system-assigned ones on the vmss's themselves

Many Thanks

[aristosvo]

https://github.com/Azure/aad-pod-identity is AFAIK the solution to explicitly grant access to resources. You can assign different Managed Identities to different (groups of ) pods by labelling them.

See also #1480 for inclusion as an AKS add-on

[jmos5156]

Yes, I agree this seems to be the method employed currently but I have also experienced that without deploying AAD POD and enabling System-Assigned Identity (SI) at the vmss level also seems to allow the pods to gain access to other Azure resources where the SI is granted permissions (ie. KeyVault).

It also seems like the latest API version of the microsoft.containerservice/managedclusters the resource has several Service principal configurations that make for a confusing setup.

We have servicePrincipalProfile and Identity listed in the documentation but limited information about what they are and how they affect the cluster creation. When created there is an agentpool MI but it doesn't seem to be assigned to anything so what should the method be? VMSS SI or AAD POD?

The only benefit of AAD POD is the granularity of permissions created which I guess real difference.

[adhodgson1]

I have developers unintentionally going down the same path as @jmos5156 where they grant the identity at the VMSS level access to external resources and this works for every pod on the cluster. It would be good to know the future of this workaround going forward so I can push people in the right direction.

[jmcshane]

The documentation for the managed identities can be found here: https://docs.microsoft.com/en-us/azure/aks/use-managed-identity

[jmos5156]

I think I've hit another problem. When assigning the identity at VMSS level, and K8 upgrade will recreate the node(s) and this clears the configuration from the scale set. Probably a 'hidden' reason for not using SI and referring back to AAD POD solution.

[github-actions]

Action required from @Azure/aks-pm

[TomGeske]

jmos5156: For KV access you should consider using kubelet user assigned MI or use pod identity

The removal of custom added MIs is a know issue. We are working on fixing it.

#1650

[jmos5156]

Hello,
Pod Identity does not yet work on windows nodes.
Which user is assigned to the kubelet? Is it the one named agentpool? If so granting this user access to KV does not work.

[TomGeske]

Okay, understand.

The one called agentpool should work. Are you using FlexVol or secret store CSI driver to access KV?

Please, share your configuration.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 15 days after stale. jmos5156 feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.