object_store: Support for Azure Microsoft Entra for authentication from AKS

[gianarb]

Which part is this question about

object_store/azure

Describe your question

I use the method described here to authenticate containers running in Azure AKS:

https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet

In practice I have a service account that looks like this:

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: afgd
    azure.workload.identity/tenant-id: 44hb
  creationTimestamp: "2023-11-22T11:47:33Z"
  labels:
    environment: test
  name: blob-access
  namespace: default

And I annotation the pod I want to authenticate as described in the documentation.

It worked out of the box for the containers written in other language (Go) but not for the one I have in Rust that uses datafusion.

This is the way I build the azure client:

MicrosoftAzureBuilder::from_env()
  .with_url(c.account)
  .with_container_name(c.container_name)
  .build()
  .unwrap()

I can't figure out the right way to make this authentication method to work
Additional context

The error I get looks like this one #4096

[gianarb]

I realized I was not using the latest datafusion/arrow-rs so I updated to arrow 0.48 and datafusion 0.33 with object_store 0.7.0.

And now I get a different error

Server failed with Object Store error: Generic MicrosoftAzure error: Error performing list request: response error "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.

My feeling is that it is not getting the credentials from the service account now. Because I am pretty sure I am using the same user I use for the other application with access to this blob store container

[tustvold]

It looks like this should be supported by https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html.

You will need to specify the following:

• https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_tenant_id
• https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_client_id
• https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_federated_token_file

Or alternatively these can be specified as environment variables:

• AZURE_TENANT_ID
• AZURE_CLIENT_ID
• AZURE_FEDERATED_TOKEN_FILE

These values should correspond to the service account annotations - https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet#service-account-annotations

There might be a nicer way to project these in, AWS EKS has an admission hook that automatically sets some environment variables for its similar WebIdentityProvider flow, but setting them manually may be good enough to prove it works first

Edit: The docs do indeed suggest AKS has an admission webhook that adds environment variables, although it doesn't specify what they are ofc 😆

Edit edit: https://github.com/Azure/azure-sdk-for-rust/blob/main/sdk/identity/src/token_credentials/environment_credentials.rs#L23 would suggest that the above environment variables should be working already...

[gianarb]

Thank you for your explanation! I am not currently setting the federated token file. So I need to figure out how to get it. I see:

azure.workload.identity/inject-proxy-sidecar

Injects a proxy init container and proxy sidecar into the pod. The proxy sidecar is used to intercept token requests to IMDS and acquire a Microsoft Entra token on behalf of the user with federated identity credential.

And yes it reads those environment variables since I am using the from_env(), ok I need to figure out what I am missing

[tustvold]

Closing this one as I believe the question has been answered, feel free to reopen / file follow on tickets as required