ImdsManagedIdentityOAuthProvider should send resource ID instead of OIDC scope

[sugibuchi]

Describe the bug

The current implementation of ImdsManagedIdentityOAuthProvider (for MSI-based authentication in Azure) tries to get tokens from IMDS endpoint by using the default OIDC scope (resource ID+permission) of Azure storage service (https://storage.azure.com/.default) as query parameter resource.

https://github.com/apache/arrow-rs/blob/master/object_store/src/azure/credential.rs#L53
https://github.com/apache/arrow-rs/blob/master/object_store/src/azure/credential.rs#L418-L428

const AZURE_STORAGE_SCOPE: &str = "https://storage.azure.com/.default";   /// <-- This is a "scope"
...
impl TokenCredential for ImdsManagedIdentityOAuthProvider {
    /// Fetch a token
    async fn fetch_token(
        &self,
        _client: &Client,
        retry: &RetryConfig,
    ) -> Result<TemporaryToken<String>> {
        let mut query_items = vec![
            ("api-version", MSI_API_VERSION),
            ("resource", AZURE_STORAGE_SCOPE),    /// <-- Set "scope" including ".default"
        ];

However, the value of resource must be a resource ID (https://storage.azure.com/) without .default. You can find a C# code example in the following official document.

https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-vm-windows-access-storage#access-data

To Reproduce

Sorry. I cannot directly reproduce this problem since I have no experience in Rust. We identified this problem when we tried to write Delta Lake file by using Python binding of delta-rs which uses Rust object_store.

deltalake.PyDeltaTableError: Failed to load checkpoint: Failed to read checkpoint content: Generic MicrosoftAzure error: Error authorizing request: Error performing token request: response error "adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_resource","error_description":"AADSTS500011: The resource principal named https://storage.azure.com/.default was not found in the tenant named *****. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.\r\nTrace ID: *****\r\nCorrelation ID: *****\r\nTimestamp: 2023-04-17 16:15:01Z","error_codes":[500011],"timestamp":"2023-04-17 16:15:01Z","trace_id":"*****","correlation_id":"*****","error_uri":"https://westeurope.login.microsoft.com/error?code=500011"}
Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=*****&resource=https%3A%2F%2Fstorage.azure.com%2F.default
", after 0 retries: HTTP status client error (403 Forbidden) for url (http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com%2F.default&client_id=*****)

An important part in this error message is the following:

The resource principal named https://storage.azure.com/.default was not found in the tenant named *****.

We can reproduce the same error by sending requests to IMDS endpoint.

# with ".default"
curl -H "Metadata: true" "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=***&resource=https%3A%2F%2Fstorage.azure.com%2F.default"

adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_resource", ...

# without ".default"
curl -H "Metadata: true" "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=***&resource=https%3A%2F%2Fstorage.azure.com%2F"

{"access_token":"...

Expected behavior
ImdsManagedIdentityOAuthProvider sends request to http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&...&resource=https%3A%2F%2Fstorage.azure.com%2F, without .default in query parameter resource.

[tustvold]

Is it possible the error is the result of a permissions restriction in your environment, the use of .default works in my test environment

[sugibuchi]

@tustvold
It might work with .default in some environments (we are using AAD Pod Identity in AKS, which is an emulation of IMDS in Kuberentes cluster. This is probably a reason why we are seeing different results).

But the documentation clearly says that a value of resource should be "App ID URI of the target resource", not scope.

https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token#get-a-token-using-http

Managed Identity credential class in Azure Java SDK accepts resource ID as configuration parameter.

https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/identity/azure-identity/src/main/java/com/azure/identity/ManagedIdentityCredentialBuilder.java#L83

And an equivalent class in Azure Python SDK explicitly drops .default from query parameter values.

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/identity/azure-identity/azure/identity/_internal/managed_identity_client.py#L112
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/identity/azure-identity/azure/identity/_internal/__init__.py#L19-L29

[saryeHaddadi]

I'd like to share my researches in case it helps.
My conclusion is that, in the below HTTP call, one needs to pass in a "resource" instead of a "scope".
This can be done by extracting the ressource from the scope (see _scopes_to_resource() mentioned above).

• https://github.com/apache/arrow-rs/blob/master/object_store/src/azure/credential.rs#L427

How I reached that conclusion.

First, what is the definition of a scope

In OAuth 2.0, scopes and permissions are used interchangeably to define the level of access that a client has to a protected resource. Scopes are used to specify the level of access that a client has to a protected resource, but they do not provide the granularity necessary to define what the client can do with that resource. Permissions are represented as string values and are used by an app to request the permissions it needs by specifying them in the scope query parameter

In other words, scopes are per client app while permissions are per user. For example, one client app can have a scope(s) to access certain API(s), but the users of this client app will have different permissions in this API based on their roles.

Scope examples:

• User.Read.All, Directory.ReadWrite.All
• There are some "well-known" scopes like email, profile
• And (in Azure at least) .default

Sources:
1. learn.microsoft.com
2. permit.io
3. stackoverflow.com
4. stackoverflow.com

What is the .default scope

→ See Microsoft documentation.

In particular, it states how, from a resource, to reference the .default scope.

The scope parameter value is constructed by using the identifier URI for the resource and .default, separated by a forward slash (/). For example, if the resource's identifier URI is https://contoso.com, the scope to request is https://contoso.com/.default.

So I understand that Scopes & Ressources are two different kinds of objects. Now I'd like to confirm how, from a ressource identifier, I can construct a scope. link to doc

The scope parameter is a space-separated list of delegated permissions that the application is requesting. Each permission is indicated by appending the permission value to the resource's identifier (the application ID URI).

Examples 1

var scopes = new [] { ResourceId+"/user_impersonation"};

Example 2

GET https://login.microsoftonline.com/common/oauth2/v2.0/authorize?
client_id=6731de76-14a6-49ae-97bc-6eba6914391e
&response_type=code
&redirect_uri=http%3A%2F%2Flocalhost%2Fmyapp%2F
&response_mode=query
&scope=
https%3A%2F%2Fgraph.microsoft.com%2Fcalendars.read%20 <---- '%20' : space separated-list
https%3A%2F%2Fgraph.microsoft.com%2Fmail.send
&state=12345

When using a Managed-Identity, how to get an Access Token

An App is deployed on a VM. And an Identity (= a service-principal) is given to the VM. The App doesn't need to have its own service-principale, but can ask the get authorized with the same rights than the VM. For that, the App needs to query the Azure Instance Metadata Service (IMDS).

IMDS is a REST API that's available at a well-known, non-routable IP address (169.254.169.254). You can only access it from within the VM.

This IMDS service exposes a number of APIs, among them the /identity/oauth2/token API. As per the documentation, this API accepts a ressource parameter, but does not accept a scope as a parameter.
See Swagger spec for IMDS API, version 2019-08-01.

• imds.json L109-114: "This is the urlencoded identifier URI of the sink resource for the requested Azure AD token." => Not a scope.
• examples/GetIdentityToken.json: The ressource parameter is not given a scope, but a ressource.

Related Readings

1. learn.microsoft.co - Instance Metadata Service
2. learn.microsoft.co - Get a Token using HTTP

Also, from the source code, this call passes a scope value to a scope argument.

While that call, passes a scope value to a resource argument, here is the issue.

[tustvold]

Thank you for your investigation, this makes sense to me and I would be happy to review a fix.

[tustvold]

label_issue.py automatically added labels {'object-store'} from #4193