[BUG] Not able to retrieve aks pod identity, identity not found

[ekersale]

Describe the bug
When the .Net core app try to get the token on an aks pod we get the following exception:

Headers:
Server: IMDS/150.870.65.512
Date: Wed, 11 Aug 2021 22:05:29 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 68

   at Azure.Identity.ImdsManagedIdentitySource.CreateRequest(String[] scopes)
   at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #2) Azure.Identity.CredentialUnavailableException: Operating system Linux 5.4.0-1055-azure #57~18.04.1-Ubuntu SMP Fri Jul 16 19:40:19 UTC 2021 isn't supported.
   at Azure.Identity.VisualStudioCredential.GetTokenProviderPath()
   at Azure.Identity.VisualStudioCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.VisualStudioCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.VisualStudioCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #3) Azure.Identity.CredentialUnavailableException: Stored credentials not found. Need to authenticate user in VSCode Azure Account.
 ---> System.DllNotFoundException: Unable to load shared library 'libsecret-1.so.0' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibsecret-1.so.0: No such file or directory
   at Azure.Identity.LinuxNativeMethods.Imports.secret_schema_new(String name, Int32 flags, String attribute1, Int32 attribute1Type, String attribute2, Int32 attribute2Type, IntPtr end)
   at Azure.Identity.LinuxNativeMethods.secret_schema_new(String name, SecretSchemaFlags flags, String attribute1, SecretSchemaAttributeType attribute1Type, String attribute2, SecretSchemaAttributeType attribute2Type)
   at Azure.Identity.LinuxVisualStudioCodeAdapter.GetLibsecretSchema()
   at Azure.Identity.LinuxVisualStudioCodeAdapter.GetCredentials(String serviceName, String accountName)
   at Azure.Identity.VisualStudioCodeCredential.GetStoredCredentials(String environmentName)
   --- End of inner exception stack trace ---
   at Azure.Identity.VisualStudioCodeCredential.GetStoredCredentials(String environmentName)
   at Azure.Identity.VisualStudioCodeCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.VisualStudioCodeCredential.GetTokenImplAsync(TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.VisualStudioCodeCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #4) Azure.Identity.CredentialUnavailableException: Azure CLI not installed
   at Azure.Identity.AzureCliCredential.RequestCliAccessTokenAsync(Boolean async, String[] scopes, CancellationToken cancellationToken)
   at Azure.Identity.AzureCliCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.AzureCliCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.AzureCliCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

 ---> (Inner Exception #5) Azure.Identity.CredentialUnavailableException: PowerShell is not installed.
   at Azure.Identity.AzurePowerShellCredential.RequestAzurePowerShellAccessTokenAsync(Boolean async, String[] scopes, CancellationToken cancellationToken)
   at Azure.Identity.AzurePowerShellCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.AzurePowerShellCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.AzurePowerShellCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)<---

   --- End of inner exception stack trace ---
   at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
   at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at QuartzX.Services.Core.Services.CredentialService.GetTokenAsync(String[] scopes, String azureManagedIdentity, CancellationToken cancellationToken)
   at QuartzX.Services.Core.Sql.BaseDapperService`1.GetAccessTokenAsync()
   at QuartzX.Services.Core.Sql.BaseEntityDapperService`1.<>c__DisplayClass2_0`1.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.AsyncPolicy.<>c__DisplayClass40_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at QuartzX.Services.Core.Sql.BaseEntityDapperService`1.WithConnectionAsync[T](Func`2 callSql)
   at QuartzX.Lookup.Common.Service.LookupService.GetAllAsync() in F:\agent01\_work\2\s\QuartzX.Lookup.Common.Service\QuartzX.Sql.Core.Generators\QuartzX.Sql.Core.Generators.SqlClassGenerator\LookupService.g.procsql.cs:line 111
   at QuartzX.Lookups.ApiServer.Lookups.Controllers.LookupController.GetLookupsAsync() in F:\agent01\_work\2\s\QuartzX.Lookups.ApiServer\Controllers\Lookups\LookupController.cs:line 31
   at lambda_method88(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at QuartzX.WebApi.Common.Api.Helpers.Exceptions.ExceptionMiddleware.InvokeAsync(HttpContext httpContext)
[22:51:48 Microsoft.AspNetCore.Server.Kestrel [Error] Connection id ""0HMAT2F8D2DD2"", Request id ""0HMAT2F8D2DD2:00000003"": An unhandled exception was thrown by the application.
Azure.Identity.CredentialUnavailableException: DefaultAzureCredential failed to retrieve a token from the included credentials.
- EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
- ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource.
Status: 400 (Bad Request)

Content:
{"error":"invalid_request","error_description":"Identity not found"}

But on the same pod when we execute the following code:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://database.windows.net/' -H Metadata:true

We got a correct answer with a valid token:

Also on our aad-pod-identity nmi the previous call generate the following entry:

But when we use the .Net Core implementation nothing react there.

Expected behavior
GetAccessToken() to be able to retrieve the token correctly.

To Reproduce
Use this implementation in an aks pod with aad-pod-identity handling pod identities.

public class CredentialService
    {
        private TokenCredential _tokenCredential;

        public CredentialService()
        {
        }

        public static TokenCredential GetNewCredential(string azureManagedIdentity)
        {
            if (!string.IsNullOrEmpty(azureManagedIdentity))
            {
                return new DefaultAzureCredential(new DefaultAzureCredentialOptions { ManagedIdentityClientId = azureManagedIdentity });
            }

            return new DefaultAzureCredential();
        }

        public async Task GetTokenAsync(string[] scopes, string azureManagedIdentity, CancellationToken cancellationToken)
        {
            if (_tokenCredential == null)
            {
                _tokenCredential = GetNewCredential(azureManagedIdentity);
            }

            var accessToken = await _tokenCredential.GetTokenAsync(
                new TokenRequestContext(scopes), 
                cancellationToken);

            return accessToken.Token;
        }
    }

Environment:

• Name and version of the Library package used: Azure.Identity v1.4
• Hosting platform or OS and .NET runtime version: AKS Pod running docker 5.0-alpine with .Net Core 5.0

[jsquire]

Thank you for your feedback. Tagging and routing to the team members best able to assist.

[christothes]

Hi @ekersale - does this work after the pod has been up for a minute or so? The IMDS endpoint is usually not available immediately, and can cause behavior like this.

[ekersale]

Hi @christothes, thank's for the reply but the code has been on our environment during a few days. So it can't come from that.

[christothes]

There must be something in your environment configured for Managed Identity or a process listening on the same endpoint - the ManagedIdentityCredential only attempts to connect to an endpoint if one of those environment variables is set, or the local only IMDS endpoint ip address (169.254. 169.254) has something listening on it.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!