[BUG] Service Bus trigger using identity-based connection uses incorrect user identity when run locally

[ashtmMSFT]

Library name and version

Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.7.0

Describe the bug

I had a lot of trouble getting a managed identity to work correctly locally for a C# Azure Function with a Service Bus Queue-triggered binding on my devbox with VS Code. The problem seems to stem from the fact that the user identity being picked up at runtime is incorrect.

Context -- Local Tooling

• Visual Studio Code -- code --version == 1.70.2
• Visual Studio Code extensions Azure Account and Azure Functions are installed.
• dotnet --version == 6.0.400
• .csproj Packages: Microsoft.NET.Sdk.Functions 4.1.1, Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.7.0

Context -- Cloud

• I have two users which belong to separate tenants: user A who is homed to tenant A, and user B who is homed to tenant B.
• Tenant B contains a Service Bus namespace with a Queue.
• User B is assigned appropriate RBAC roles to read and write from the Queue.

Context -- Local Dev Workflow

• I am logged into Windows as user A.
• From Terminal, I perform az account show, which shows me I am logged in as user A (associated with tenant A).
• From Terminal, I do an az login and login as user B.
• From Terminal, I perform az account show, which shows me I am now logged in as user B (associated with tenant B).
• From this Terminal, I launch VS Code (code .) in the folder with my Azure Functions project containing a Service Bus Queue-triggered function.
• Inside VS code, I login to the Azure Account extension with user B. I confirm this login succeeded: I see user B's email address in the bottom status bar AND see user B's Azure subscriptions/resources in the Azure extension blade.
• In my local.settings.json file, I specify the ServiceBusConnection__fullyQualifiedNamespace setting as the service bus namespace's hostname.
• In my local.settings.json file, I specify the ServiceBusConnection__tenantId setting as the id for tenant B.
• I start debugging the project from VS Code.

The Error / Actual Behavior

When I run the code, I see this error spammed in the VS Code terminal:

System.Private.CoreLib: Put token failed. status-code: 401, status-description: InvalidIssuer: Token issuer is invalid. TrackingId:d4148e00-3538-4ddc-b199-7a39f897138f, SystemTracker:NoSystemTracker, Timestamp:2022-09-06T17:11:15.

The error message is indicative of the fact that the auth token presented to the Service Bus in tenant B was issued by tenant A.

In short, the ServiceBusConnection__tenantId setting does not seem to work.

The Fix / Expected Behavior

Instead, what did work, and what I only found thanks to this seemingly incorrectly closed Github issue, was specifying the id for tenant B in AZURE_TENANT_ID in local.settings.json.

After specifying this setting, the error went away, and the function was triggered by messages picked up from the queue.

It's not clear to me why this setting is needed, because if the user context is already being picked up from the VS Code extension OR the login done with Azure CLI, the proper tenant for that user should already be known.

At the very least, the documentation should be updated to reflect the need to specify the AZURE_TENANT_ID setting. If this approach is taken, please revise these documents:

• Azure Service Bus trigger for Azure Functions | Identity-based connections
• Azure Functions developer guide | Local development with identity-based connections

Expected behavior

No error is seen and the function can be triggered by messages picked up from the queue.

Actual behavior

When I run the project that hosts the function, I see this error spammed in the VS Code terminal:

System.Private.CoreLib: Put token failed. status-code: 401, status-description: InvalidIssuer: Token issuer is invalid. TrackingId:d4148e00-3538-4ddc-b199-7a39f897138f, SystemTracker:NoSystemTracker, Timestamp:2022-09-06T17:11:15.

As best I can tell, this error message is indicative of the fact that the auth token presented to the Service Bus in tenant B was issued by tenant A.

In short, the ServiceBusConnection__tenantId setting does not seem to work.

Reproduction Steps

With a setup as described in the bug description, run the project to start the functions host with a SB queue-triggered function like the one below. Wait until you see the error spam.

public class MyTestClass
    {
        [FunctionName("MyTestFunction")]

        // ServiceBusConnection resolves to ServiceBusConnection__fullyQualifiedNamespace
        // which is referenced within the local.settings.json file to use the managed identity
        public void Run([ServiceBusTrigger("%MyQueueName%", Connection = "ServiceBusConnection")]string command, ILogger log)
        {
            log.LogInformation($"Function triggered with command: {command}");
        }       
    }

Environment

• Visual Studio Code -- code --version == 1.70.2
• Visual Studio Code extensions Azure Account and Azure Functions are installed.
• dotnet --version == 6.0.400
• .csproj Packages: Microsoft.NET.Sdk.Functions 4.1.1, Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.7.0

[JoshLove-msft]

Hi @ashtmMSFT,
Setting only a tenantId is not a supported configuration for the Functions triggers when using Identity. If you set the tenantId, clientId, and clientSecret then the trigger will create a ClientSecretCredential using these values.
If you only set the tenantId, the trigger will end up falling back to the DefaultAzureCredential. This is why you are getting the behavior that you are looking for when setting the AZURE_TENANT_ID environment variable, as the DefaultAzureCredential respects the value from this environment variable. If all you want is for the tenant ID to be respected, you can simply set this in your app settings for a deployed app and in an environment variable for local dev.

I would agree that the docs should be improved to specifically mention which settings map to which credential types and link to the docs for these credential types. We should also mention that if you don't specify any settings beyond the Uri (or only specify an incomplete set such as just the tenantId), we will fall back to DefaultAzureCredential. We should also link to the DefaultAzureCredential docs which mention the different supported environment variables.

[ghost]

Hi @ashtmMSFT. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @ashtmMSFT. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ashtmMSFT]

/unresolve

Folks, I do greatly appreciate the response to the thread, but I don't think the issue is resolved. I would consider the issue resolved after discussion has concluded and at a minimum the documentation is updated. I linked the most relevant articles which would need updates in my original post. Please let me know if you have a different criteria for the issue being resolved.

I would like to minimize the chance of another dev hitting this issue with identity-based connections, as it was the biggest pain point by far in seamlessly developing and sharing a small Functions proof of concept with my customer. We should make it as easy as possible for devs to build solutions with a workload identity story that doesn't introduce secrets.

[ashtmMSFT]

Hi @JoshLove-msft, thank you for the prompt reply.

If what you are saying is true, that we are falling back to using a DefaultAzureCredential which first tries to use an EnvironmentCredential and recognize the AZURE_* environment variables, then my code should work if I specify AZURE_TENANT_ID, AZURE_USERNAME, and AZURE_PASSWORD values accordingly for user B in my local.settings.json.

Unfortunately, this isn't the case:

Revised Local Context -- Still Broken with AZURE_* Settings

• Inside a VS code terminal, I do an az login and login as user A. (Theoretically, this AZ CLI context for user A should be ignored because environment variables are provided below.)
• Inside VS code, I login to the Azure Account extension with user B. (This context is also ignored.)
• In my local.settings.json file, I specify the AZURE_TENANT_ID setting as the id for tenant B.
• In my local.settings.json file, I specify the AZURE_USERNAME setting as user B's email address.
• In my local.settings.json file, I specify the AZURE_PASSWORD setting as user B's password.
• I start debugging the project from VS Code.

I see the same error about an invalid token issuer.

===

Furthermore, if DefaultAzureCredential is being used and I were to specify no environment variables, shouldn't the identity context be picked up from either the user identity I used to login to VS Code (via the Azure Account extension) OR from AZ CLI (via my az login of user B)?

Revised Local Context -- Still Broken with VSCode or AZ CLI User Context

• Inside a VS code terminal, I do an az login and login as user B.
• Inside VS code, I login to the Azure Account extension with user B.
• In my local.settings.json file, I don't specify any AZURE_* settings.
• I start debugging the project from VS Code.

I see the same error about an invalid token issuer.

===

The Only Workaround That Does The Trick
The only way I have been able to get this to work is by doing an az login as user B and specifying the AZURE_TENANT_ID setting with user B's tenantId in my local.settings.json.

To reinforce an earlier point I made:

It's not clear to me why this [AZURE_TENANT_ID] setting is needed, because if the user context is already being picked up from the VS Code extension OR the login done with Azure CLI, the proper tenant for that user should already be known.

Am I misunderstanding something here? Does this imply a bug with DefaultAzureCredential?

[JoshLove-msft]

Do you have logging enabled in your function app via your local.settings.json file? This would help us determine which credential Identity is using.

[ashtmMSFT]

I was able to get additional logs by adding the "AzureFunctionsJobHost__logging__logLevel__Default": "Debug" key/value pair to my local.settings.json.

I re-ran all the scenarios described above and was able to determine how the usage of DefaultAzureCredential (and all the underlying chained credentials) failed to match my expectations for its behavior. Rather than share logs and the exact type of credential ultimately used in each scenario above, I'll just highlight the points of confusion.

The DefaultAzureCredential attempted to use the following credentials in this order on my machine: EnvironmentCredential, ManagedIdentityCredential, VisualStudioCredential, VisualStudioCodeCredential, AzureCliCredential. (Note the SharedTokenCacheCredential was never used despite being mentioned in the documentation.)

EnvironmentCredential
The EnvironmentCredential documentation indicates that a multitude of AZURE_* environment variables can be set to to facilitate authentication to Azure AD. This led me to believe that I could specify AZURE_TENANT_ID, AZURE_USERNAME, and AZURE_PASSWORD to have the DefaultAzureCredential login to a particular tenant as the user I specify. This belief was further supported by the documentaton stating that a UsernamePasswordCredential is used under the covers, and that class is constructed by providing a username, password, and tenantId.

Unfortunately, specifying a (username, password, and tenantId) tuple via environment variables is not supported. If you specify a tenantId, the EnvironmentCredential assumes you are not using a username and password and will throw Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot.

The table in the troubleshooting guide contains all valid tuples of AZURE_* environment variables that can be used for authentication.

I suggest that we update the currently ambiguous EnvironmentCredential documentation to include this list of valid environment variable combinations.

VisualStudioCredential
This credential provided the biggest surprise: even if you aren't running your project in Visual Studio, VisualStudioCredential will still pick up the credentials you used to login to Visual Studio if you have it installed. VisualStudioCredential documentation does say it "enables authentication to Azure Active Directory using data from Visual Studio", but I'd bet good money I'm not the only developer that read that with the implied statement "if you're running your code in Visual Studio."

This meant that most of the time my identity was unknowingly being pulled from VS and I never got to leverage the credentials further down the chain where I was logged in with the appropriate identity.

Interestingly, if the AZURE_TENANT_ID environment variable is set when your code runs and it does not match the tenantId for the user with which you're logged into Visual Studio, VisualStudioCredential will try to authenticate your stored user identity against this specified tenant. In my case, I saw this: Exception: Azure.Identity.CredentialUnavailableException (0x80131500): Process "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\Asal\TokenService\Microsoft.Asal.TokenService.exe" has failed with unexpected error: TS003: Error, TS002: The account '<REDACTED EMAIL OF USER A>' was not found in the tenant '<REDACTED ID OF TENANT B>'.

Triggering this failure was significant in my case because it meant a credential further down the chain (Azure CLI) would be used and succeed.

VisualStudioCodeCredential
The VisualStudioCodeCredential documentation states that it "enables authentication to Azure Active Directory using data from Visual Studio Code," but it doesn't really mention how. It should be updated to clarify that you need to install and login to the Azure Account extension.

Unfortunately, when you figure that out, it flat out doesn't work: Exception: Azure.Identity.CredentialUnavailableException (0x80131500): Stored credentials not found. Need to authenticate user in VSCode Azure Account. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/vscodecredential/troubleshoot

The troubleshooting guide mentions that "it's a known issue that VisualStudioCodeCredential doesn't work with Azure Account extension versions newer than 0.9.11."

Ouch. Great to know, but seeing as how this affects literally every fresh install of that extension, could we please publicize this information more broadly? The VisualStudioCodeCredential documentation and Azure Account extension pages are great candidates.

===

ASK
Please update the *CredentialClass documentation pages to provide more detail and clarity. The DefaultAzureCredential page lists the other credential class types and suggests one "consult the documentation of these credential types for more information on how they attempt authentication", even though little can be learned from them.

[JoshLove-msft]

Assigning to @christothes for his thoughts on what we can do to improve Identity docs.

I will look into clarifying the functions docs.

[christothes]

This is a good one for @scottaddie

[JoshLove-msft]

We've updated the behavior on the Azure Functions extensions side to respect the passed in tenantId, clientId, and resourceId even if we fall back to the DefaultAzureCredential. Removing myself from this issue as I believe all that remains is to improve Identity docs.

[christothes]

Addressed in #31867