[Core] ADAL to MSAL migration

[jiasli]

Close #18944

Description

As ADAL has been deprecated, Azure CLI needs to be migrated from ADAL to MSAL.

How CLI authentication works

During initialization:

             _get_mgmt_service_client 
             /                      \
          1 /    CredentialAdaptor   \ 2
           v                          v
get_login_credential                SDK client

1. The client factory azure.cli.core.commands.client_factory._get_mgmt_service_client calls azure.cli.core._profile.Profile.get_login_credentials to get an instance of CredentialAdaptor.
2. _get_mgmt_service_client feeds the CredentialAdaptor instance into the SDK client.

During invocation:

           Track 1/2 SDK client
                    |
                    v     signed_session/get_token
            CredentialAdaptor
                    |    
                    v     get_token
UserCredential / ServicePrincipalCredential    

CredentialAdaptor

1. When Track 2 SDK client makes an HTTP call, azure.core.pipeline.policies._authentication.BearerTokenCredentialPolicy calls the get_token method on the CredentialAdaptor.
2. When Track 1 SDK client makes an HTTP call, msrest.pipeline.requests.RequestsCredentialsPolicy calls the signed_session method on the CredentialAdaptor, signed_session calls get_token.

UserCredential/ServicePrincipalCredential

Since Azure Identity has been dropped for User and Service Principal authentication, the MSAL integration is implemented in src/azure-cli-core/azure/cli/core/auth/msal_authentication.py.

There are 2 kinds of credentials:

1. UserCredential (inherit from PublicClientApplication)
2. ServicePrincipalCredential (inherit from ConfidentialClientApplication)

They all expose get_token so that they implement azure.core.credentials.TokenCredential.

Cross-tenant authentication

Cross-tenant authentication is implemented by simply adding header x-ms-authorization-auxiliary by _prepare_mgmt_client_kwargs_track2.

There were asks (Azure/azure-sdk-for-python#8313, Azure/azure-sdk-for-python#17764) on Python SDK to implement cross-tenant authentication but it hasn't been implemented.

Previously, CLI considered implementing a custom policy ExternalAuthenticationPolicy to achieve it, but is not implemented either due to its unnecessary complexity. Querying long-running operation result doesn't need auxiliary token, so no need to implement the get_token callback.

BREAKING CHANGES

• az account show/az ad sp create-for-rbac: --sdk-auth is removed ([Profile] Deprecate --sdk-auth #19414).
• ADAL token cache and service principal secrets are no longer written to ~/.azure/accessTokens.json

Changes on following objects are not considered breaking changes, but they may be used by end users or libraries:

• azure.cli.core.adal_authentication is removed, including
  • AdalAuthentication
    • _token_retriever
    • _external_tenant_token_retriever

If you depend on them, consider using public methods:

• azure.cli.core._profile.Profile.get_raw_token
• azure.cli.core._profile.Profile.get_login_credentials

Testing Guide

Test for different account types

• User
  • auth code flow
  • device code flow
  • username password flow
• Service Principal
  • secret
  • certificate
  • client assertion (federated token)
• Managed Identity
  • clientId
  • objectId
  • resourceId

Test for different SDK scenarios

• mgmt plane SDK client
  • resource
  • storage account
  • vm
  • network
  • ...
• data plane SDK client
  • storage: blob, table, ...
  • keyvault: keys, secrets, ...
  • ...
• cross-tenant operations
  • resource: template deployment
  • network: vnet peering

Test for different clouds

• AzureCloud
• Sovereign clouds
• Azure Stack
  • AAD authentication
  • ADFS authentication

References

• {Identity} Identity new features: {Identity} Identity new features #14690
• Old PRs related to Azure Identity and MSAL integration: https://github.com/Azure/azure-cli/pulls?q=is%3Apr+label%3AMSAL
• ADAL to MSAL migration guide for Python: https://docs.microsoft.com/en-us/azure/active-directory/develop/migrate-python-adal-msal

[jiasli]

Credential location

If you are curious, the token cache and service principal entries are saved under ~/.azure folder:

	encrypted	plaintext
MSAL token cache	msal_token_cache.bin	msal_token_cache.json
Service principal entries	service_principal_entries.bin	service_principal_entries.json

⚠ They are private to Azure CLI. Please don’t decrypt or read from them in any way.

We don’t use Azure Identity’s shared token cache ~\AppData\Local\.IdentityService\msal.cache anymore, because we have dropped Azure Identity.

Supported API

If you are an Azure CLI command module or extension, you may

1. Call azure.cli.core._profile.Profile.get_raw_token:

   azure-cli/src/azure-cli-core/azure/cli/core/_profile.py

   Line 350 in 6208421

   def get_raw_token(self, resource=None, scopes=None, subscription=None, tenant=None):

2. (The Azure SDK Track 2 approach) Call azure.cli.core._profile.Profile.get_login_credentials:

   azure-cli/src/azure-cli-core/azure/cli/core/_profile.py

   Line 300 in 6208421

   def get_login_credentials(self, resource=None, client_id=None, subscription_id=None, aux_subscriptions=None,

   This method returns an CredentialAdaptor in the first element of the result tuple. The CredentialAdaptor instance exposes get_token method. See the previous discussion at Can't resolve Keyvault reference in App Config with managed identity #15805 (comment).

If you are another application or SDK, please follow https://docs.microsoft.com/en-us/cli/azure/msal-based-azure-cli

[jiasli]

Why Azure Identity is dropped in the migration

Azure Identity has several limitations:

No logout

Azure Identity doesn't have a logout functionality which is required by az logout. Azure CLI has to directly talk to MSAL to call ClientApplication.remove_account.

No data parameter

Azure Identity can't pass data parameter to get_token method. This parameter is used to get VM SSH certificate.

• Azure Identity removed the functionality of passing data argument to get_token for user account: Support CAE in azure-identity azure-sdk-for-python#16323 (comment)
• Azure Identity rejected the proposed change of passing data argument to get_token for service principal: [Identity] Let GetTokenMixin.get_token pass kwargs to MSAL azure-sdk-for-python#16397

Because of this limitation, Azure CLI has to directly talk to MSAL to get VM SSH certificate. As getting a certificate is a superset of setting an access token, once getting certificate with MSAL is implemented, there is no need to use Azure Identity to get access token. Doing so will make Azure CLI dual-stack and hard to maintain.

Not able to turn off token encryption

There are scenarios where encryption is available but not working as expected. Azure Identity uses the allow_unencrypted_storage approach which makes it impossible to force unencrypted token cache. Application using Azure Identity will be blocked in such case with no workaround.

Azure CLI adds an option core.encrypt_token_cache to disable token encryption so that Azure CLI can run without encryption (#19506).