AzureContainerInstancesOperator uses a deprecated authentication flow in apache-airflow-providers-microsoft-azure

[strictness]

Apache Airflow version

2.7.1

What happened

My previous bug report (#34736) was closed, but upon further investigation I am pretty sure that the fix should be handled in this repository. I have added the solution in a separate section below.

We currently have to hardcode our version of apache-airflow-providers-microsoft-azure to 6.2.4 in order for AzureContainerInstancesOperator to work as expected. If we bump the version to 6.3.0 or higher we get this error during execution:

[2023-10-03, 19:07:37 EEST] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 196, in execute
    if self._ci_hook.exists(self.resource_group, self.name):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 198, in exists
    for container in self.connection.container_groups.list_by_resource_group(resource_group):
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 123, in __next__
    return next(self._page_iterator)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 75, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/containerinstance/operations/_container_groups_operations.py", line 182, in get_next
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 230, in run
    return first_node.send(pipeline_request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
    self.on_request(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
    self._token = self._credential.get_token(*self._scopes)
AttributeError: 'ServicePrincipalCredentials' object has no attribute 'get_token'

What you think should happen instead

The hook in container_instance.py should use the azure.identity package instead of azure.common.credentials. Link to code

Microsoft posted deprecation warnings about using that package since early 2021 - azure.common.

Here is a related issue, where the solution is to start using azure.identity instead.

How to reproduce

Install airflow 2.7.1 and try creating a DAG with AzureContainerInstancesOperator and run it.

Operating System

Windows 10

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure=6.3.0 or 7.0.0

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

Hi
thank you for this input!
We are updating the library in #34738
and tracking the task in #30199
can you share your input on the PR itself?
cc @pankajastro

[eladkal]

fixed in #34738