AzureContainerInstancesOperator doesn't work any more after version 6.3.0 of apache-airflow-providers-microsoft-azure

[strictness]

Apache Airflow version

2.7.1

What happened

We have to hardcode our version of apache-airflow-providers-microsoft-azure to 6.2.4 in order for AzureContainerInstancesOperator to work as expected. If we bump the version to 6.3.0 or higher we get this error during execution:

[2023-10-03, 19:07:37 EEST] {taskinstance.py:1935} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/operators/container_instances.py", line 196, in execute
    if self._ci_hook.exists(self.resource_group, self.name):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/microsoft/azure/hooks/container_instance.py", line 198, in exists
    for container in self.connection.container_groups.list_by_resource_group(resource_group):
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 123, in __next__
    return next(self._page_iterator)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/paging.py", line 75, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/containerinstance/operations/_container_groups_operations.py", line 182, in get_next
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 230, in run
    return first_node.send(pipeline_request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/_base.py", line 86, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/mgmt/core/policies/_base.py", line 46, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect.py", line 197, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_retry.py", line 531, in send
    response = self.next.send(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 124, in send
    self.on_request(request)
  File "/home/airflow/.local/lib/python3.9/site-packages/azure/core/pipeline/policies/_authentication.py", line 99, in on_request
    self._token = self._credential.get_token(*self._scopes)
AttributeError: 'ServicePrincipalCredentials' object has no attribute 'get_token'

What you think should happen instead

I would expect the operator to be able to connect to Azure without an issue.

How to reproduce

Install airflow 2.7.1 and try running AzureContainerInstancesOperator with some image and run it.

Operating System

Windows 10

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-azure=6.3.0 or 7.0.0

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

I not familiar with internals of Azure libraries but guess it somehow related to:

• AttributeError: 'ServicePrincipalCredentials' object has no attribute 'get_token' on resource_groups.list() Azure/azure-sdk-for-python#16908
• AttributeError: 'UserPassCredentials' object has no attribute 'get_token' Azure/azure-sdk-for-python#14919
• Bump azure-mgmt-containerinstance>=7.0.0,<9.0.0 #33696

[eladkal]

Note that version 6.3.0 updated upstream library:
Bump azure-mgmt-containerinstance>=7.0.0,<9.0.0 (#33696)
https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/6.3.0/changelog.html#misc

Airflow did not break the interface of the operator (this is why this is not a breaking change release from Airlfow side) but the upstream library did change logic and bumped major release.
You will need to review your code and make sure it's up to date with the newer version of azure-mgmt-containerinstance
For the moment - it does not seem like a bug in the provider code.