Respect AZURE_CLIENT_ID, ANSIBLE_AZURE_AUTH_SOURCE on inventory plugin #713

kingsleyadam · 2021-12-20T15:35:18Z

SUMMARY

Fixes #712, set the inventory plugin to respect the ANSIBLE_AZURE_AUTH_SOURCE environment variable, and ensures AZURE_CLIENT_ID environment variable is pulled in if set.

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

plugin: azure.azcollection.azure_rm

ADDITIONAL INFORMATION

Before Change with ANSIBLE_AZURE_AUTH_SOURCE and AZURE_CLIENT_ID set

Using /opt/ansible-repo/ansible.cfg as config file
[WARNING]:  * Failed to parse /opt/ansible-repo/inventory/ops-az/azure_rm.yml with ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin: Failed to get credentials. Either pass as parameters, set environment
variables, define a profile in ~/.azure/credentials, or log in with Azure CLI (`az login`).

After Change, a complete list of the inventory using ansible-inventory command.

kingsleyadam · 2022-01-30T18:04:15Z

@Fred-sun, any chance someone can review this PR? I'm in need this enhancement/fix and I really don't want to fork the repo.

Thanks!

plugins/doc_fragments/azure.py

Fred-sun · 2022-03-09T07:40:05Z

any chance someone can review this PR? I'm in need this enhancement/fix and I really don't want to fork the repo.

Thanks!

@kingsleyadam Small change requests.

Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com>

kingsleyadam · 2022-03-09T07:43:10Z

any chance someone can review this PR? I'm in need this enhancement/fix and I really don't want to fork the repo.
Thanks!

@kingsleyadam Small change requests.

Updated, thanks for reviewing!

plugins/doc_fragments/azure.py

Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com>

xuzhang3 · 2022-04-24T06:15:36Z

Changes:

Fix inventory ignore the ENV configure
Support MSI auth using custom managed identity

xuzhang3 · 2022-04-24T07:00:37Z

@kingsleyadam can you provider test result of the fix/enhancement? I cannot get this work with MSI authorization

testotxt · 2022-07-29T05:20:15Z

hi any news on this? #712 is still present on azure.azcollection 1.12.0

Fred-sun · 2022-07-30T08:11:08Z

o

@testotxt This PR still needs to be improved by contributors, so the merger cannot be promoted for the time being. Thank you very much!

Fred-sun · 2022-07-30T08:11:19Z

@kingsleyadam Kindly ping!

kingsleyadam · 2022-11-07T14:56:28Z

@Fred-sun, @xuzhang3, finally getting back to this. Sorry for the delay. Here's the testing I've done from an Azure VM with multiple Managed/User Assigned Identities.

No Fix, setting MSI

This is the code before any changes. Setting ANSIBLE_AZURE_AUTH_SOURCE=msi gets ignored, the error indicates it requires the azure cli login.

(ansible) user@stackstorm:/opt/ansible-repo# export ANSIBLE_AZURE_AUTH_SOURCE=msi
(ansible) user@stackstorm:/opt/ansible-repo# ansible-inventory -i inventory/sandbox-az --graph -vvvv
ansible-inventory 2.9.27
  config file = /opt/ansible-repo/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/stackstorm/virtualenvs/ansible/lib/python3.8/site-packages/ansible
  executable location = /opt/stackstorm/virtualenvs/ansible/bin/ansible-inventory
  python version = 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
Using /opt/ansible-repo/ansible.cfg as config file
setting up inventory plugins
[WARNING]:  * Failed to parse /opt/ansible-repo/inventory/sandbox-az/azure_rm.yml with ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin: Failed to get credentials. Either pass as parameters, set environment variables, define a profile in
~/.azure/credentials, or log in with Azure CLI (`az login`).
  File "/opt/stackstorm/virtualenvs/ansible/lib/python3.8/site-packages/ansible/inventory/manager.py", line 280, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 218, in parse
    self._credential_setup()
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 239, in _credential_setup
    self.azure_auth = AzureRMAuth(**auth_options)
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1454, in __init__
    self.fail("Failed to get credentials. Either pass as parameters, set environment variables, "
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1562, in fail
    self._fail_impl(msg)
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1565, in _default_fail_impl
    raise AzureRMAuthException(msg)

Fix with MSI but no CLIENT ID

The below includes the recommended fix. But without setting the AZURE_CLIENT_ID environment variable. This VM has multiple identities, a system assigned and a user assigned. By default azure selects the system assigned.

In this case the system assigned identity does not have any access. The error shows an attempt to use MSI, but failed due to access.

(ansible) user@stackstorm:/opt/ansible-repo# export ANSIBLE_AZURE_AUTH_SOURCE=msi
(ansible) user@stackstorm:/opt/ansible-repo# ansible-inventory -i inventory/sandbox-az --graph -vvvv
ansible-inventory 2.9.27
  config file = /opt/ansible-repo/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/stackstorm/virtualenvs/ansible/lib/python3.8/site-packages/ansible
  executable location = /opt/stackstorm/virtualenvs/ansible/bin/ansible-inventory
  python version = 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
Using /opt/ansible-repo/ansible.cfg as config file
setting up inventory plugins
[WARNING]:  * Failed to parse /opt/ansible-repo/inventory/sandbox-az/azure_rm.yml with ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin: Failed to get MSI token: End of paging. Please check whether your machine enabled MSI or grant access
to any subscription.
  File "/opt/stackstorm/virtualenvs/ansible/lib/python3.8/site-packages/ansible/inventory/manager.py", line 280, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 221, in parse
    self._credential_setup()
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/inventory/azure_rm.py", line 243, in _credential_setup
    self.azure_auth = AzureRMAuth(**auth_options)
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1458, in __init__
    self.credentials = self._get_credentials(
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1720, in _get_credentials
    return self._get_msi_credentials(subscription_id=params.get('subscription_id'), client_id=params.get('client_id'),
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1666, in _get_msi_credentials
    self.fail("Failed to get MSI token: {0}. "
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1605, in fail
    self._fail_impl(msg)
  File "/home/user/.ansible/collections/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 1608, in _default_fail_impl
    raise AzureRMAuthException(msg)

Fix with MSI and CLIENT ID

The following fix tests setting both ANSIBLE_AZURE_AUTH_SOURCE and AZURE_CLIENT_ID. Shows a successful call to the inventory plugin with results.

(ansible) user@stackstorm:/opt/ansible-repo# export ANSIBLE_AZURE_AUTH_SOURCE=msi
(ansible) user@stackstorm:/opt/ansible-repo# export AZURE_CLIENT_ID=<redacted>
(ansible) user@stackstorm:/opt/ansible-repo# ansible-inventory -i inventory/sandbox-az --graph -vvvv
ansible-inventory 2.9.27
  config file = /opt/ansible-repo/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/stackstorm/virtualenvs/ansible/lib/python3.8/site-packages/ansible
  executable location = /opt/stackstorm/virtualenvs/ansible/bin/ansible-inventory
  python version = 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0]
Using /opt/ansible-repo/ansible.cfg as config file
setting up inventory plugins
Parsed /opt/ansible-repo/inventory/sandbox-az/azure_rm.yml inventory source with ansible_collections.azure.azcollection.plugins.inventory.azure_rm plugin
setting up inventory plugins
Parsed /opt/ansible-repo/inventory/sandbox-az/constructed.yml inventory source with constructed plugin
@all:
  |--@apiserver:
  |  |--apiserver-0.sandbox.az.internal.example.com
  |  |--apiserver-1.sandbox.az.internal.example.com
  |--@backend:
  |  |--apiserver-0.sandbox.az.internal.example.com
  |  |--apiserver-1.sandbox.az.internal.example.com
  |  |--console-0.sandbox.az.internal.example.com
  |  |--kwc-0-0.sandbox.az.internal.example.com
  |  |--nlpserver-0.sandbox.az.internal.example.com
  |  |--replicationserver-0.sandbox.az.internal.example.com
  |  |--sputnik-0.sandbox.az.internal.example.com
  |  |--workflow-0.sandbox.az.internal.example.com
  |--@chatbot:
  |  |--chatbot-0.sandbox.az.internal.example.com
  |--@console:
  |  |--console-0.sandbox.az.internal.example.com
  |--@kwc:
  |  |--kwc-0-0.sandbox.az.internal.example.com
  |--@nlpserver:
  |  |--nlpserver-0.sandbox.az.internal.example.com
  |--@rabbitmq:
  |  |--services-0.sandbox.az.internal.example.com
  |--@realm_hosts:
  |  |--apiserver-0.sandbox.az.internal.example.com
  |  |--apiserver-1.sandbox.az.internal.example.com
  |  |--chatbot-0.sandbox.az.internal.example.com
  |  |--console-0.sandbox.az.internal.example.com
  |  |--kwc-0-0.sandbox.az.internal.example.com
  |  |--nlpserver-0.sandbox.az.internal.example.com
  |  |--render-0.sandbox.az.internal.example.com
  |  |--replicationserver-0.sandbox.az.internal.example.com
  |  |--services-0.sandbox.az.internal.example.com
  |  |--services-1.sandbox.az.internal.example.com
  |  |--services-2.sandbox.az.internal.example.com
  |  |--services-3.sandbox.az.internal.example.com
  |  |--services-4.sandbox.az.internal.example.com
  |  |--services-5.sandbox.az.internal.example.com
  |  |--services-6.sandbox.az.internal.example.com
  |  |--sftp-0.sandbox.az.internal.example.com
  |  |--sputnik-0.sandbox.az.internal.example.com
  |  |--webapp-0.sandbox.az.internal.example.com
  |  |--workflow-0.sandbox.az.internal.example.com
  |--@render:
  |  |--render-0.sandbox.az.internal.example.com
  |--@replicationserver:
  |  |--replicationserver-0.sandbox.az.internal.example.com
  |--@services:
  |  |--services-0.sandbox.az.internal.example.com
  |  |--services-1.sandbox.az.internal.example.com
  |  |--services-2.sandbox.az.internal.example.com
  |  |--services-3.sandbox.az.internal.example.com
  |  |--services-4.sandbox.az.internal.example.com
  |  |--services-5.sandbox.az.internal.example.com
  |  |--services-6.sandbox.az.internal.example.com
  |--@sftp:
  |  |--sftp-0.sandbox.az.internal.example.com
  |--@sputnik:
  |  |--sputnik-0.sandbox.az.internal.example.com
  |--@ui:
  |  |--chatbot-0.sandbox.az.internal.example.com
  |  |--render-0.sandbox.az.internal.example.com
  |  |--webapp-0.sandbox.az.internal.example.com
  |--@ungrouped:
  |--@webapp:
  |  |--webapp-0.sandbox.az.internal.example.com
  |--@workflow:
  |  |--workflow-0.sandbox.az.internal.example.com

kingsleyadam · 2023-03-20T13:04:33Z

Friendly ping @Fred-sun

Fred-sun · 2023-03-21T00:47:59Z

plugins/inventory/azure_rm.py

@@ -223,8 +224,9 @@ def parse(self, inventory, loader, path, cache=True):
            raise

    def _credential_setup(self):
+        auth_source = environ.get('ANSIBLE_AZURE_AUTH_SOURCE', None) or self.get_option('auth_source')


@kingsleyadam I think this order should be adjusted, because the auth_source of the configuration is fetched first, and if the fetch is auto, the environment variable is fetched, thanks!

@kingsleyadam I think this order should be adjusted, because the auth_source of the configuration is fetched first, and if the fetch is auto, the environment variable is fetched, thanks!

That's the issue I was running into. If I run self.get_option('auth_source') first, it has a default value of auto. Which won't fetch the environment variable ANSIBLE_AZURE_AUTH_SOURCE value. To avoid this my logic here is to first check the environment variable for the auth source, if it's not set then fetch it from the config.

Fred-sun · 2023-12-01T08:57:38Z

@kingsleyadam Are you still paying attention to this PR? There is a conflict in this PR, can you help solve the conflict? I will push forward the merger of this PR as soon as possible, thank you!

kingsleyadam · 2023-12-01T11:02:00Z

@kingsleyadam Are you still paying attention to this PR? There is a conflict in this PR, can you help solve the conflict? I will push forward the merger of this PR as soon as possible, thank you!

Thanks for the heads up, I've merged the upstream/dev branch to mine and resolved the conflict. Should be good now.

ansible-collections#713) * Attemp to pull environment variables if not set. * Set ansible azure auth source further upstream * Updates to documentation * Update plugins/doc_fragments/azure.py Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com> * doc_fragments/azure.py Documentation update * Update plugins/doc_fragments/azure.py Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com> * Move all logic within azure_rm_common.py * Removed unused import * Move back to setting auth source in inventory --------- Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com>

kingsleyadam added 3 commits December 20, 2021 15:50

Attemp to pull environment variables if not set.

0e74e24

Set ansible azure auth source further upstream

aae2fa6

Updates to documentation

79a5add

Fred-sun added medium_priority Medium priority work in In trying to solve, or in working with contributors inventory plugin/Inventory/azure_rm.py related issues labels Jan 7, 2022

Fred-sun reviewed Mar 9, 2022

View reviewed changes

plugins/doc_fragments/azure.py Outdated Show resolved Hide resolved

Fred-sun added the enhancement New feature or request label Mar 9, 2022

kingsleyadam and others added 2 commits March 9, 2022 08:40

Update plugins/doc_fragments/azure.py

0ebfde0

Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com>

doc_fragments/azure.py Documentation update

90b1793

Fred-sun reviewed Mar 10, 2022

View reviewed changes

plugins/doc_fragments/azure.py Outdated Show resolved Hide resolved

Update plugins/doc_fragments/azure.py

48d4b70

Co-authored-by: Fred-sun <37327967+Fred-sun@users.noreply.github.com>

Fred-sun mentioned this pull request Jul 30, 2022

Inventory Plugin ignores ANSIBLE_AZURE_AUTH_SOURCE, AZURE_CLIENT_ID env variables #712

Closed

kingsleyadam closed this Nov 7, 2022

kingsleyadam force-pushed the dev branch from 48d4b70 to 8707edf Compare November 7, 2022 12:59

kingsleyadam added 4 commits November 7, 2022 14:00

Merge branch 'dev' of github.com:kingsleyadam/azure into dev

e68ff07

Move all logic within azure_rm_common.py

3bad211

Removed unused import

ae85a89

Move back to setting auth source in inventory

f2ff451

kingsleyadam reopened this Nov 7, 2022

Fred-sun reviewed Mar 21, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/dev' into dev

b4132e2

Fred-sun added ready_for_review The PR has been modified and can be reviewed and merged and removed work in In trying to solve, or in working with contributors labels Feb 19, 2024

Fred-sun assigned xuzhang3 Feb 19, 2024

xuzhang3 approved these changes Mar 22, 2024

View reviewed changes

xuzhang3 merged commit d753177 into ansible-collections:dev Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Respect AZURE_CLIENT_ID, ANSIBLE_AZURE_AUTH_SOURCE on inventory plugin #713

Respect AZURE_CLIENT_ID, ANSIBLE_AZURE_AUTH_SOURCE on inventory plugin #713

kingsleyadam commented Dec 20, 2021

kingsleyadam commented Jan 30, 2022

Fred-sun commented Mar 9, 2022

kingsleyadam commented Mar 9, 2022

xuzhang3 commented Apr 24, 2022

xuzhang3 commented Apr 24, 2022

testotxt commented Jul 29, 2022

Fred-sun commented Jul 30, 2022

Fred-sun commented Jul 30, 2022

kingsleyadam commented Nov 7, 2022

kingsleyadam commented Mar 20, 2023

Fred-sun Mar 21, 2023

kingsleyadam Mar 21, 2023

Fred-sun commented Dec 1, 2023

kingsleyadam commented Dec 1, 2023

Respect AZURE_CLIENT_ID, ANSIBLE_AZURE_AUTH_SOURCE on inventory plugin #713

Respect AZURE_CLIENT_ID, ANSIBLE_AZURE_AUTH_SOURCE on inventory plugin #713

Conversation

kingsleyadam commented Dec 20, 2021

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

kingsleyadam commented Jan 30, 2022

Fred-sun commented Mar 9, 2022

kingsleyadam commented Mar 9, 2022

xuzhang3 commented Apr 24, 2022

xuzhang3 commented Apr 24, 2022

testotxt commented Jul 29, 2022

Fred-sun commented Jul 30, 2022

Fred-sun commented Jul 30, 2022

kingsleyadam commented Nov 7, 2022

No Fix, setting MSI

Fix with MSI but no CLIENT ID

Fix with MSI and CLIENT ID

kingsleyadam commented Mar 20, 2023

Fred-sun Mar 21, 2023

Choose a reason for hiding this comment

kingsleyadam Mar 21, 2023

Choose a reason for hiding this comment

Fred-sun commented Dec 1, 2023

kingsleyadam commented Dec 1, 2023