Do not follow path

[mysiki]

hello,

I just try this tool but I have some issue on path file.

I have the following file mapping (for sample I have deleted some folder):

├── 000-initialize-afs.ym
├── tasks
        ├── include_galaxy.yml
        ├── loop_tasks
                └── _galaxy_resolve_role.yml

When I run : ansible-playbook-grapher 000-initialize-afs.yml

I get this error :

Traceback (most recent call last):
  File "/usr/local/bin/ansible-playbook-grapher", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/ansibleplaybookgrapher/cli.py", line 84, in main
    cli.run()
  File "/usr/local/lib/python2.7/dist-packages/ansibleplaybookgrapher/cli.py", line 32, in run
    skip_tags=self.options.skip_tags)
  File "/usr/local/lib/python2.7/dist-packages/ansibleplaybookgrapher/grapher.py", line 198, in make_graph
    skip_tags=skip_tags)
  File "/usr/local/lib/python2.7/dist-packages/ansibleplaybookgrapher/grapher.py", line 292, in _include_tasks_in_blocks
    skip_tags=skip_tags)
  File "/usr/local/lib/python2.7/dist-packages/ansibleplaybookgrapher/grapher.py", line 276, in _include_tasks_in_blocks
    data = self.data_loader.load_from_file(include_file)
  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/dataloader.py", line 88, in load_from_file
    (b_file_data, show_content) = self._get_file_contents(file_name)
  File "/usr/local/lib/python2.7/dist-packages/ansible/parsing/dataloader.py", line 156, in _get_file_contents
    raise AnsibleFileNotFound("Unable to retrieve file contents", file_name=file_name)
ansible.errors.AnsibleFileNotFound: Unable to retrieve file contents
Could not find or access '/home/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure/loop_tasks/_galaxy_resolve_role.yml'

Of course, /home/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure/loop_tasks/_galaxy_resolve_role.yml do not exist. I have this mapping

grep _galaxy 000-initialize-afs.yml
      include_tasks: "tasks/include_galaxy.yml"

 grep _galaxy tasks/include_galaxy.yml
    stat_galaxy_file: none
  register: stat_galaxy_file
      include_tasks: "loop_tasks/_galaxy_resolve_role.yml"
  when: stat_galaxy_file.stat.exists

If I summarize : 000-initialize-afs.yml call tasks/include_galaxy.yml call loop_tasks/_galaxy_resolve_role.yml

[mysiki]

Hey, happy to see that you taken this bug :). I just try it, it's work for the first case, but have the same problem for import_playbook.

My folder structure :

├── 001-deploy-afs.yml
├── ansible.cfg
├── playbooks
│   ├── 000-providers
│   │   ├── 001_play_infra.yml
│   │   ├── rhone
│   │   │   ├── _play_infra.yml
│   │   │   ├── tasks
│   │   │   │   ├── networks
│   │   │   │   │   ├── set-bastion-network-dict.yml
│   │   │   │   │   └── set-cluster-network-dict.yml
│   │   │   │   └── servers
│   │   │   │       ├── loops
│   │   │   │       │   └── manage-group.yml
│   │   │   │       ├── set-bastion.yml
│   │   │   │       ├── set-managers.yml
│   │   │   │       ├── set-nodes.yml
│   │   │   │       └── set-vip-cluster.yml
│   │   │   └── vars
│   │   │       └── afs-rhone-configuration.yml
│   ├── _play_bastion.yml
│   ├── _play_cluster.yml
│   └── _play_export.yml
├── README.md
├── tasks
│   ├── check_and_include_vars.yml
│   ├── check_cloud_init_finished.yml
│   ├── include_begin_custom_tasks.yml
│   ├── include_end_custom_tasks.yml
│   ├── include_galaxy.yml
│   ├── initialize
│   │   └── git-import.yml
│   ├── loop_tasks
│   │   └── _galaxy_resolve_role.yml
│   └── maintenance
│       └── move-docker-graph-to-volume.yml

My playbook follow this workflow :
001-deploy-afs.yml --> import_playbook: playbooks/000-providers/001_play_infra.yml --> import_playbook: "rhone/_play_infra.yml" --> include_tasks: tasks/networks/set-bastion-network-dict.yml

I get the same error :

ansible.errors.AnsibleFileNotFound: Unable to retrieve file contents
Could not find or access '/home/demo/ansible/AFS/PIC/playbook-athena-foundation-service-core-infrastructure/tasks/networks/set-bastion-network-dict.yml'

It try to find tasks/networks/set-bastion-network-dict.yml from the root path and not in relative path from rhone/_play_infra.yml (which call it)

[haidaraM]

Hey @mysiki Trying to implement this but as you may know include_* and import_* are not that simple 😅.

I'm facing a weird issue on ansible <2.7.0 import_role that causes other tests to fail.

[haidaraM]

Hi @mysiki , can you test with the latest version on #17 (branch fixIssue16NestedInclude) ? The latest changes requires ansible >= 2.7 for all the features to work

[mysiki]

hey, just test it, it's ok for the path following but I get another error.
/playbooks/000-providers/sub/playbooks/networks/{{ private_playbook_dir }}/tasks/customs/run-custom-tasks.yml
Tested in Ansible 2.5.14.

I will test it in 2.7 asap

[mysiki]

Same in 2.7.

#Define playbook_dir
./playbooks/000-providers/sub/000-run-playbook.yml:        private_playbook_dir: "{{ playbook_dir }}"

#Use it in include task 
include_tasks: "{{ private_playbook_dir }}/tasks/customs/run-custom-tasks.yml"

The tool search :
playbook-athena-foundation-service-core-infrastructure/playbooks/000-providers/sub/playbooks/networks/{{ private_playbook_dir }}/tasks/customs/run-custom-tasks.yml
in place of
playbook-athena-foundation-service-core-infrastructure/playbooks/000-providers/sub/tasks/customs

This action is running by a import playbook present in "sub", so private_playbook_dir is set to sub-playbook path.

-rw-r--r-- 1 1000 ubuntu  2317 Jan 29 09:52 000-run-playbook.yml
-rw-r--r-- 1 1000 ubuntu  1613 Jan 29 09:52 galaxy.yml
drwxr-xr-x 6 1000 ubuntu  4096 Jan 29 09:52 playbooks/

I maybe need to give you more information ?

[haidaraM]

Looks like a variable interpolation issue. Can you test with my latest changes ?

[mysiki]

I have no more interpolation :p

ansible.errors.AnsibleUndefinedVariable: 'private_playbook_dir' is undefined

[haidaraM]

Hmm, what is the whole stack trace ? Would like to which line is causing the error.

I'm trying to reproduce it locally, can you show me the related lines of the files used ?

[mysiki]

Trace :

Traceback (most recent call last):
  File "/usr/local/bin/ansible-playbook-grapher", line 11, in <module>
    load_entry_point('ansible-playbook-grapher', 'console_scripts', 'ansible-playbook-grapher')()
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/cli.py", line 89, in main
    cli.run()
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/cli.py", line 31, in run
    grapher.make_graph()
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 219, in make_graph
    play_vars=play_vars, node_name_prefix='[task] ')
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 293, in _include_tasks_in_blocks
    play_vars=play_vars, node_name_prefix=node_name_prefix)
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 293, in _include_tasks_in_blocks
    play_vars=play_vars, node_name_prefix=node_name_prefix)
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 308, in _include_tasks_in_blocks
    templar=templar)
  File "/mnt/ansible-playbook-grapher/ansibleplaybookgrapher/utils.py", line 294, in handle_include_path
    return templar.template(include_file)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/template/__init__.py", line 495, in template
    disable_lookups=disable_lookups,
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/template/__init__.py", line 777, in do_template
    raise AnsibleUndefinedVariable(e)
ansible.errors.AnsibleUndefinedVariable: 'private_playbook_dir' is undefined

It's possible to active/add "debug" mode ? The stack trace are really small, if you can (temporary) add the current file parsing, it will be more easy to debug it.

How the tools work, it use --check mode of ansible (because I'm not sure that all my playbook reply correctly to --check :p )

[haidaraM]

I will definitely add some traces to be able to debug more easily :-) Pushed some changes last night, can you check (again) ?

The grapher works by using the same classes Ansible uses internally to parse the entire playbook but as I have discovered now, the parsing is scattered between multiple functions/classes. Need to figure out which ones are necessary to get all the information I need to make a graph.
Some tasks such as include_* are special because they are evaluated during execution time. "The facts" are not available also as are they will require to run the playbook.

[mysiki]

near ! :D

I got this trace :

 [WARNING]: Unable to compute the task '{{ tasks_comment }} : Run before custom tasks' due an undefined variable: 'private_playbook_dir' is undefined. Some variables are available only during the real execution.

 [WARNING]: Unable to compute the task '{{ tasks_comment }} : Run after custom tasks' due an undefined variable: 'private_playbook_dir' is undefined. Some variables are available only during the real execution.

Traceback (most recent call last):
  File "/usr/local/bin/ansible-playbook-grapher", line 11, in <module>
    load_entry_point('ansible-playbook-grapher', 'console_scripts', 'ansible-playbook-grapher')()
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/cli.py", line 89, in main
    cli.run()
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/cli.py", line 31, in run
    grapher.make_graph()
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 217, in make_graph
    play_vars=play_vars, node_name_prefix='[task] ')
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 288, in _include_tasks_in_blocks
    play_vars=play_vars, node_name_prefix=node_name_prefix)
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 331, in _include_tasks_in_blocks
    node_name_prefix=node_name_prefix)
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 331, in _include_tasks_in_blocks
    node_name_prefix=node_name_prefix)
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 331, in _include_tasks_in_blocks
    node_name_prefix=node_name_prefix)
  File "/mnt/ubuntu/ansible-playbook-grapher/ansibleplaybookgrapher/grapher.py", line 315, in _include_tasks_in_blocks
    data = self.data_loader.load_from_file(include_file)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/parsing/dataloader.py", line 89, in load_from_file
    (b_file_data, show_content) = self._get_file_contents(file_name)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/ansible/parsing/dataloader.py", line 157, in _get_file_contents
    raise AnsibleFileNotFound("Unable to retrieve file contents", file_name=file_name)
ansible.errors.AnsibleFileNotFound: Unable to retrieve file contents
Could not find or access '/mnt/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure/playbooks/000-providers/sub/playbooks/servers/__manage_volume.yml' on the Ansible Controller.
If you are using a module and expect the file to exist on the remote, see the remote_src option

If I grep file (read it from bottom to top) :

root@afssamplectl01t:/mnt/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure# grep -ri __manage_volume.yml
roles/role-openstack-cinder-volume/tasks/_delete_volume.yml:  include_tasks: __manage_volume.yml
roles/role-openstack-cinder-volume/tasks/_create_volume.yml:  include_tasks: __manage_volume.yml
root@afssamplectl01t:/mnt/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure# grep -ri roles/role-openstack-cinder-volume
root@afssamplectl01t:/mnt/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure# grep -ri role-openstack-cinder-volume
playbooks/000-providers/sub/playbooks/servers/tasks/manage-cinder-disk.yml:        name: role-openstack-cinder-volume
playbooks/000-providers/sub/galaxy.yml:# Install role-openstack-cinder-volume role from Athena
playbooks/000-providers/sub/galaxy.yml:- src: https://gitlab.agora-t.net/Automation-Toolbox/Virtualization/openstack-rhone-provisioning/role-openstack-cinder-volume.git
playbooks/000-providers/sub/galaxy.yml:  name: role-openstack-cinder-volume
root@afssamplectl01t:/mnt/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure# grep -ri manage-cinder-disk.yml
playbooks/000-providers/sub/playbooks/servers/manage-servers.yml:          include_tasks: "tasks/manage-cinder-disk.yml"
Binary file playbooks/000-providers/sub/.git/index matches
root@afssamplectl01t:/mnt/ubuntu/AFS/142/playbook-athena-foundation-service-core-infrastructure# grep -ri manage-servers.yml
playbooks/000-providers/sub/playbooks/manage-infrastructure.yml:  import_playbook: servers/manage-servers.yml
Binary file playbooks/000-providers/sub/.git/index matches

So, manage-infrastructure.yml: import_playbook: servers/manage-servers.yml --> manage-servers.yml: include_tasks: "tasks/manage-cinder-disk.yml" --> manage-cinder-disk.yml: name: role-openstack-cinder-volume --> _create_volume.yml: include_tasks: __manage_volume.yml

It seem that the include in role do not start from her path ('./') but form calling playbook_dir

[haidaraM]

Can you pull my latest changes, an run it with the option -vv ?

[mysiki]

hey, it's really good ! I have some warning but I think it's normal like :
'ansible_distribution' is undefined. Some variables are available only during the real execution.

But I have my svg (I canno't check if all the step are correctly present right now, but it seem to be ok)

Test make in ansible 2.7 without -vv

[haidaraM]

Cool :-). Will make a release in the next few days