Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors when building multiple collections at once #193

Closed
wlitke opened this issue Aug 3, 2023 · 12 comments
Closed

Errors when building multiple collections at once #193

wlitke opened this issue Aug 3, 2023 · 12 comments

Comments

@wlitke
Copy link

wlitke commented Aug 3, 2023

If I try to build a documentation site for multiple (small) collections it takes long time and fails afterwards with the below error messages. Any idea about a possible root cause?

Versions in use

  • Ubuntu 20.04
  • Virtual python environment (based on virtualenv) with Python v3.11.4
  • antsibull-docs v2.3.1
  • ansible [core 2.15.2]
    • config file = None
    • configured module search path = ['/home/vagrant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    • ansible python module location = /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible
    • ansible collection location = /home/vagrant/.ansible/collections:/usr/share/ansible/collections
    • executable location = /home/vagrant/venv/ans_py311/bin/ansible
    • python version = 3.11.4 (main, Jun 7 2023, 12:45:49) [GCC 9.4.0] (/home/vagrant/venv/ans_py311/bin/python)
    • jinja version = 3.1.2
    • libyaml = True

Commands

source /home/vagrant/venv/ans_py311/bin/activate
antsibull-docs sphinx-init --use-current --dest-dir /home/vagrant/ans_test/DEST mynamespace.first_collection mynamespace.second_collection
cd /home/vagrant/ans_test/DEST
pip install -r requirements.txt 
./build.sh

Output

Skipping /home/vagrant/ans_test/DEST/.gitignore
Skipping /home/vagrant/ans_test/DEST/antsibull-docs.cfg
Skipping /home/vagrant/ans_test/DEST/build.sh
Skipping /home/vagrant/ans_test/DEST/conf.py
Skipping /home/vagrant/ans_test/DEST/requirements.txt
Skipping /home/vagrant/ans_test/DEST/rst/index.rst
To build the docsite, go into /home/vagrant/ans_test/DEST and run:
    pip install -r requirements.txt  # possibly use a venv
    ./build.sh
Requirement already satisfied: antsibull-docs<3.0.0,>=2.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from -r requirements.txt (line 5)) (2.3.1)
Requirement already satisfied: ansible-pygments in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from -r requirements.txt (line 6)) (0.1.1)
Requirement already satisfied: sphinx in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from -r requirements.txt (line 7)) (6.2.1)
Requirement already satisfied: sphinx-ansible-theme>=0.9.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from -r requirements.txt (line 8)) (0.10.2)
Requirement already satisfied: aiohttp>=3.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (3.8.5)
Requirement already satisfied: antsibull-core<3.0.0,>=2.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (2.0.0)
Requirement already satisfied: antsibull-docs-parser<2.0.0,>=1.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.0.0)
Requirement already satisfied: asyncio-pool in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.6.0)
Requirement already satisfied: docutils in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.18.1)
Requirement already satisfied: jinja2>=3.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (3.1.2)
Requirement already satisfied: packaging in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (23.1)
Requirement already satisfied: pydantic<2.0.0,>=1.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.10.12)
Requirement already satisfied: pyyaml in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (6.0.1)
Requirement already satisfied: rstcheck<7.0.0,>=3.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (6.1.2)
Requirement already satisfied: semantic-version in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (2.10.0)
Requirement already satisfied: twiggy in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.5.1)
Requirement already satisfied: pygments>=2.4.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from ansible-pygments->-r requirements.txt (line 6)) (2.15.1)
Requirement already satisfied: sphinxcontrib-applehelp in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (1.0.4)
Requirement already satisfied: sphinxcontrib-devhelp in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (1.0.2)
Requirement already satisfied: sphinxcontrib-jsmath in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (1.0.1)
Requirement already satisfied: sphinxcontrib-htmlhelp>=2.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (2.0.1)
Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (1.1.5)
Requirement already satisfied: sphinxcontrib-qthelp in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (1.0.3)
Requirement already satisfied: snowballstemmer>=2.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (2.2.0)
Requirement already satisfied: babel>=2.9 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (2.12.1)
Requirement already satisfied: alabaster<0.8,>=0.7 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (0.7.13)
Requirement already satisfied: imagesize>=1.3 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (1.4.1)
Requirement already satisfied: requests>=2.25.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx->-r requirements.txt (line 7)) (2.31.0)
Requirement already satisfied: sphinx-rtd-theme>=1.1.1 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx-ansible-theme>=0.9.0->-r requirements.txt (line 8)) (1.2.2)
Requirement already satisfied: attrs>=17.3.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (3.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from aiohttp>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.3.1)
Requirement already satisfied: aiofiles in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-core<3.0.0,>=2.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (23.1.0)
Requirement already satisfied: build in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-core<3.0.0,>=2.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.10.0)
Requirement already satisfied: perky in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-core<3.0.0,>=2.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.9.2)
Requirement already satisfied: sh<2.0.0,>=1.0.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from antsibull-core<3.0.0,>=2.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.14.3)
Requirement already satisfied: MarkupSafe>=2.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from jinja2>=3.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (2.1.3)
Requirement already satisfied: typing-extensions>=4.2.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from pydantic<2.0.0,>=1.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (4.7.1)
Requirement already satisfied: idna<4,>=2.5 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from requests>=2.25.0->sphinx->-r requirements.txt (line 7)) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from requests>=2.25.0->sphinx->-r requirements.txt (line 7)) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from requests>=2.25.0->sphinx->-r requirements.txt (line 7)) (2023.7.22)
Requirement already satisfied: rstcheck-core<2.0.0,>=1.0.2 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.0.3)
Requirement already satisfied: typer[all]<0.8,>=0.4.1 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.7.0)
Requirement already satisfied: sphinxcontrib-jquery<5,>=4 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from sphinx-rtd-theme>=1.1.1->sphinx-ansible-theme>=0.9.0->-r requirements.txt (line 8)) (4.1)
Requirement already satisfied: six in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from twiggy->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.16.0)
Requirement already satisfied: types-docutils<0.20,>=0.18 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from rstcheck-core<2.0.0,>=1.0.2->rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.19.1.9)
Requirement already satisfied: click<9.0.0,>=7.1.1 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from typer[all]<0.8,>=0.4.1->rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (8.1.6)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from typer[all]<0.8,>=0.4.1->rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.4.6)
Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from typer[all]<0.8,>=0.4.1->rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.5.0.post1)
Requirement already satisfied: rich<13.0.0,>=10.11.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from typer[all]<0.8,>=0.4.1->rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (12.6.0)
Requirement already satisfied: pyproject_hooks in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from build->antsibull-core<3.0.0,>=2.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (1.0.0)
Requirement already satisfied: commonmark<0.10.0,>=0.9.0 in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages (from rich<13.0.0,>=10.11.0->typer[all]<0.8,>=0.4.1->rstcheck<7.0.0,>=3.0.0->antsibull-docs<3.0.0,>=2.0.0->-r requirements.txt (line 5)) (0.9.1)
~/ans_test/DEST ~/ans_test/DEST
Traceback (most recent call last):
  File "/home/vagrant/venv/ans_py311/bin/antsibull-docs", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/cli/antsibull_docs.py", line 797, in main
    return run(sys.argv)
           ^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/cli/antsibull_docs.py", line 778, in run
    return ARGS_MAP[parsed_args.command]()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/cli/doc_commands/collection.py", line 125, in generate_docs
    return generate_collection_docs(None, output_format, squash_hierarchy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/cli/doc_commands/collection.py", line 40, in generate_collection_docs
    return generate_docs_for_all_collections(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/cli/doc_commands/_build.py", line 112, in generate_docs_for_all_collections
    plugin_info, full_collection_metadata = asyncio.run(
                                            ^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/docs_parsing/parsing.py", line 76, in get_ansible_plugin_info
    return await ansible_doc_core_213_get_ansible_plugin_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/docs_parsing/ansible_doc_core_213.py", line 113, in get_ansible_plugin_info
    ansible_doc_output = await _call_ansible_doc(venv, env)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_docs/docs_parsing/ansible_doc_core_213.py", line 35, in _call_ansible_doc
    p = await venv.async_log_run(
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_core/venv.py", line 187, in async_log_run
    return await subprocess_util.async_log_run(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/antsibull_core/subprocess_util.py", line 132, in async_log_run
    completed.check_returncode()
  File "/usr/lib/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['ansible-doc', '-vvv', '--metadata-dump', '--no-fail-on-errors']' died with <Signals.SIGKILL: 9>.
~/ans_test/DEST
@felixfontein
Copy link
Collaborator

What happens if you run ansible-doc -vvv --metadata-dump --no-fail-on-errors in the venv directly? (Probably it's best to redirect stdout to a file, since the output is in general very huge.)

@wlitke
Copy link
Author

wlitke commented Aug 4, 2023

@felixfontein Thanks for your feedback! It was a good idea to execute the last command (ansible-doc -vvv --metadata-dump --no-fail-on-errors &> ansible-doc_output.log) without any other context. I've checked the output (see attachment ansible-doc_output.log) and noticed following:

  • The command needs much time to be executed
  • The command gets killed at the end
  • Many entries are listed multiple times (e. g. the following line is listed 12 times => Found installed collection ansible.windows:1.14.0 at '/home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible_collections/ansible/windows')

Most probably the issue isn't related to antsibull-docs but maybe someone has an idea for what reason the command could be killed. It couldn't find any google result at all where the ansible-doc command got killed.

ansible-doc_output.log

@felixfontein
Copy link
Collaborator

@wlitke my guess is that it got killed because it ran out of memory. You have a large amount of collections installed (all of the ones included in the ansible package), which results in the JSON output that ansible-doc tries to create to be very large.

You could set ANSIBLE_COLLECTIONS_PATH to the minimal list of directories so that the collections you are interested in are found by ansible. That will decrease the output size dramatically, and might avoid that crash.

@wlitke
Copy link
Author

wlitke commented Aug 4, 2023

@felixfontein Thank you very much for your time and for the hint! I've checked the system logs via command dmesg -T | grep -E -i -B100 'killed process' and see that the command got killed due to insufficient memory:

[Fri Aug  4 08:50:38 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-4.scope,task=ansible-doc,pid=2333,uid=1000
[Fri Aug  4 08:50:38 2023] Out of memory: Killed process 2333 (ansible-doc) total-vm:1456180kB, anon-rss:1415816kB, file-rss:1680kB, shmem-rss:0kB, UID:1000 pgtables:2892kB oom_score_adj:0

When I increase the VM memory to 4096 megabytes the command runs without issues. I'll check now the ANSIBLE_COLLECTIONS_PATH configuration to enhance the process flow thus the memory consumption.
I'll provide feedback.

@wlitke
Copy link
Author

wlitke commented Aug 4, 2023

@felixfontein Unfortunatelly setting the environment variable ANSIBLE_COLLECTIONS_PATH doesn't work as expected.

I've set the variable with command export ANSIBLE_COLLECTIONS_PATH=/home/vagrant/.ansible/collections and checked the availability with command env | grep ANSIBLE_COLLECTIONS_PATH which returned the same value.
When executing command ansible --version the new path is contained as expected in line ansible collection location:

ansible [core 2.15.2]
  config file = None
  configured module search path = ['/home/vagrant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible
  ansible collection location = /home/vagrant/.ansible/collections
  executable location = /home/vagrant/venv/ans_py311/bin/ansible
  python version = 3.11.4 (main, Jun  7 2023, 12:45:49) [GCC 9.4.0] (/home/vagrant/venv/ans_py311/bin/python)
  jinja version = 3.1.2
  libyaml = True

If I re-execute the origin command ansible-doc -vvv --metadata-dump --no-fail-on-errors it still scans all collections stored in /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible_collections. Do you have an idea how this path can be excluded? Obviously the variable ANSIBLE_COLLECTIONS_PATH doesn't limit the scan range as expected.

@gotmax23
Copy link
Collaborator

gotmax23 commented Aug 4, 2023

I believe collections that are part of the ansible package and installed in site-packages/ansible_collections are discovered by ansible regardless of ANSIBLE_COLLECTIONS_PATH.

@wlitke
Copy link
Author

wlitke commented Aug 4, 2023

@gotmax23 I have the same assumption but maybe there's a kind of hack which enables to ignore the collections in folder path /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible_collections.

Just to try it out I moved all collection folders from mentioned path into a temporary directory somewhere else (directory may not be part of any path visible in output of command ansible --version) and re-build the documentation website.
The result is as expected (without issues) and documentation generation time and computer load are dramatically reduced!

Maybe this is the kind of hack which must be performed but it really doesn't feel right to move folders temporary, just in order to generate a documentation. I hope strongly there's a better solution than moving folders around :)


Addition: Temporary rename of folder /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible_collections has the same (positive) result as moving the folder contents to a temporary folder

@gotmax23
Copy link
Collaborator

gotmax23 commented Aug 4, 2023

Thanks for following up. You can try creating a separate venv without the ansible package installed. There's not much the antsibull project can do about this issue, unfortunately.

@wlitke
Copy link
Author

wlitke commented Aug 4, 2023

@gotmax23 Yes, you are right. This issue is related to ansible and not to antsibull-docs project.

Summarized you need an environment where the folder /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible_collections is removed to prevent performance/memory issues during documentation generation.

Let's hope that this part can be configured in the future for ansible.

@gotmax23 @felixfontein Thank you very much guys for your great support!

@felixfontein
Copy link
Collaborator

I started an initiative (ansible/ansible#81450) to make it possible to build docs for just a few collections more efficiently, even if many more collections are installed. If we can get this (or something similar) merged, such problems should go away...

@gotmax23
Copy link
Collaborator

gotmax23 commented Aug 6, 2023

Summarized you need an environment where the folder /home/vagrant/venv/ans_py311/lib/python3.11/site-packages/ansible_collections is removed to prevent performance/memory issues during documentation generation.

Can't you just install ansible-core without ansible? I don't see the point in installing ansible just to manually rm it.

@wlitke
Copy link
Author

wlitke commented Aug 7, 2023

@felixfontein Awesome! Thank you very much for your proactive support.

@gotmax23 Yes, you could be right. I haven't tried the approach yet since I wanted to avoid the maintenance of multiple ansible installations (meaning an installation for the regular ansible usage and an installation for the creation of documentations). The temporary folder rename (better than folder removal) is a feasible approach to avoid multiple installations. Please correct me if I miss something or if I missunderstood your suggestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants