[Bug] FileNotFoundError for addons_paths.json

[wenzeslaus]

Describe the bug

I get traceback with FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.grass7/addons/addons_paths.json' when rate limit is exceeded to access GitHub.

To Reproduce

Rate limit related, so hard to reproduce. The command was:

grass --tmp-location XY --exec g.extension g.download.location --verbose

Expected behavior

No traceback, maybe an error message.

Screenshots

Output:

Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.extension g.download.location --verbose> ...
WARNING: The download of the json file with add-ons paths from the github
         server wasn't successful, rate limit exceeded. The previous
         downloaded json file will be used if exists.
Type of source identified as 'official'.
Fetching <g.download.location> from GRASS GIS Addons repository (be
patient)...
A    g.download.location
A    g.download.location/Makefile
A    g.download.location/g.download.location.html
A    g.download.location/g.download.location.py
Exported revision 16693.
Compiling...
mkdir -p /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/bin
mkdir -p /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/etc
mkdir -p /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/html
mkdir -p /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/man
mkdir -p /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/man/man1
mkdir -p /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/scripts
GISRC=/tmp/grass7-runner-31625/gisrc GISBASE=/home/runner/install/grass78 PATH="/home/runner/install/grass78/bin:/home/runner/install/grass78/bin:/home/runner/install/grass78/scripts:$PATH" PYTHONPATH="/home/runner/install/grass78/etc/python:/home/runner/install/grass78/gui/wxpython:$PYTHONPATH" LD_LIBRARY_PATH="/tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/bin:/home/runner/install/grass78/bin:/home/runner/install/grass78/scripts:/home/runner/install/grass78/lib:/home/runner/install/grass78/lib:/home/runner/install/grass78/lib:/home/runner/install/lib" LC_ALL=C LANG=C LANGUAGE=C g.parser -t g.download.location.py | sed s/\"/\\\\\"/g | sed 's/.*/_("&")/' > /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/g.download.location_to_translate.c
/usr/bin/install -c  g.download.location.py /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/scripts/g.download.location
if [ "/tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/scripts/g.download.location" != "" ] ; then GISRC=/tmp/grass7-runner-31625/gisrc GISBASE=/home/runner/install/grass78 PATH="/home/runner/install/grass78/bin:/home/runner/install/grass78/bin:/home/runner/install/grass78/scripts:$PATH" PYTHONPATH="/home/runner/install/grass78/etc/python:/home/runner/install/grass78/gui/wxpython:$PYTHONPATH" LD_LIBRARY_PATH="/tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/bin:/home/runner/install/grass78/bin:/home/runner/install/grass78/scripts:/home/runner/install/grass78/lib:/home/runner/install/grass78/lib:/home/runner/install/grass78/lib:/home/runner/install/lib" LC_ALL=C LANG=C LANGUAGE=C /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/scripts/g.download.location --html-description < /dev/null | grep -v '</body>\|</html>' > g.download.location.tmp.html ; fi
VERSION_NUMBER=7.8.8dev VERSION_DATE=2022 MODULE_TOPDIR=/home/runner/install/grass78 \
        python3 /home/runner/install/grass78/tools/mkhtml.py g.download.location > /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/html/g.download.location.html
WARNING: The download of the commit from the GitHub API server wasn't
         successful, <rate limit exceeded>. Commit and commit date will not
         be included in the <g.download.location> addon html manual page.
VERSION_NUMBER=7.8.8dev /home/runner/install/grass78/tools/g.html2man.py "/tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/html/g.download.location.html" "/tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/man/man1/g.download.location.1"
rm g.download.location.tmp.html
Installing...
/usr/bin/install -c  /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/scripts/g.download.location /home/runner/.grass7/addons/scripts/
/usr/bin/install -c  -m 644 /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/html/g.download.location.html /home/runner/.grass7/addons/docs/html/
if [ -n "" ] ; then \
	/usr/bin/install -c  -m 644   /home/runner/.grass7/addons/docs/html/ ; \
fi
/usr/bin/install -c  -m 644 /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/docs/man/man1/g.download.location.1 /home/runner/.grass7/addons/docs/man/man1/
if [ -d "/tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/etc/g.download.location" ] ; then \
	cp -RL /tmp/grass7-runner-31625/tmp9t2jxy98/g.download.location/etc/g.download.location /home/runner/.grass7/addons/etc/ ; \
fi
Updating extensions metadata file...
Updating extension modules metadata file...
Manual page for <g.download.location> updated
WARNING: The download of the json file with add-ons paths from the github
         server wasn't successful, rate limit exceeded. The previous
         downloaded json file will be used if exists.
Traceback (most recent call last):
  File "/home/runner/install/grass78/scripts/g.extension", line 2679, in <module>
    sys.exit(main())
  File "/home/runner/install/grass78/scripts/g.extension", line 2657, in main
    install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)
  File "/home/runner/install/grass78/scripts/g.extension", line 1035, in install_extension
    update_manual_page(module)
  File "/home/runner/install/grass78/scripts/g.extension", line 2225, in update_manual_page
    for a in get_multi_addon_addons_which_install_only_html_man_page():
  File "/home/runner/install/grass78/scripts/g.extension", line 1283, in get_multi_addon_addons_which_install_only_html_man_page
    with open(addons_paths_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.grass7/addons/addons_paths.json'
Execution of <g.extension g.download.location --verbose> finished.
Cleaning up temporary files...

System description (please complete the following information):

• Operating System: Linux, Ubuntu 20.04, GitHub CI
• GRASS GIS version 7.8 (branch: releasebranch_7_8)

Additional context

I did not investigate if this is applicable to v8.

[tmszi]

We don't have backported this PR #2140. releasebranch_8_2 is not affected by this bug.

Affected is the main branch (8.3.dev) too, but bug does not appear regularly. This may be related to how the number of allowed GitHub API requests is evaluated in CI ENV workflows (number of allowed requests per CI workflow unit or sum of all CI workflows units etc.)

Running ./scripts/g.extension/testsuite/test_addons_download.py...
========================================================================
WARNING: Bug in UI description. Missing module description
...F.
======================================================================
FAIL: test_github_install_official_multimodule (__main__.TestModuleDownloadFromDifferentSources)
Test installing multi-module extension from official addons repository
----------------------------------------------------------------------
Traceback (most recent call last):
  File "etc/python/grass/gunittest/case.py", line 1403, in assertModule
    module.run()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 823, in run
    self.wait()
grass.exceptions.CalledModuleError: Module run `g.extension extension=i.sentinel operation=add prefix=testreport/scripts/g.extension/test_addons_download/gextension_test_install_path` ended with an error.
The subprocess ended with a non-zero return code: 1. See the following errors:
b'WARNING: The download of the json file with add-ons paths from the github server wasn\'t successful, rate limit exceeded. The previous downloaded json file  will be used if exists.\nFetching <i.sentinel> from GRASS GIS Addons repository (be patient)...\nCompiling...\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.coverage> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.download> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.import> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.mask> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.parallel.download> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.preproc> addon html manual page.\nInstalling...\nUpdating extensions metadata file...\nUpdating extension modules metadata file...\nWARNING: The download of the json file with add-ons paths from the github server wasn\'t successful, rate limit exceeded. The previous downloaded json file  will be used if exists.\nTraceback (most recent call last):\n  File "scripts/g.extension", line 2670, in <module>\n    sys.exit(main())\n  File "scripts/g.extension", line 2650, in main\n    install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)\n  File "scripts/g.extension", line 1048, in install_extension\n    install_module_xml(new_modules)\n  File "scripts/g.extension", line 1359, in install_module_xml\n    mlist = filter_multi_addon_addons(\n  File "scripts/g.extension", line 1337, in filter_multi_addon_addons\n    for addon in get_multi_addon_addons_which_install_only_html_man_page():\n  File "scripts/g.extension", line 1299, in get_multi_addon_addons_which_install_only_html_man_page\n    with open(addons_paths_file) as f:\nFileNotFoundError: [Errno 2] No such file or directory: \'testreport/scripts/g.extension/test_addons_download/gextension_test_install_path/addons_paths.json\'\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/g.extension/testsuite/test_addons_download.py", line 149, in test_github_install_official_multimodule
    self.assertModule(
  File "etc/python/grass/gunittest/case.py", line 1423, in assertModule
    self.fail(self._formatMessage(msg, stdmsg))
AssertionError: Running <g.extension> module ended with non-zero return code (1)
Called (Python): g.extension(extension='i.sentinel', operation='add', prefix='testreport/scripts/g.extension/test_addons_download/gextension_test_install_path')
Called (Bash): g.extension extension=i.sentinel operation=add prefix=testreport/scripts/g.extension/test_addons_download/gextension_test_install_path
See the following errors:
WARNING: The download of the json file with add-ons paths from the github server wasn't successful, rate limit exceeded. The previous downloaded json file  will be used if exists.
Fetching <i.sentinel> from GRASS GIS Addons repository (be patient)...
Compiling...
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.coverage> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.download> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.import> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.mask> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.parallel.download> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.preproc> addon html manual page.
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...
WARNING: The download of the json file with add-ons paths from the github server wasn't successful, rate limit exceeded. The previous downloaded json file  will be used if exists.
Traceback (most recent call last):
  File "scripts/g.extension", line 2670, in <module>
    sys.exit(main())
  File "scripts/g.extension", line 2650, in main
    install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)
  File "scripts/g.extension", line 1048, in install_extension
    install_module_xml(new_modules)
  File "scripts/g.extension", line 1359, in install_module_xml
    mlist = filter_multi_addon_addons(
  File "scripts/g.extension", line 1337, in filter_multi_addon_addons
    for addon in get_multi_addon_addons_which_install_only_html_man_page():
  File "scripts/g.extension", line 1299, in get_multi_addon_addons_which_install_only_html_man_page
    with open(addons_paths_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'testreport/scripts/g.extension/test_addons_download/gextension_test_install_path/addons_paths.json'


----------------------------------------------------------------------
Ran 5 tests in 8.909s
FAILED (failures=1)
========================================================================
FAILED ./scripts/g.extension/testsuite/test_addons_download.py (1 test failed)

[wenzeslaus]

Evaluating this again, the issue is 1) the irregular failures and 2) that the result is a traceback. I get the the same for main and for 7.8 branch.

The issue number 1 makes the CI much less useful. I may get failures anytime, so I need to manually restart, but additionally I need to restart one by one to avoid another API limit hit. Having the code more forgiving towards rate exceeded, e.g., by more and increasingly longer re-tries might be one way how to cope with that limited resource if we need to keep using it.

The issue number 2 seems like an situation not addressed by the code. The rate limit was reached and, at the same time, the file does not exist. As long as there is the dependence on something with a rate limit, we need a really good message to tell what happened.