[Bug] CI: Too many requests to GitHub by g.extension

[wenzeslaus]

Describe the bug

g.extension CI fails a lot with messages such as "rate limit exceeded" and "429" (Too Many Requests).

To Reproduce

Does not happen all the time, but often enough. Merge a PR or see if there some failing tests already on main (or see previous run attempts in details). Likely the same for PRs.

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

Example 1

Running ./scripts/g.extension/testsuite/test_addons_download.py...
========================================================================
WARNING: Bug in UI description. Missing module description
....F.F.
======================================================================
FAIL: test_github_install_official_multimodule (__main__.TestModuleDownloadFromDifferentSources.test_github_install_official_multimodule)
Test installing multi-module extension from official addons repository
----------------------------------------------------------------------
Traceback (most recent call last):
  File "etc/python/grass/gunittest/case.py", line 1402, in assertModule
    module.run()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 823, in run
    self.wait()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 844, in wait
    raise CalledModuleError(
grass.exceptions.CalledModuleError: Module run `g.extension extension=i.sentinel operation=add prefix=testreport/scripts/g.extension/test_addons_download/gextension_test_install_path` ended with an error.
The subprocess ended with a non-zero return code: 1. See the following errors:
b'WARNING: The download of the json file with add-ons paths from the github server wasn\'t successful, rate limit exceeded. The previous downloaded json file  will be used if exists.\nFetching <i.sentinel> from <https://github.com/OSGeo/grass-addons/> (be patient)...\nAlready on \'grass8\'\nCompiling...\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.coverage> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.download> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.import> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.mask> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.parallel.download> addon html manual page.\nWARNING: The download of the commit from the GitHub API server wasn\'t successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.preproc> addon html manual page.\nInstalling...\nUpdating extensions metadata file...\nUpdating extension modules metadata file...\nWARNING: The download of the json file with add-ons paths from the github server wasn\'t successful, rate limit exceeded. The previous downloaded json file  will be used if exists.\nTraceback (most recent call last):\n  File "scripts/g.extension", line 2928, in <module>\n    sys.exit(main())\n             ^^^^^^\n  File "scripts/g.extension", line 2908, in main\n    install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)\n  File "scripts/g.extension", line 1300, in install_extension\n    install_module_xml(new_modules)\n  File "scripts/g.extension", line 1610, in install_module_xml\n    mlist = filter_multi_addon_addons(\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "scripts/g.extension", line 1588, in filter_multi_addon_addons\n    for addon in get_multi_addon_addons_which_install_only_html_man_page():\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "scripts/g.extension", line 1550, in get_multi_addon_addons_which_install_only_html_man_page\n    with open(addons_paths_file) as f:\n         ^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory: \'testreport/scripts/g.extension/test_addons_download/gextension_test_install_path/addons_paths.json\'\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/g.extension/testsuite/test_addons_download.py", line 156, in test_github_install_official_multimodule
    self.assertModule(
  File "etc/python/grass/gunittest/case.py", line 1422, in assertModule
    self.fail(self._formatMessage(msg, stdmsg))
AssertionError: Running <g.extension> module ended with non-zero return code (1)
Called (Python): g.extension(extension='i.sentinel', operation='add', prefix='testreport/scripts/g.extension/test_addons_download/gextension_test_install_path')
Called (Bash): g.extension extension=i.sentinel operation=add prefix=testreport/scripts/g.extension/test_addons_download/gextension_test_install_path
See the following errors:
WARNING: The download of the json file with add-ons paths from the github server wasn't successful, rate limit exceeded. The previous downloaded json file  will be used if exists.
Fetching <i.sentinel> from <https://github.com/OSGeo/grass-addons/> (be patient)...
Already on 'grass8'
Compiling...
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.coverage> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.download> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.import> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.mask> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.parallel.download> addon html manual page.
WARNING: The download of the commit from the GitHub API server wasn't successful, <rate limit exceeded>. Commit and commit date will not be included in the <i.sentinel.preproc> addon html manual page.
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...
WARNING: The download of the json file with add-ons paths from the github server wasn't successful, rate limit exceeded. The previous downloaded json file  will be used if exists.
Traceback (most recent call last):
  File "scripts/g.extension", line 2928, in <module>
    sys.exit(main())
             ^^^^^^
  File "scripts/g.extension", line 2908, in main
    install_extension(source=source, url=url, xmlurl=xmlurl, branch=branch)
  File "scripts/g.extension", line 1300, in install_extension
    install_module_xml(new_modules)
  File "scripts/g.extension", line 1610, in install_module_xml
    mlist = filter_multi_addon_addons(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "scripts/g.extension", line 1588, in filter_multi_addon_addons
    for addon in get_multi_addon_addons_which_install_only_html_man_page():
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "scripts/g.extension", line 1550, in get_multi_addon_addons_which_install_only_html_man_page
    with open(addons_paths_file) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'testreport/scripts/g.extension/test_addons_download/gextension_test_install_path/addons_paths.json'


======================================================================
FAIL: test_github_official_module_man_page_src_code_links_exists (__main__.TestModuleDownloadFromDifferentSources.test_github_official_module_man_page_src_code_links_exists)
Test if the installed extension HTML manual page from the
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/g.extension/testsuite/test_addons_download.py", line 210, in test_github_official_module_man_page_src_code_links_exists
    self.assertTrue(url)
AssertionError: '' is not true

----------------------------------------------------------------------
Ran 8 tests in 41.983s
FAILED (failures=2)
========================================================================
FAILED ./scripts/g.extension/testsuite/test_addons_download.py (2 tests failed)

Example 2

Running ./scripts/g.extension/testsuite/test_addons_download.py...
========================================================================
WARNING: Bug in UI description. Missing module description
......F.
======================================================================
FAIL: test_github_official_module_man_page_src_code_links_exists (__main__.TestModuleDownloadFromDifferentSources)
Test if the installed extension HTML manual page from the
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scripts/g.extension/testsuite/test_addons_download.py", line 218, in test_github_official_module_man_page_src_code_links_exists
    self.assertEqual(response, 200)
AssertionError: 429 != 200

----------------------------------------------------------------------
Ran 8 tests in 66.609s
FAILED (failures=1)
========================================================================
FAILED ./scripts/g.extension/testsuite/test_addons_download.py (1 test failed)

System description

• Operating System: Linux, macOS
• GRASS GIS version: main branch

[tmszi]

This should be fixed with #2717, and may be closed.