Katello: Pulp-To-Pulp sync ends in 403 error

[ianballou]

Version
Katello versions:

  "versions": [
    {
      "component": "core",
      "version": "3.28.5",
      "package": "pulpcore",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.22.3",
      "package": "pulp-rpm",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.10.0",
      "package": "pulp-python",
      "domain_compatible": false
    },
    {
      "component": "ostree",
      "version": "2.1.1",
      "package": "pulp-ostree",
      "domain_compatible": false
    },
    {
      "component": "file",
      "version": "1.14.3",
      "package": "pulp-file",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "2.21.1",
      "package": "pulp_deb",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.15.2",
      "package": "pulp-container",
      "domain_compatible": false
    },
    {
      "component": "certguard",
      "version": "1.6.5",
      "package": "pulp-certguard",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.18.0",
      "package": "pulp-ansible",
      "domain_compatible": false
    }
  ],

Smart proxy 1 versions:

{
  "versions": [
    {
      "component": "core",
      "version": "3.28.10",
      "package": "pulpcore",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.22.3",
      "package": "pulp-rpm",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.10.0",
      "package": "pulp-python",
      "domain_compatible": false
    },
    {
      "component": "ostree",
      "version": "2.1.1",
      "package": "pulp-ostree",
      "domain_compatible": false
    },
    {
      "component": "file",
      "version": "1.14.3",
      "package": "pulp-file",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "2.21.1",
      "package": "pulp_deb",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.15.2",
      "package": "pulp-container",
      "domain_compatible": false
    },
    {
      "component": "certguard",
      "version": "1.6.5",
      "package": "pulp-certguard",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.18.0",
      "package": "pulp-ansible",
      "domain_compatible": false
    }
  ],

Smart proxy 2 versions:

{
  "versions": [
    {
      "component": "core",
      "version": "3.22.2",
      "package": "pulpcore"
    },
    {
      "component": "rpm",
      "version": "3.19.0",
      "package": "pulp-rpm"
    },
    {
      "component": "python",
      "version": "3.8.0",
      "package": "pulp-python"
    },
    {
      "component": "ostree",
      "version": "2.0.0a6",
      "package": "pulp-ostree"
    },
    {
      "component": "file",
      "version": "1.11.3",
      "package": "pulp-file"
    },
    {
      "component": "deb",
      "version": "2.20.1",
      "package": "pulp_deb"
    },
    {
      "component": "container",
      "version": "2.14.3",
      "package": "pulp-container"
    },
    {
      "component": "certguard",
      "version": "1.5.6",
      "package": "pulp-certguard"
    },
    {
      "component": "ansible",
      "version": "0.16.0",
      "package": "pulp-ansible"
    }
  ],

Describe the bug
On both smart proxy one and smart proxy 2 (versions above) in Katello, we hit the following error when syncing ansible repositories from one Pulp to another:

Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]: pulp [4b763587-8712-4679-90de-f6e099d11c3d]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 801df179-2c8d-4c56-bd00-e932dcb0317a
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-api[4461]: pulp [4b763587-8712-4679-90de-f6e099d11c3d]:  - - [16/Aug/2023:17:10:01 +0000] "GET /pulp/api/v3/tasks/801df179-2c8d-4c56-bd00-e932dcb0317a/ HTTP/1.1" 200 680 "-" "OpenAPI-Generator/3.28.10/ruby"
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://centos8-katello-devel-stable.example.com/pulp_ansible/galaxy/Default_Organization/Library/custom/test/test_collection/api/v3/collections/all/'))
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]: pulp [4b763587-8712-4679-90de-f6e099d11c3d]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://centos8-katello-devel-stable.example.com/pulp_ansible/galaxy/Default_Organization/Library/custom/test/test_collection/api/v3/collections/all/'))
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]: pulp [4b763587-8712-4679-90de-f6e099d11c3d]: pulpcore.tasking.pulpcore_worker:INFO: Task 801df179-2c8d-4c56-bd00-e932dcb0317a failed ('ClientResponseError' object has no attribute 'path')
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]: pulp [4b763587-8712-4679-90de-f6e099d11c3d]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 450, in _perform_task
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     result = func(*args, **kwargs)
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 180, in sync
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     repo_version = d_version.create()
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     loop.run_until_complete(pipeline)
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     return future.result()
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     await asyncio.gather(*futures)
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     await self.run()
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 962, in run
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     await self._download_unpaginated_metadata()
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 817, in _download_unpaginated_metadata
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     collection_metadata_list = parse_metadata(col_results)
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:   File "/usr/lib/python3.9/site-packages/pulp_ansible/app/tasks/utils.py", line 97, in parse_metadata
Aug 16 17:10:01 centos8-proxy-devel2.sajha.example.com pulpcore-worker-2[4704]:     with open(download_result.path) as fd:

To Reproduce
Sync any ansible repo from Katello to a smart proxy (using Foreman).

Remote on the smart proxy 2 for example:

vagrant@centos8-katello-devel ~/katello $ pulp --profile proxy-2  ansible remote list
[
  {
    "pulp_href": "/pulp/api/v3/remotes/ansible/collection/832bef2b-d3f4-44cd-852d-fac5ad84f79c/",
    "pulp_created": "2023-08-16T18:39:37.155741Z",
    "name": "1-Ansible_View-Precipitation-109dd0c5-089d-48e0-81b9-d076cfb00b9f",
    "url": "https://katello.example.com/pulp_ansible/galaxy/Default_Organization/Precipitation/Ansible_View/custom/Buttermilk_Pancakes/ansible_repo/api/",
    "ca_cert": "-----BEGIN CERTIFICATE-----\nblah\n-----END CERTIFICATE-----",
    "client_cert": "-----BEGIN CERTIFICATE-----\nblah\n-----END CERTIFICATE-----",
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2023-08-16T18:39:37.155772Z",
    "download_concurrency": null,
    "max_retries": null,
    "policy": "immediate",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": null,
    "rate_limit": null,
    "hidden_fields": [
      {
        "name": "client_key",
        "is_set": true
      },
      {
        "name": "proxy_username",
        "is_set": false
      },
      {
        "name": "proxy_password",
        "is_set": false
      },
      {
        "name": "username",
        "is_set": false
      },
      {
        "name": "password",
        "is_set": false
      },
      {
        "name": "token",
        "is_set": false
      }
    ],
    "requirements_file": null,
    "auth_url": null,
    "sync_dependencies": true,
    "signed_only": false
  }
]

Expected behavior
Sync succeeds

Additional context
We just upgraded from Pulpcore 3.22 / pulp_ansible 0.16 to 3.28 and 0.18... were there any new features that would make the "upstream" Pulp ansible content inaccessible even with certificates set up on the remote?