Add support for Galaxy v3 API #45

felixfontein · 2023-04-08T13:06:08Z

Tested by building docs (with antsibull-docs) from a collection on Galaxy where galaxy_url = "https://beta-galaxy.ansible.com" is in antsibull-docs.cfg.

felixfontein · 2023-04-08T13:11:15Z

CC @samccann

felixfontein · 2023-04-08T20:26:50Z

Turns out the v3 API of beta-galaxy.ansible.com is incredibly slow. A simple curl -i 'https://beta-galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/?limit=50' takes 30 seconds for me, and that's just the first page (out of three at the moment).

The Galaxy v3 API versions list test is disabled since it is so incredibly slow. Just listing the first page takes 30 seconds (!). The total listing takes > 100 seconds.

gotmax23 · 2023-04-08T21:33:32Z

src/antsibull_core/galaxy.py

@@ -7,9 +7,11 @@

 from __future__ import annotations

+import asyncio
 import os.path
 import shutil
 import typing as t


There's some instances of old type hinting syntax. Do you have a reason for keeping those or should I point them all out?

The main reason is that this needs to be backported to stable-1, so it's easier to keep them and remove them later (after backporting). Otherwise I have to do the work twice.

src/antsibull_core/galaxy.py

gotmax23 · 2023-04-08T21:45:57Z

tests/functional/test_galaxy.py

+        cache_path = tmp_path_factory.mktemp('cache')
+        downloader = CollectionDownloader(
+            aio_session,
+            str(download_path),


Note to self: we should change this API to accept PathLike objects as well in a separate PR.

Fixed in ea50a8d.

gotmax23 · 2023-04-08T21:47:39Z

src/antsibull_core/galaxy.py

+class GalaxyContext:
+    server: str
+    version: GalaxyVersion
+    base_url: str
+
+    def __init__(self, server: str, version: GalaxyVersion, base_url: str) -> None:
+        self.server = server
+        self.version = version
+        self.base_url = base_url


This would be a good candidate for a dataclass. After adding import dataclasses:

Suggested change

class GalaxyContext:

server: str

version: GalaxyVersion

base_url: str

def __init__(self, server: str, version: GalaxyVersion, base_url: str) -> None:

self.server = server

self.version = version

self.base_url = base_url

@dataclasses.dataclass

class GalaxyContext:

server: str

version: GalaxyVersion

base_url: str

This is 3.7+, so not backportable. We can do this in a follow-up PR.

Ack. For what it's worth, there's a backport on PyPI that we could add for python_version<'3.7', but I don't think that's worth it.

src/antsibull_core/galaxy.py

…be called 'results' also for v3.

gotmax23

Ack. Thanks!

gotmax23 · 2023-04-09T13:28:18Z

src/antsibull_core/galaxy.py

+class GalaxyContext:
+    server: str
+    version: GalaxyVersion
+    base_url: str
+
+    def __init__(self, server: str, version: GalaxyVersion, base_url: str) -> None:
+        self.server = server
+        self.version = version
+        self.base_url = base_url


Ack. For what it's worth, there's a backport on PyPI that we could add for python_version<'3.7', but I don't think that's worth it.

gotmax23 · 2023-04-09T13:38:42Z

src/antsibull_core/galaxy.py

+        return cls(galaxy_server, version, base_url)
+
+
+_GALAXY_CONTEXT_CACHE: dict[str, t.Union[GalaxyContext, asyncio.Future]] = {}


This will need to be changed to t.Dict when backporting.

samccann · 2023-04-10T15:43:09Z

I forgot to mention this before but it's a known issue that the galaxy beta site is slow. That's one reason they haven't broadcast the announcement yet of things switching over. They are working on improving the performance for sure.

felixfontein · 2023-04-14T04:39:22Z

The curl request from above is down to 10 seconds now, BTW. Still way too slow, but hey, three times faster than a week ago...

patchback · 2023-04-14T04:40:12Z

Backport to stable-1: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply af8f28d on top of patchback/backports/stable-1/af8f28d394c63996909c8c7f1caa7f78a7994dc6/pr-45

Backporting merged PR #45 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/ansible-community/antsibull-core.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/stable-1/af8f28d394c63996909c8c7f1caa7f78a7994dc6/pr-45 upstream/stable-1

Now, cherry-pick PR Add support for Galaxy v3 API #45 contents into that branch:
```
$ git cherry-pick -x af8f28d394c63996909c8c7f1caa7f78a7994dc6
```
If it'll yell at you with something like fatal: Commit af8f28d394c63996909c8c7f1caa7f78a7994dc6 is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x af8f28d394c63996909c8c7f1caa7f78a7994dc6
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Add support for Galaxy v3 API #45 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/stable-1/af8f28d394c63996909c8c7f1caa7f78a7994dc6/pr-45

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

felixfontein · 2023-04-14T04:40:20Z

@gotmax23 thanks for reviewing this!

* Add support for Galaxy v3 API. * Add integration tests for the Galaxy API. The Galaxy v3 API versions list test is disabled since it is so incredibly slow. Just listing the first page takes 30 seconds (!). The total listing takes > 100 seconds. * Reorganize imports. * Improve docstrings; validate context.server == galaxy_server. * Fix check, update docstring. * According to ansible-galaxy CLI code, the result value can sometimes be called 'results' also for v3. (cherry picked from commit af8f28d)

* Add support for Galaxy v3 API (#45) * Add support for Galaxy v3 API. * Add integration tests for the Galaxy API. The Galaxy v3 API versions list test is disabled since it is so incredibly slow. Just listing the first page takes 30 seconds (!). The total listing takes > 100 seconds. * Reorganize imports. * Improve docstrings; validate context.server == galaxy_server. * Fix check, update docstring. * According to ansible-galaxy CLI code, the result value can sometimes be called 'results' also for v3. (cherry picked from commit af8f28d) * Make work with older environments.

felixfontein requested a review from gotmax23 April 8, 2023 13:11

felixfontein force-pushed the galaxy-v3 branch from 375b608 to bf64609 Compare April 8, 2023 20:52

felixfontein added 3 commits April 8, 2023 23:19

Add support for Galaxy v3 API.

7f270b1

Add integration tests for the Galaxy API.

7832084

The Galaxy v3 API versions list test is disabled since it is so incredibly slow. Just listing the first page takes 30 seconds (!). The total listing takes > 100 seconds.

Reorganize imports.

f1f3361

felixfontein force-pushed the galaxy-v3 branch from bf64609 to f1f3361 Compare April 8, 2023 21:20

gotmax23 reviewed Apr 8, 2023

View reviewed changes

felixfontein added the backport-1 Automatically create backport to stable-1 branch label Apr 9, 2023

felixfontein added 3 commits April 9, 2023 12:58

Improve docstrings; validate context.server == galaxy_server.

ea50a8d

Fix check, update docstring.

67f0062

According to ansible-galaxy CLI code, the result value can sometimes …

42efb43

…be called 'results' also for v3.

gotmax23 approved these changes Apr 9, 2023

View reviewed changes

felixfontein merged commit af8f28d into ansible-community:main Apr 14, 2023

felixfontein deleted the galaxy-v3 branch April 14, 2023 04:40

This was referenced Apr 14, 2023

[stable-1] Add support for Galaxy v3 API #55

Merged

Cleanup galaxy code with dataclasses, and use dict[] instead of t.Dict[] #56

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Galaxy v3 API #45

Add support for Galaxy v3 API #45

felixfontein commented Apr 8, 2023

felixfontein commented Apr 8, 2023

felixfontein commented Apr 8, 2023

gotmax23 Apr 8, 2023

felixfontein Apr 9, 2023

gotmax23 Apr 8, 2023

felixfontein Apr 9, 2023

gotmax23 Apr 8, 2023

felixfontein Apr 9, 2023

gotmax23 Apr 9, 2023

gotmax23 left a comment

gotmax23 Apr 9, 2023

gotmax23 Apr 9, 2023

samccann commented Apr 10, 2023

felixfontein commented Apr 14, 2023

patchback bot commented Apr 14, 2023 •

edited

Loading

felixfontein commented Apr 14, 2023

		return cls(galaxy_server, version, base_url)


		_GALAXY_CONTEXT_CACHE: dict[str, t.Union[GalaxyContext, asyncio.Future]] = {}

Add support for Galaxy v3 API #45

Add support for Galaxy v3 API #45

Conversation

felixfontein commented Apr 8, 2023

felixfontein commented Apr 8, 2023

felixfontein commented Apr 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gotmax23 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samccann commented Apr 10, 2023

felixfontein commented Apr 14, 2023

patchback bot commented Apr 14, 2023 • edited Loading

Backport to stable-1: 💔 cherry-picking failed — conflicts found

Backporting merged PR #45 into main

felixfontein commented Apr 14, 2023

patchback bot commented Apr 14, 2023 •

edited

Loading