ci: periodically run the unit tests of all GitHub-hosted published charms #1365
Conversation
|
Nice! Do you have an example run? |
Yes, on my fork. There are a few failures:
|
Co-authored-by: Ben Hoyt <benhoyt@gmail.com>
|
Rebased the PR and slightly improved it, and posted in Matrix to see if someone can work with me to get it merged.
Ah, this is the one about the change for relation-broken not including the broken relation in the list any more. That's actually the Juju behaviour by now, so the charm really ought to be upgraded. This is non-trivial, but I'll take a look tomorrow and see if I can figure it out in an hour or so. |
| @@ -0,0 +1,132 @@ | |||
| #! /usr/bin/env python | |||
There was a problem hiding this comment.
It wasn't obvious to me that this script needed to be run manually 🙈
There was a problem hiding this comment.
What would you recommend here? I could add a section to CONTRIBUTING.md that explains that this exists and how to update it, or I could just add that in the docstring for the script itself. Or I guess it could be run automatically on a schedule, which is probably cleanest, but I wasn't sure if we were sold enough on this idea to do that straight off.
There was a problem hiding this comment.
Yeah, I think manual updates are a good start. We can always run this automatically later if desired.
.github/workflows/tiobe.yaml
Outdated
| @@ -3,6 +3,7 @@ name: TIOBE Quality Checks | |||
| on: | |||
| schedule: | |||
| - cron: '0 7 1 * *' | |||
| workflow_dispatch: | |||
There was a problem hiding this comment.
Yeah, I'm not sure what the story is with this, because main@HEAD has:
on:
workflow_dispatch:
schedule:
- cron: '0 7 1 * *'So I don't understand why there's no complaint about a conflict, or the workflow_dispatch line being both above and below (and I could just remove the below one, which I must have had in an earlier version in my fork). I'll try to figure out what's happening here.
| @@ -0,0 +1,132 @@ | |||
| #! /usr/bin/env python | |||
There was a problem hiding this comment.
Yeah, I think manual updates are a good start. We can always run this automatically later if desired.
Once a month, before the typical release time, run a workflow that tests all charms published on CharmHub where the source location is in the charm metadata and that location is in a
canonicalGitHub repository.This is much the same as the existing data, hello, and observability tests, but at a larger scale and without the expectation that all of the tests would pass in every PR. The intention is that this gives us an insight into when ops changes break the most important charms - not that we would necessarily ensure that the tests were always passing (but we would at least look into it).
This also tests the
charmcraft initprofiles (other than the framework profiles, which don't start out with usable tests) to detect breakage there.To avoid manually maintaining the list of charms, a script is added that will update the workflow file with the latest list, pulled from CharmHub. This is not currently automatically run.