feature wishlist

[keewis]

The action works pretty well already, but it is far from perfect.

Here's a list of features I'd like to have:

• at the moment, the CI will either open a new issue or update an issue that has the configured title and label. However, sometimes the issue stays open long enough that something else changes and we get a combined issue for all changes. Renaming the issue or removing the label allows working around that, but requires manual action. Instead, we might want to (optionally) open separate issues for each test file?
• in addition to tests, the CI might be able to collect DeprecationWarnings, PendingDeprecationWarnings, and FutureWarnings and present them in a separate section / issue? For that, pytest-reportlog would need to serialize warning events, though, and we might need to create a separate action, too.

[Zeitsperre]

All good ideas!

I would love to see an option in the action to automatically assign a specific user to the opened issue!

[keewis]

you mean, a constant list of users? Sure, that should not be too difficult (the key is assignees). We'd only have to make sure that not having assignees is still possible (and stays the default).

[agriyakhetarpal]

Hi, @keewis and @Zeitsperre! Thanks for writing this action – as it was recommended to us recently, we're trying to use it currently and I would like to share some feedback. If one were to use this in a matrix configuration, say, for running different test suites (unit, integration, notebooks, etc.), one can generate different logs, which I'm trying to generate issues with different issue titles for, instead of editing the existing issue. Would it be possible to uniquely identify existing issues with their titles before editing them?

Here is what I am trying right now: https://github.com/agriyakhetarpal/PyBOP/actions/runs/8748265833/workflow?pr=1#L52-L64
which just seems to edit the description for the same issue: agriyakhetarpal/PyBOP#2 and overwrite the logs.

I would say, having multiple issues for the same workflow run (even if it uses matrices or not) would help but isn't great either (because it could then open too many issues), so if the action can perhaps collate multiple JSONL files into one single file and then parse them, and add a new comment for any further error logs on an existing issue instead of editing the issue description, that would be really helpful. Another option could be to make this action a composite action and use https://github.com/peter-evans/create-issue-from-file to create the issue instead – doing this would be cleaner, and then to provide the parsing of the logs as a separate action that can be used with uses: xarray-contrib/issue-from-pytest-log/parse-log@main (but I'm not sure how much the latter would be in scope for this action). I would be very happy to contribute, too, and submit a patch for anything that is in scope, so please let me know.

[keewis]

(@Zeitsperre is a user just like you, at the moment I'm the only author / maintainer of this action, which is based on the original action of xarray)

Would it be possible to uniquely identify existing issues with their titles before editing them?

definitely, and you're not the first one to ask. I didn't have time to look into why this is happening, though. Can you open a new issue for that?

having multiple issues for the same workflow run (even if it uses matrices or not) would help but isn't great either (because it could then open too many issues)

I get that, but there's also the problem that issues have a limit on the amount of characters you can put into a comment. We're already summarizing if that limit would be exceeded (which is not ideal because we lose some information), and concatenating multiple CI would cause us to hit that limit even earlier. So if you want to put everything in the same issue the only way forward I can see is to put each different entry in the matrix in a separate comment, with a heading that contains the specific title. That would be a bigger undertaking, though, so for now I'd vote for fixing the renaming issue above.

[agriyakhetarpal]

definitely, and you're not the first one to ask. I didn't have time to look into why this is happening, though. Can you open a new issue for that?

Sure! I'll also investigate a fix for this (I hope it's easy enough).

I get that, but there's also the problem that issues have a limit on the amount of characters you can put into a comment. We're already summarizing if that limit would be exceeded (which is not ideal because we lose some information), and concatenating multiple CI would cause us to hit that limit even earlier.
So if you want to put everything in the same issue the only way forward I can see is to put each different entry in the matrix in a separate comment, with a heading that contains the specific title. That would be a bigger undertaking, though, so for now I'd vote for fixing the renaming issue above.

That's reasonable. Having breached the limit by adding long logs before elsewhere on GitHub, I think the limit is 65536 characters for comments. I would suppose a user-provided option for not showing the logs but just adding the link to the workflow run could help, but that defeats the purpose of the ease of convenience that's provided through the action... I'll look into the above issue, will have to browse through the GraphQL API reference and see if I can tinker something up.