Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignoring a description for all languages makes it pointless to run the test at all #5631

Open
andreaTP opened this issue Oct 18, 2024 · 1 comment
Labels
status:waiting-for-author-feedback Issue that we've responded but needs author feedback to close type:question An issue that's a question

Comments

@andreaTP
Copy link
Contributor

Here:

"Language": "all",

the Github API IT test is ignored for all the languages, and it breaks the expectation that the description is actually tested.

Either:

The best would the latter.

@baywet
Copy link
Member

baywet commented Oct 18, 2024

Thanks for pointing this out, after working on #5610 I noticed a couple of things went wrong:

  • "all" suppressions are too broad and often mask other errors in other languages: This is now mitigated by the fact that we don't have any of them anymore. We could also remove the parsing ability for all in the future to prevent people from using them.
  • Outdated language suppressions: effectively the linked issues had been fixed, but never removed. We could probably automate checks like that to parse the configuration file, and fail if a suppression links to an issue that is closed.
  • Language suppressions might mask other issues: we often have things like "/path/a" creates one kind of issue which we suppressed, but a regression introduces a breakage in "/path/b" instead. Moving from "global" language suppressions to only path exclusions with rationale might help with that. I don't know how much work it'd represent though.
  • For some reasons, suppressions meant for the github API ended up on the petstore API. This is fixed now.
  • Generally the defects that are used for suppressions don't end up being fixed. This is more of an organizational issue, coupled with other priorities. We've had yet another re-org last month or so, and this ones seems to improve this front: we're able to fix issues faster than they get created.

I hope this context helps. Let me know if I missed anything.

@baywet baywet added status:waiting-for-author-feedback Issue that we've responded but needs author feedback to close type:question An issue that's a question labels Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status:waiting-for-author-feedback Issue that we've responded but needs author feedback to close type:question An issue that's a question
Projects
Status: Proposed 💡
Development

No branches or pull requests

2 participants