Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix unused snapshots not triggering failure in CI #128162

Merged
merged 2 commits into from
Oct 26, 2024

Conversation

epenet
Copy link
Contributor

@epenet epenet commented Oct 11, 2024

Proposed change

syrupy is currently unable to detect unused snapshots when it is run in conjuction with xdist
It is a know existing limitation of syrupy https://github.com/syrupy-project/syrupy?tab=readme-ov-file#known-limitations

See also:
syrupy-project/syrupy#535

I have started work on implementing a clean version upstream: syrupy-project/syrupy#901, but I recommend that we add this workaround temporarily, until it is finalised there.

Sample failed run:
https://github.com/home-assistant/core/actions/runs/11380993290/job/31661830329

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

@epenet epenet changed the title Set pytest numprocesses to 1 Fix unused snapshots not triggering failure in CI Oct 11, 2024
@mib1185
Copy link
Contributor

mib1185 commented Oct 11, 2024

unfortunately, I think reverting #127736 is the more reasonable option in this case, since not detecting unused snapshots while running with pytest-xdist is a known limitation of syrupy (syrupy-project/syrupy#668)

@epenet
Copy link
Contributor Author

epenet commented Oct 16, 2024

Aaargghh. This works for me locally - but still fails to catch them inside CI
image

@epenet epenet marked this pull request as ready for review October 17, 2024 07:09
Comment on lines 158 to 156
# Override default finish to detect unused snapshots despite xdist
SnapshotSession.finish = override_syrupy_finish
Copy link
Member

@frenck frenck Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like a workaround for an upstream issue?

I honestly really don't think this belongs in our codebase.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like a workaround for an upstream issue?

Yes it is a workaround for an upstream issue.
There is a PR open there, but it fills a gap right now, see #128477

I honestly really don't think this belongs in our codebase.

I have added a comment to indicate that it can be removed when upstream is implemented

@home-assistant home-assistant bot marked this pull request as draft October 17, 2024 07:11
@home-assistant
Copy link

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

@epenet epenet mentioned this pull request Oct 17, 2024
19 tasks
@epenet epenet force-pushed the 20241011-1431 branch 3 times, most recently from a4429d3 to 7b12941 Compare October 17, 2024 08:13
@epenet
Copy link
Contributor Author

epenet commented Oct 17, 2024

@frenck, this now passes CI (as #128549 has been merged)

Would you consider adding this as a temporary workaround until the upstream issue is resolved?

@epenet epenet marked this pull request as ready for review October 20, 2024 09:17
@epenet epenet requested a review from a team as a code owner October 20, 2024 09:17
@home-assistant home-assistant bot requested a review from frenck October 20, 2024 09:17
@frenck
Copy link
Member

frenck commented Oct 26, 2024

@frenck, this now passes CI (as #128549 has been merged)

Would you consider adding this as a temporary workaround until the upstream issue is resolved?

Makes sense 👍

Copy link
Member

@frenck frenck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @epenet 👍

../Frenck

@frenck frenck merged commit e888a95 into home-assistant:dev Oct 26, 2024
46 checks passed
@epenet epenet deleted the 20241011-1431 branch October 26, 2024 06:45
@jbouwh
Copy link
Contributor

jbouwh commented Oct 26, 2024

This fix seems to cause a performance regression as a side effect.

E.g.:
pytest -n auto tests/components/mqtt/

This takes 25.75 sec without this fix on my dev env. and 72.24 sec with.

When coverage is tested too, it gets worse:
This takes 46.24 sec without this fix and 189.09 sec with.

@frenck
Copy link
Member

frenck commented Oct 26, 2024

Than we need to revert 👍

@epenet
Copy link
Contributor Author

epenet commented Oct 26, 2024

Happy to revert - but I had checked overall run time and didn't see such discrepancy.

jbouwh pushed a commit that referenced this pull request Oct 26, 2024
Revert "Fix unused snapshots not triggering failure in CI (#128162)"

This reverts commit e888a95.
@github-actions github-actions bot locked and limited conversation to collaborators Oct 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants