Add new ecosystem comparison modes for the formatter

[zanieb]

Previously, the ecosystem checks formatted with the baseline then formatted again with --diff to get the changed files.

Now, the ecosystem checks support a new mode where we:

• Format with the baseline
• Commit the changes
• Reset to the target ref
• Format again
• Check the diff from the baseline commit

This effectively tests Ruff changes on unformatted code rather than changes in previously formatted code (unless, of course, the project is already using Ruff).

While this mode is the new default, I've retained the old one for local checks. The mode can be toggled with --format-comparison <type>.

Includes some more aggressive resetting of the GitHub repositories when cached.

Here, I've also stubbed comparison modes in which black is used as the baseline. While these do nothing here, #8419 adds support.

I tested this with the commit from #8216 and ecosystem changes appear https://gist.github.com/zanieb/a982ec8c392939043613267474471a6e

[github-actions]

PR Check Results

Ecosystem

✅ ecosystem check detected no linter changes.

✅ ecosystem check detected no format changes.

[MichaReiser]

Thank you. I love the different modes for running it locally

[MichaReiser]

Oh that's cool

[MichaReiser]

Python noob question: What's the reason for using inline comments here instead of docstrings?

[zanieb]

I don't think there's a formalized way to document class attributes individually. Generally what you do is create a class docstring and then document the members using an 'Attributes' section, but that felt like more work than I wanted to do here 😬

tldr a docstring for the class is proper but I was lazy. I wish we had tooling to generate the scaffold.

[zanieb]

I even just wrote it out and it feels awkward; I guess I write Rust now

class FormatComparison(Enum):
    """
    The type of format comparison to use.

    Attributes:
        ruff_then_ruff: Run Ruff baseline then Ruff comparison; checks for changes in
            behavior when formatting previously "formatted" code
        ruff_and_ruff: Run Ruff baseline then reset and run Ruff comparison; checks
            changes in behavior when formatting "unformatted" code
        black_then_ruff: Run Black baseline then Ruff comparison; checks for changes in
            behavior when formatting previously "formatted" code
        black_and_ruff: Run Black baseline then reset and run Ruff comparison; checks
            changes in behavior when formatting "unformatted" code
    """

[zanieb]

Some additional resources:

• https://peps.python.org/pep-0258/#attribute-docstrings
• Class attribute documentation? microsoft/pylance-release#1576

[zanieb]

Since you nudged, I switched to individual attribute docstrings — while the PEP is rejected, VSCode supports it.