refactor(ci): transform conformance snapshot include when output mismatch

[overlookmotel]

Include "Output mismatch" in transform conformance snapshots when a test also fails due to scopes/symbols mismatches.

At present, many of the tests are failing on scopes/symbols mismatches, so it's hard to see from snapshots which also have an output mismatch. In particular, when working to fix scope mismatches, it's hard to notice if changes you make cause the output to be wrong. These extra messages make that visible.

[overlookmotel]

• refactor(ci): transform conformance snapshot include when output mismatch #5135 👈
• fix(ci): transform conformance do not skip any errors #5134
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on Graphite

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[codspeed-hq]

CodSpeed Performance Report

Merging #5135 will not alter performance

_{Comparing 08-23-refactor_ci_transform_conformance_snapshot_include_when_output_mismatch (eb71a32) with main (36e4a28)}

Summary

✅ 29 untouched benchmarks

[Dunqing]

Personally, if the output doesn't match we shouldn't show the semantic errors.

[graphite-app]

Merge activity

• Aug 24, 12:50 AM EDT: The merge label 'merge' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.
• Aug 24, 12:50 AM EDT: Boshen added this pull request to the Graphite merge queue.
• Aug 24, 12:58 AM EDT: Boshen merged this pull request with the Graphite merge queue.

[overlookmotel]

@Dunqing Yes maybe that would be a better way to do it. Check that output matches first, and then only check that semantic is correct if the output match test passes. You think that'd work?

[Dunqing]

@Dunqing Yes maybe that would be a better way to do it. Check that output matches first, and then only check that semantic is correct if the output match test passes. You think that'd work?

Yes, I agree we do that. From my experience development of the transformer. When the output is matched, fixing semantic errors is not too hard