Add golden tests for public configs

[fendor]

Changes to the vscode schema need to be communicated to vscode-haskell plugin maintainers, otherwise users can't make use of the new configurations.

In general, changes to the schema need to be done consciously when new plugin or features are added. We add these golden tests as an additional contract to inform relevant parties whenever the configs change.

To fix a failing of these tests, review the change. If it is expected, accept the change via:

TASTY_PATTERN="generate schema" cabal test func-test --test-options=--accept

As changes need to be applied for all GHC version specific configs, you either need to run this command for each GHC version that is affected by the config change, or manually add the change to all other golden config files. Likely, the easiest way is to run CI and apply the generated diffs manually.

[fendor]

This should make it much harder for us to miss updates for vscode-haskell as the testdata would be modified since the latest release.

For example:

git log 2.5.0.0..HEAD -- test/testdata/schema

would give us all changes to the public interface.

However, this might be really annoying for new plugin maintainers and maybe even release managers.

[michaelpj]

This is a good idea and I like it.

I would like to do more in this direction, e.g. generate the lists of supported plugins from this and then have a test that the table in the docs agrees and so on. This is a good start, though.

[michaelpj]

Failures are hard to read, better to use one of the goldenWithDiff functions so we get a diff. Also it looks like maybe something is not available on windows, and so we get a config diff there? 😱

[michaelpj]

I think we probably do want it to be the case that our "plugin support matrix" varies across the "GHC version" axis but not the "platform" axis. That would just be too much.

[fendor]

I think we probably do want it to be the case that our "plugin support matrix" varies across the "GHC version" axis but not the "platform" axis. That would just be too much.

I also don't think our plugin support matrix should be even allowed to be different per platform. A plugin that only works on a certain platform feels like a bad user experience.