Tests: Use YARP fixtures

[snutij]

Motivation

Closes: #448

Started with syntax-tree fixtures we decided during this PR to move forward and use YARP fixtures, which is a larger set.

Implementation

• Add as a submodule the YARP repository
• Edit CI Action to pull submodule dependency, see documentation
• Ensure before running bin/test that the submodule is well initialized.
• Then some fix for expectations_test_runner.rb:
  • include only ruby-lsp and YARP fixtures (and not other ruby files from YARP)
  • ensure unique test example name for YARP fixtures
  • rescue Rubocop errors as this is out of ruby-lsp control (I will report to them gradually)
• Finally, some fix for requests on ruby-lsp side, that fixtures point out.

Automated Tests

More than 14k tests now, all should be green or skipped with "Fixture requires a fix from Rubocop" message.

Manual Tests

• checkout this branch
• git submodule update --init --recursive
• bin/test

Then to have a complete stack trace of Rubocop fail: bin/rubocop -d path/for/fixture

[andyw8]

This looks great! Some of the failures might take a while to address, so it could be useful to have a way to mark those as known failures, which can be fixed incrementally in future.

[snutij]

• I know that in test/expectations/expectations_test_runner.rb we already skip some tests (based on ruby version requirement). Maybe we can do something similar for tests files included in a WAINTING_FOR_RUBOCOP_FIX array, for example.
• In the same time, I can open some issues on rubocop to report failing fixtures with the related stack trace, and keep an eye on it to update this array when fixes are released.

This is not ideal, but allow us to go forward, and already provide a lot more fixture. wdyt?

[andyw8]

Maybe we can do something similar for tests files included in a WAINTING_FOR_RUBOCOP_FIX array, for example.

That seems reasonable. I haven't looked in detail at the failures yet, are they all from RuboCop? I would have thought we might see some caused by bugs in ruby-lsp itself.

Btw, one thing I noticed when trying this locally: Initially everything was wrongly passing, because I hadn't run yet git submodule update --init, so test/fixtures/ruby-syntax-fixtures/ was empty. Maybe we should have a check that this is non-empty when running the tests?

[andyw8]

Also, @kddnewton suggested using https://github.com/ruby/yarp/tree/main/test/fixtures which is a larger set which includes all the syntax tree fixtures, but that could be done later.

[vinistock]

This is awesome and already showing value given that it caught many errors

[andyw8]

👋 After some more internal discussion:

• We would like to go with using https://github.com/ruby/yarp/tree/main/test/fixtures (still as a submodule) for testing against.
• Let's leave out the Dependabot configuration for now - the fixtures don't change too often, and we can manually pull in the latest when needed.
• For the failures due to InternalRuboCopError exceptions, let's report these upstream to RuboCop, but ignore them for our test suite, since they are outside of our control.
• For the failures that are due to ruby-lsp's code, let's add the breaking examples to our own fixtures. Feel free to try and fix those if you would like to, otherwise we can assist.

Thanks again for your contribution!

[snutij]

Thanks for coming back @andyw8. 🙏

For the first point, YARP fixtures are not a repository (but only a subfolder from a repository). If we want to use it as a submodule, we have to point to the full git project (and pull the full YARP repo). It's look not ideal to me, maybe I'm missing your point? Perhaps there is a way to do that, that I'm not aware of yet. 😄

For the rest of your comment, I'll do it 👍

[andyw8]

Yes we're ok with pulling the full YARP repo, it's fairly small. (@kddnewton said he may publish the fixtures to a separate repo at some point).

[snutij]

I moved from ruby-syntax-tree to YARP fixtures: between 40 and 60 errors depending on ruby version.

I'll dig asap on these errors to know which are on ruby-lsp side or not, to exclude correct files.

Please feel free to give feedback on the expectation_test_runner. I tried to make it simple, but I'm not sure if it worked. 😄

[vinistock]

There seems to be at least two failures that are also related to infinite loops in RuboCop when running formatting.

I think we should just use the same technique and ignore any RuboCop related errors by rescuing RubyLsp::Requests::Formatting::Error.

[snutij]

@vinistock thanks for your feedbacks 🙏

rubocop and syntax-tree errors are now rescue and skip. Older ruby version fail on diagnostics requests because of the early return in case of invalid syntax (which are not in newer version), so the T.cast() is not as expected (nil instead of T::Array[RubyLsp::Requests::Support::RuboCopDiagnostic]). I'm looking to fix it.

[snutij]

Well, CI is green 🎉 feel free to review it! Do you want a refresh on PR description? This is not any more exact, as we move on YARP fixture.

[vinistock]

This is amazing! Thank you very much for your work on this.

We'll need to wait for a Syntax Tree release with the fix, but it looks good to me.

[vinistock]

Let's not wait for RuboCop to fix stuff. We should start benefiting from the fixtures right away.

In RuboCop Runner, let's include StandardError as part of the rescued exceptions that we raise as InternalRuboCopError.

That should get the CI passing and we can merge.

[vinistock]

Thank you very much for pushing this over the finish line, I know it took a while.

Amazing contribution 🚀

[andyw8]

I wonder if we should update the Dependabot config to periodically pull in the submodule updates, it's supported as a package-ecosystem option:

https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file

[snutij]

@andyw8 yep we totally can, it was part of the first iteration based on SyntaxTree fixtures. If you want it, I can do it. 👍