Breaking: Validate test options (refs eslint/eslint#2179)

[btmills]

This is the other part of enabling automatic validation of rule options. With this change, eslint-tester will begin validating options in test cases, a process which has already caught a few bugs.

[nzakas]

We should export configvalidator so we don't have to do it this way, but OK for now.

[btmills]

I'll push an update to both PRs to do it that way.

[nzakas]

Looks good to me. What happens when a rule has no schema?

@ilyavolodin thoughts?

[btmills]

What happens when a rule has no schema?

If you look at config-validator line 25, it will provide a default schema which validates only that the first value is either 0, 1, or 2 and allows any other options.

[ilyavolodin]

Looks good to me. My only concern is the order in which we need to merge this in between eslint-tester and eslint core.

[btmills]

Core then tester. Core will work without the changes in tester, but but test case options won't be validated, so we'll need to be vigilant and not let any changes to options sneak through until tester is updated.

[nzakas]

So we need to do a new ESLint release before we can release the tester?

[btmills]

So we need to do a new ESLint release before we can release the tester?

Correct, unless you want me to do a two-phase update for tester that only conditionally tests validation in the first phase, then makes it always-on in the second?

[ilyavolodin]

So we could merge this in, merge the one in core, and then make sure that we release eslint-tester right before we do next release for core? Would that work?

[btmills]

So I think, taking all constraints into account, it should look like this:

1. Merge Breaking: Automatically validate rule options (fixes #2595) eslint#2685
2. Merge Breaking: Validate test options (refs eslint/eslint#2179) #21
3. Release core
4. Release tester

If we consider ourselves at step 0 right now, tests in both core and tester are both passing. The change in tester depends on the change in core. Though the reverse is not true, if we don't catch them, changes to options could slip through in core that would break core's CI when tester is released. In theory nothing should break in between steps so there aren't technical time constraints, but minimizing the time between now and the end of step 4 would reduce the chance that something gets missed.

[nzakas]

OK, 1 is done.

[btmills]

Just confirmed that eslint v0.23 is still working with this branch of eslint-tester. Are we good to merge this for a release?

[ilyavolodin]

Yes, I think we should be good. Should I merge this in and do a release of eslint-tester?

[btmills]

Yep! 🚢