[test] use randomized runner in packaging tests

[andyb-elastic]

Use the randomized runner from the test framework and add some basic
logging to make the packaging tests behave more similarly to how we use
junit in the rest of the project

[elasticmachine]

Pinging @elastic/es-core-infra

[andyb-elastic]

The reason for creating this class and replicating some of the stuff in ESTestCase instead of using it directly is that there are some things that ESTestCase does (e.g. bootstrap checks) that don't play well with how we run these tests

[alpar-t]

Could we extend ESTestCase and disable what we don't need or doesn't work ?
Some things might still be good to have here, I'm thinking about especially the reproduction line,
but I guess that wouldn't really work either because of how the order of test methods is inter-dependent.
What is the advantage o f having ti like that ? Would it make sense to bring everything that needs to run in order under a single method ? Then we could run in truly random fashion and perhaps keep more functionality from the test framework. As it is I'm not fully convinced that it's worth adding it just to throw out most of what it does and without really making use of any randomness in these tests.

[andyb-elastic]

I should have made this more clear - the real purpose of doing this is so that these tests will read similarly to the other junit tests wrt method names and such. There isn't as far as I can tell a way to configure that with any vanilla junit runners

There is no randomization here because the method ordering matters. The order of the test classes (not methods) could be randomized, but I don't think randomized runner does that and I didn't figure out a way to have it handle the suite itself rather than using @RunWith(Suite.class)

I spent some time on making it work with ESTestCase a while ago the first time I tried this and disabling/working around/making configurable the things in there that don't play nice with these tests was nontrivial as I remember, and I'm not sure I was really successful. It seemed much more straightforward to do it with its own parent test case

Currently we get a repro line from the vagrant test plugin

[rjernst]

Additionally, using ESTestCase means bringing in elasticsearch as a dependency, which doesn't really make sense from a packaging test perspective (we won't make use of most of the test utilities provided there). We currently have elasticsearch as a dependency for the rest test runner, but it would be good to split that out so we can avoid the dependencies that ES pulls in (and can conflict with things we would like to do in the rest test runner, eg using the rest client).

[alpar-t]

You could look at GradleUnitTestCase it does the same by pulling int the randomized runner only.

What I was wondering about w.r.t order is that if it really makes sense to have it fixed. If all we are doing is going trough methods sequentially what advantage does it bring to have them in separate methods ? Maybe better error reporting ? Should we keep the randomized method order and make sure it actually works like that? I'm not saying we need to change it just looking to understand the implications.

[andyb-elastic]

That's right, the benefit of the method ordering is mostly just to make it a little easier to read where something went wrong. And to separate out parts that aren't directly related to each other so it doesn't end up as just one very long test

I'd ideally like to randomize them and reinstall the distribution before each one so it starts from a predictable state (and some of them definitely can work like this already) but it would add too much runtime to an already very long test suite

[andyb-elastic]

This will need to be updated and merged after #31943 is merged so that the test cases it adds can be updated too - the changes from there will be more of the same

[rjernst]

Thanks for doing this. I have a couple comments.

[rjernst]

I think you could pull in randomized runner on its own? Additionally, there is a JUnit3MethodProvider in randomized runner, can we use that instead of the lucene one?

[andyb-elastic]

That makes sense, I changed it back to use randomizedtesting instead of the framework - I'd added it when I was trying to make it work with ESTestCase

[rjernst]

Is this really necessary? Seems like it will produce a lot of noise.

[andyb-elastic]

I think we want it because it'll be necessary to make sense of any other logging we do in the tests. The noise won't show up for people running it in the console because of how the vagrant tasks moderate their output - it'll only show up in CI because it runs with --console plain --info

[rjernst]

Do we really need a before and after? These are run completely sequentially, so the "before" of one test is the "after" of the previous. I'm just thinking of what the old output used to look like (a single line per test in most cases with "OK") compared to what we are moving to here (many lines per test, if I understand correctly).

[andyb-elastic]

Makes sense, I removed the after test logging

[andyb-elastic]

Jenkins run gradle build tests please