Gives warning when encountering free range .enc files in test directory

[supercooldave]

The existing test script silently passed over .enc files in the test directories that were not set up as proper tests or ignored in the DISABLED_TESTS.grep file. This patch changes the test script to give a warning when encountering those files.

[kaeluka]

I'll check this out in the afternoon.

[kaeluka]

This does what it says, but it adds so much output to some test suites that I'd rather not have this feature. The reason is that so many files are not tests by design. There should at least be a way to disable the warning for files that are supposed to be no test.

Example:

>>> RUNNING TEST_SUITE ./encore/modules
    Found Makefile in ./encore/modules. Running!
make[3]: *** No rule to make target `clean'.  Stop.
           make | make[3]: Nothing to be done for `build'.
 - CallImported...
 - ClosureImp...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - ClosureImporter...
   ... disabled in src/tests/DISABLED_TESTS.grep
 - DeepImporter...
 - DeepLib...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - DupImporter...
 - FarImporter...
 - Importer...
 - Lib...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - List...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - MidImport...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - PassiveImport...
 - Qualified...
 - StdLib...
 - Utils...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - Vector...
   ... warning: .enc file present but missing .out, .fail or .chk file.
 - VectorTest...
 - trait_import...
 - trait_import_module...
   ... warning: .enc file present but missing .out, .fail or .chk file.

[kaeluka]

Would it be an option to add an empty .out file for these? I'm guessing that they can't have a Main class, right? (and therefore can't run)

[supercooldave]

The point of the feature is to force the tester to clean up the directory, for instance by putting the relevant file names in the DISABLED_TESTS.grep file. The point is not to disable the warnings, but to indicate which files are not tests.

If anything, the warnings about tests occurring in the DISABLED_TEST.grep file should be disabled.

I do not like the default empty .out file idea. It is too error prone.

[supercooldave]

I should point out that this feature was present in your original test suite.

[kaeluka]

Adding it to DISABLED_TESTS.grep would be much better than the empty .out file (which I do not like either). However, I tried this out, and it does work, but now a different warning appears: disabled in src/tests/DISABLED_TESTS.grep. If we go this way, we should disable the warning about a test being being skipped. I'm worried that we have lots of warnings constantly and people start to not read them.

I'd be happy to merge if:

• the warning message would be amended to suggest adding the test to DISABLED_TESTS.grep,
• we don't warn about disabled tests any more.

If we want to keep an overview of how many tests are warned, we might consider adding an extra counter for number of skipped tests per test suite in a separate PR.

[kaeluka]

I'd like to point out that going through DISABLED_TESTS.grep is a bit hacky: a file without a specification is not a test in the first place. We shouldn't need to disable a test that is not a test. But it seems pragmatic to not over engineer and reuse the infrastructure we already have.

[supercooldave]

I can make the two suggested changes.

[kaeluka]

And I'll watch this space!

[supercooldave]

If DISABLED_TESTS.grep was called something different, like .gitingore but not, then there would be no semantic problem.

[kaeluka]

If DISABLED_TESTS.grep was called something different, like .gitingore but not, then there would be no semantic problem.

That's true.

[supercooldave]

I've addressed the issues mentioned above and added support for .flags, as previously discussed.

[kaeluka]

You forgot to remove these echo calls :)

[supercooldave]

Shoot!

[kaeluka]

I can actually do that

[supercooldave]

Thanks.