Need acceptance test suite

[AlexanderYastrebov]

E.g. current "winner" produces garbage output on trivial input:

$ ./create_measurements.sh 2
Created file with 2 measurements in 18 ms
$ cat measurements.txt 
Calgary;1.6
Tamanrasset;13.7
$ ./calculate_average_ebarlas.sh 
{=2.147483647E8/0.0/-2.147483648E8, manrasset=13.7/13.7/13.7, y=2.147483647E8/0.0/-2.147483648E8}

real    0m0,206s
user    0m0,415s
sys     0m0,048s

[AlexanderYastrebov]

@ebarlas FYI

[AlexanderYastrebov]

Something like

$ for n in 1 2 5 10 100 1000 10000; do ./create_measurements.sh $n && diff <(./calculate_average_AlexanderYastrebov.sh 2>/dev/null | ./tocsv.sh) <(./calculate_average.sh 2>/dev/null | ./tocsv.sh) && echo "OK $n" || echo "FAIL $n"; done

Created file with 1 measurements in 19 ms
OK 1
Created file with 2 measurements in 17 ms
OK 2
Created file with 5 measurements in 21 ms
OK 5
Created file with 10 measurements in 18 ms
OK 10
Created file with 100 measurements in 22 ms
OK 100
Created file with 1,000 measurements in 29 ms
107c107
< Dolisie;5.1;22.5;39.5
---
> Dolisie;5.1;22.6;39.5
124c124
< Gagnoa;17.7;22.4;25.4
---
> Gagnoa;17.7;22.3;25.4
126c126
< Garissa;21.6;41.5;72.0
---
> Garissa;21.6;41.4;72.0
135c135
< Hamburg;-0.9;9.6;17.9
---
> Hamburg;-0.9;9.5;17.9
FAIL 1000
Created file with 10,000 measurements in 62 ms
369c369
< Timbuktu;10.6;29.9;53.7
---
> Timbuktu;10.6;30.0;53.7
FAIL 10000

could be used to verify a given implementation.

[ddimtirov]

I manually run the reference impl on my generated file and pipe it to a file (name needs to start with measurements* or the license checker would complain).

Then in my code I have:

assert Files.readAllLines(Path.of("measurements_result.txt")).getFirst().equals(result);

It is crude, but it gets the job done... also I find assertions useful alternative to defensive coding when you cannot aford the extra checks on a normal run.

[gunnarmorling]

Yeah, if someone wanted to create this kind of test (bonus points for automatically running it against all provided entries), that would be awesome and highly welcome. A test runner which sets System.out to a stream which allows to capture all the output written by a submission and assert that one should do the trick.

[gunnarmorling]

@AlexanderYastrebov, I feel a bit dense, how do you define the expected values in your proposal? Where is tocsv.sh coming from?

[AlexanderYastrebov]

@gunnarmorling

We can create a set of test measurement and expected output files (aka golden files) and check each implementation against it.
This set can contain edge cases like 1, 2, 10, 100 records.
It can also contain files created manually e.g. using station ids not used by the baseline impl.
Users can write unit-test to check their implementation.

I've created #82 for that.

Additionally we can have a test like proposed in #61 (comment) that verifies implementation against baseline implementation.

Where is tocsv.sh coming from?

Was added by #36 but its just to simplify debugging of failures, the validator may simply compare impl output to the expected output.

[radughiorma]

I added a Unit test in this PR #89
What made it difficult is that the FILE is not passed as an argument to the main method.
And also finding the classes to test.

Here is a list with the submissions that I also found to be failing the rules.

[AlexanderYastrebov]

I added a Unit test in this PR #89

I've also created #84 earlier but since implementations rely on various jdks, modules and parameters the validation should be performed by running calculate_average_*.sh scripts.

[AlexanderYastrebov]

@gunnarmorling I guess we can close this and work on additional tests case by case.

[gunnarmorling]

Makes sense. Thx for improving the test suite that much, @AlexanderYastrebov!