Incorrect set of trigger tests [Bears benchmark] [apache-incubator-dubbo-329355311-329773929 bug]

[jose]

Context

When applied to the apache-incubator-dubbo-329355311-329773929 bug of the Bears benchmark, GZoltar reports more failing test cases than the expected. GZoltar reports the following failing test cases:

com.alibaba.dubbo.rpc.cluster.router.script.ScriptRouterTest#testRoute_PickInvokers
com.alibaba.dubbo.rpc.cluster.router.script.ScriptRouterTest#testRoute_ReturnAll
com.alibaba.dubbo.rpc.cluster.router.condition.ConditionRouterTest#testRoute_matchFilter

However, and according to the official documentation, there is only one failing test case in the project's test suite, i.e.,

com.alibaba.dubbo.rpc.cluster.router.condition.ConditionRouterTest#testRoute_matchFilter

Steps to Reproduce

$ rm -rf "/tmp/apache-incubator-dubbo-329355311-329773929_issue"
$ mkdir "/tmp/apache-incubator-dubbo-329355311-329773929_issue"
$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue"

Get apache-incubator-dubbo-329355311-329773929 bug from Bears benchmark

$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue"
$ git clone https://github.com/bears-bugs/bears-benchmark.git
$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark"
$ git checkout apache-incubator-dubbo-329355311-329773929
$ buggy_commit=$(git log --format=format:%H --grep='Changes in the tests')
$ git checkout "$buggy_commit"

Compile apache-incubator-dubbo-329355311-329773929 bug from Bears benchmark

$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark"
$ mvn clean install -DskipTests=true # Compile the entire project
$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster"
$ mvn clean package -DskipTests=true # Compile the buggy module

Run tests of the buggy module of the apache-incubator-dubbo-329355311-329773929 bug from Bears benchmark

$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster"
$ mvn test

...
Results :

Failed tests:   testRoute_matchFilter(com.alibaba.dubbo.rpc.cluster.router.condition.ConditionRouterTest): expected:<1> but was:<0>

Tests run: 90, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.999 s
[INFO] Finished at: 2020-04-09T10:29:09+01:00
[INFO] ------------------------------------------------------------------------

Get GZoltar

$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue"
$ git checkout https://github.com/GZoltar/gzoltar.git
$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue/gzoltar"
$ mvn clean package -DskipTests=true
$ export GZOLTAR_CLI_JAR="/tmp/apache-incubator-dubbo-329355311-329773929_issue/gzoltar/com.gzoltar.cli/target/com.gzoltar.cli-1.7.3-SNAPSHOT-jar-with-dependencies.jar"
$ export GZOLTAR_AGENT_JAR="/tmp/apache-incubator-dubbo-329355311-329773929_issue/gzoltar/com.gzoltar.agent.rt/target/com.gzoltar.agent.rt-1.7.3-SNAPSHOT-all.jar"

Perform fault localization

$ cd "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster"

$ classpath=$(mvn dependency:build-classpath | grep -v "\[INFO\]" | grep -v "\[WARNING\]")
$ test_classes_dir="/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster/target/test-classes"
$ src_classes_dir="/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster/target/classes"

Find list of test cases to run

$ list_of_tests_to_run_file="/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster/target/tests_to_run.txt"
$ rm -f "$list_of_tests_to_run_file"

$ java -cp $src_classes_dir:$test_classes_dir:$classpath:$GZOLTAR_CLI_JAR \
    com.gzoltar.cli.Main listTestMethods \
      "$test_classes_dir" \
      --outputFile "$list_of_tests_to_run_file" \
      --includes "com.alibaba.dubbo.rpc.cluster.*"

$ cat "$list_of_tests_to_run_file"

Collect coverage

$ ser_file="/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster/target/gzoltar.ser"
$ rm -f "$ser_file"
$ classes_to_debug="com.alibaba.dubbo.rpc.cluster.*"

$ java -XX:MaxPermSize=2048M -javaagent:$GZOLTAR_AGENT_JAR=destfile=$ser_file,buildlocation=$src_classes_dir,includes=$classes_to_debug,excludes="",inclnolocationclasses=false,output="FILE" \
        -cp $src_classes_dir:$test_classes_dir:$classpath:$GZOLTAR_CLI_JAR \
        com.gzoltar.cli.Main runTestMethods \
          --testMethods "$list_of_tests_to_run_file" \
          --collectCoverage

Create fault localization report

$ java -cp $src_classes_dir:$test_classes_dir:$classpath:$GZOLTAR_CLI_JAR \
      com.gzoltar.cli.Main faultLocalizationReport \
        --buildLocation "$src_classes_dir" \
        --granularity "line" \
        --inclPublicMethods \
        --inclStaticConstructors \
        --inclDeprecatedMethods \
        --dataFile "$ser_file" \
        --outputDirectory "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster/target" \
        --family "sfl" \
        --formula "ochiai" \
        --metric "entropy" \
        --formatter "txt"

$ grep ",FAIL," "/tmp/apache-incubator-dubbo-329355311-329773929_issue/bears-benchmark/dubbo-cluster/target/sfl/txt/tests.csv" | cut -f1 -d','
com.alibaba.dubbo.rpc.cluster.router.script.ScriptRouterTest#testRoute_PickInvokers
com.alibaba.dubbo.rpc.cluster.router.script.ScriptRouterTest#testRoute_ReturnAll
com.alibaba.dubbo.rpc.cluster.router.condition.ConditionRouterTest#testRoute_matchFilter   <<< true fault revealing test case

Environment (please complete the following information, if relevant):

• OS: MacOS and Linux
• Commit: 310fba7

$ java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)

$ mvn -version
Apache Maven 3.6.1