can not find the IllegalStateException of google/closure-compiler

[wuxiaojie555]

I tried to test 10 times google/closure-compiler ，but i didn't find the IllegalStateException.I only find NullPointerException. should I provide seeds about the test？what should i do？

[rohanpadhye]

Thanks for the issue. Can you clarify what generator and test driver you are using, and for how long you ran the fuzzing session? Are you trying to reproduce the results of the ISSTA'19 paper using the artifact accompanying the paper?

[rohanpadhye]

Further: The Google closure compiler sometimes wraps certain exceptions inside RuntimeException(s). So, you might want to look for wrapped RuntimeExceptions and query their getCause().

[wuxiaojie555]

Thanks for your reply! yes ,I am trying to reproduce the results of the ISSTA'19 paper.I used the ZEST driver , JavaScriptCodeGenerator in the examples directory ，I ran 3hours for a fuzz session. I didn't find the RuntimeException either，I only find NullPointerException.

[rohanpadhye]

Thanks for the feedback. According to the paper, the IllegalStateException for Closure (bug U in Table 2) occurred in only 5% of the runs; that is, in only 1 of the 20 runs. It is possible that this is an incredibly hard-to-find bug that Zest found by accident during our paper evaluation; we of course reported it as such in the paper. I don't know what the actual probability of finding that bug in 3 hours is, so perhaps in your 10 runs you were not lucky enough to find it. I would not classify that as a bug that Zest can "reliably" find.

If you want to investigate that bug nonetheless, see google/closure-compiler#3220 for the bug report. You can also get a copy of the entire experimental results from our paper as a Docker image: rohanpadhye/zest-artifact on Docker Hub.