Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libfuzzer] Fix -runs=X flaky test (fuzzer-finalstats.test) #96914

Merged
merged 1 commit into from
Jul 9, 2024

Conversation

hctim
Copy link
Collaborator

@hctim hctim commented Jun 27, 2024

Disables LSan in order to remove a 1% flake rate in this test.
There's some logic in LeakSanitizer and its integration into libFuzzer
that will disable LSan and re-run the input. This only happens when more
malloc()s are detected than free()s. Under high system load, this
appears to be possible as the "more mallocs than frees" is dependent on
walltime. In these instances, the number of runs ends up being n + 1,
which is undesirable.

Disables LSan in order to remove a 1% flake rate in this test.
There's some logic in LeakSanitizer and its integration into libFuzzer
that will disable LSan and re-run the input. This only happens when more
malloc()s are detected than free()s. Under high system load, this
appears to be possible as the "more mallocs than frees" is dependent on
walltime. In these instances, the number of runs ends up being `n + 1`,
which is undesirable.
@llvmbot
Copy link
Member

llvmbot commented Jun 27, 2024

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Mitch Phillips (hctim)

Changes

Disables LSan in order to remove a 1% flake rate in this test.
There's some logic in LeakSanitizer and its integration into libFuzzer
that will disable LSan and re-run the input. This only happens when more
malloc()s are detected than free()s. Under high system load, this
appears to be possible as the "more mallocs than frees" is dependent on
walltime. In these instances, the number of runs ends up being n + 1,
which is undesirable.


Full diff: https://github.com/llvm/llvm-project/pull/96914.diff

1 Files Affected:

  • (modified) compiler-rt/test/fuzzer/fuzzer-finalstats.test (+10-2)
diff --git a/compiler-rt/test/fuzzer/fuzzer-finalstats.test b/compiler-rt/test/fuzzer/fuzzer-finalstats.test
index d8c991e308478..3a54b2ee003e5 100644
--- a/compiler-rt/test/fuzzer/fuzzer-finalstats.test
+++ b/compiler-rt/test/fuzzer/fuzzer-finalstats.test
@@ -1,12 +1,20 @@
 RUN: %cpp_compiler %S/SimpleTest.cpp -o %t-SimpleTest
-RUN: %run %t-SimpleTest -seed=1 -runs=77 -print_final_stats=1 2>&1 | FileCheck %s --check-prefix=FINAL_STATS
+
+// Note: `-detect_leaks=0` is required to prevent flakiness in this test.
+// There's some logic in LeakSanitizer and its integration into libFuzzer that
+// will disable LSan and re-run the input. This only happens when more malloc()s
+// are detected than free()s. Under high system load, this appears to be
+// possible as the "more mallocs than frees" is dependent on walltime. In these
+// instances, the number of runs ends up being `n + 1`, which is undesirable.
+
+RUN: %run %t-SimpleTest -seed=1 -runs=77 -print_final_stats=1 -detect_leaks=0 2>&1 | FileCheck %s --check-prefix=FINAL_STATS
 FINAL_STATS: stat::number_of_executed_units: 77
 FINAL_STATS: stat::average_exec_per_sec:     0
 FINAL_STATS: stat::new_units_added:
 FINAL_STATS: stat::slowest_unit_time_sec:    0
 FINAL_STATS: stat::peak_rss_mb:
 
-RUN: %run %t-SimpleTest %S/dict1.txt -runs=33 -print_final_stats=1 2>&1 | FileCheck %s --check-prefix=FINAL_STATS1
+RUN: %run %t-SimpleTest %S/dict1.txt -runs=33 -print_final_stats=1 -detect_leaks=0 2>&1 | FileCheck %s --check-prefix=FINAL_STATS1
 FINAL_STATS1: stat::number_of_executed_units: 33
 FINAL_STATS1: stat::peak_rss_mb:
 

@hctim
Copy link
Collaborator Author

hctim commented Jul 9, 2024

FYI @vitalybuka

@hctim hctim merged commit bb90e2e into llvm:main Jul 9, 2024
10 checks passed
@hctim hctim deleted the libfuzzer/flake branch July 9, 2024 09:23
aaryanshukla pushed a commit to aaryanshukla/llvm-project that referenced this pull request Jul 14, 2024
Disables LSan in order to remove a 1% flake rate in this test.
There's some logic in LeakSanitizer and its integration into libFuzzer
that will disable LSan and re-run the input. This only happens when more
malloc()s are detected than free()s. Under high system load, this
appears to be possible as the "more mallocs than frees" is dependent on
walltime. In these instances, the number of runs ends up being `n + 1`,
which is undesirable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants