From 13e9ab23733ab6a59d776c281ed48751ad02c3ff Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Sun, 8 May 2022 16:05:30 +0200 Subject: [PATCH 1/2] :memo: document fuzzer usage --- .gitignore | 4 +++ tests/fuzzing.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) create mode 100644 tests/fuzzing.md diff --git a/.gitignore b/.gitignore index db6dcecb45..72eb2c2257 100644 --- a/.gitignore +++ b/.gitignore @@ -19,6 +19,10 @@ # build directories (vscode-cmake-tools, user-defined, ...) /build*/ +# fuzzers +/tests/parse_*_fuzzer +/tests/corpus_* + /docs/mkdocs/docs/examples/ /docs/mkdocs/docs/__pycache__/ /docs/mkdocs/site/ diff --git a/tests/fuzzing.md b/tests/fuzzing.md new file mode 100644 index 0000000000..3ac696a59e --- /dev/null +++ b/tests/fuzzing.md @@ -0,0 +1,79 @@ +# Fuzz testing + +Each parser of the library (JSON, BJData, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently, +[LibFuzzer](https://llvm.org/docs/LibFuzzer.html) and [afl++](https://github.com/AFLplusplus/AFLplusplus) are supported. + +## Corpus creation + +For most effective fuzzing, a [corpus](https://llvm.org/docs/LibFuzzer.html#corpus) should be provided. A corpus is a +directory with some simple input files that cover several features of the parser and is hence a good starting point +for mutations. + +```shell +TEST_DATA_VERSION=3.1.0 +wget https://github.com/nlohmann/json_test_data/archive/refs/tags/v$TEST_DATA_VERSION.zip +unzip v$TEST_DATA_VERSION.zip +rm v$TEST_DATA_VERSION.zip +for FORMAT in json bjdata bson cbor msgpack ubjson +do + rm -fr mkdir corpus_$FORMAT + mkdir corpus_$FORMAT + find json_test_data-$TEST_DATA_VERSION -size -5k -name "*.$FORMAT" -exec cp "{}" "corpus_$FORMAT" \; +done +rm -fr json_test_data-$TEST_DATA_VERSION +``` + +The generated corpus can be used with both LibFuzzer and afl++. + +## LibFuzzer + +To use LibFuzzer, you need to pass `-fsanitize=fuzzer` as `FUZZER_ENGINE`: + +```shell +make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" +``` + +This creates a fuzz tester binary for each parser that supports these +[command line options](https://llvm.org/docs/LibFuzzer.html#options). + +Note the compiler provided by Xcode (AppleClang) does not contain libFuzzer. Please install Clang via Homebrew calling +`brew install llvm` and add `CXX=$(brew --prefix llvm)/bin/clang` to the `make` call: + +```shell +make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" CXX=$(brew --prefix llvm)/bin/clang +``` + +Then pass the corpus directory as command-line argument: + +```shell +./parse_cbor_fuzzer corpus_cbor +``` + +The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is dumped into +a file starting with `crash-`. + +## afl++ + +To use afl++, you need to pass `-fsanitize=fuzzer` as `FUZZER_ENGINE`. It will be replaced by a `libAFLDriver.a` to +re-use the same code written for LibFuzzer with afl++. Furthermore, set `afl-clang-fast++` as compiler. + +```shell +CXX=afl-clang-fast++ make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" +``` + +Then the fuzzer is called like this: + +```shell +afl-fuzz -i corpus_cbor -o out -- ./parse_cbor_fuzzer +``` + +The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is written to the +directory `out`. + +## OSS-Fuzz + +The library is further fuzz-tested 24/7 by Google's [OSS-Fuzz project](https://github.com/google/oss-fuzz). It uses +the same `fuzzers` target as above and also relies on the `FUZZER_ENGINE` variable. See the used +[build script](https://github.com/google/oss-fuzz/blob/master/projects/json/build.sh) for more information. + +In case the build at OSS-Fuzz fails, and issue will be created automatically. From f92e6d0979190347796329a66b76fd9b16ad22a5 Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Sun, 8 May 2022 17:06:23 +0200 Subject: [PATCH 2/2] :memo: address review comments --- tests/fuzzing.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/tests/fuzzing.md b/tests/fuzzing.md index 3ac696a59e..cfbf4f2499 100644 --- a/tests/fuzzing.md +++ b/tests/fuzzing.md @@ -1,12 +1,12 @@ # Fuzz testing Each parser of the library (JSON, BJData, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently, -[LibFuzzer](https://llvm.org/docs/LibFuzzer.html) and [afl++](https://github.com/AFLplusplus/AFLplusplus) are supported. +[libFuzzer](https://llvm.org/docs/LibFuzzer.html) and [afl++](https://github.com/AFLplusplus/AFLplusplus) are supported. ## Corpus creation For most effective fuzzing, a [corpus](https://llvm.org/docs/LibFuzzer.html#corpus) should be provided. A corpus is a -directory with some simple input files that cover several features of the parser and is hence a good starting point +directory with some simple input files that cover several features of the parser and is hence a good starting point for mutations. ```shell @@ -16,18 +16,19 @@ unzip v$TEST_DATA_VERSION.zip rm v$TEST_DATA_VERSION.zip for FORMAT in json bjdata bson cbor msgpack ubjson do - rm -fr mkdir corpus_$FORMAT + rm -fr corpus_$FORMAT mkdir corpus_$FORMAT find json_test_data-$TEST_DATA_VERSION -size -5k -name "*.$FORMAT" -exec cp "{}" "corpus_$FORMAT" \; done rm -fr json_test_data-$TEST_DATA_VERSION ``` -The generated corpus can be used with both LibFuzzer and afl++. +The generated corpus can be used with both libFuzzer and afl++. The remainder of this documentation assumes the corpus +directories have been created in the `tests` directory. -## LibFuzzer +## libFuzzer -To use LibFuzzer, you need to pass `-fsanitize=fuzzer` as `FUZZER_ENGINE`: +To use libFuzzer, you need to pass `-fsanitize=fuzzer` as `FUZZER_ENGINE`. In the `tests` directory, call ```shell make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" @@ -36,14 +37,15 @@ make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" This creates a fuzz tester binary for each parser that supports these [command line options](https://llvm.org/docs/LibFuzzer.html#options). -Note the compiler provided by Xcode (AppleClang) does not contain libFuzzer. Please install Clang via Homebrew calling -`brew install llvm` and add `CXX=$(brew --prefix llvm)/bin/clang` to the `make` call: +In case your default compiler is not a Clang compiler that includes libFuzzer (Clang 6.0 or later), you need to set the +`CXX` variable accordingly. Note the compiler provided by Xcode (AppleClang) does not contain libFuzzer. Please install +Clang via Homebrew calling `brew install llvm` and add `CXX=$(brew --prefix llvm)/bin/clang` to the `make` call: ```shell make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" CXX=$(brew --prefix llvm)/bin/clang ``` -Then pass the corpus directory as command-line argument: +Then pass the corpus directory as command-line argument (assuming it is located in `tests`): ```shell ./parse_cbor_fuzzer corpus_cbor @@ -55,13 +57,13 @@ a file starting with `crash-`. ## afl++ To use afl++, you need to pass `-fsanitize=fuzzer` as `FUZZER_ENGINE`. It will be replaced by a `libAFLDriver.a` to -re-use the same code written for LibFuzzer with afl++. Furthermore, set `afl-clang-fast++` as compiler. +re-use the same code written for libFuzzer with afl++. Furthermore, set `afl-clang-fast++` as compiler. ```shell CXX=afl-clang-fast++ make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" ``` -Then the fuzzer is called like this: +Then the fuzzer is called like this in the `tests` directory: ```shell afl-fuzz -i corpus_cbor -o out -- ./parse_cbor_fuzzer @@ -76,4 +78,4 @@ The library is further fuzz-tested 24/7 by Google's [OSS-Fuzz project](https://g the same `fuzzers` target as above and also relies on the `FUZZER_ENGINE` variable. See the used [build script](https://github.com/google/oss-fuzz/blob/master/projects/json/build.sh) for more information. -In case the build at OSS-Fuzz fails, and issue will be created automatically. +In case the build at OSS-Fuzz fails, an issue will be created automatically.