Add a Rust CLI #103

killercup · 2018-04-25T17:40:45Z

No description provided.

killercup · 2018-04-25T17:41:02Z

killercup · 2018-04-25T17:41:51Z

As a next step we should add a way to easily collect the results of the fuzzers in a convenient way.

frewsxcv · 2018-04-26T02:36:16Z

.gitignore

@@ -6,3 +6,6 @@ fuzzer-*/src/bin
 fuzzer-honggfuzz/hfuzz_target/
 fuzzer-honggfuzz/hfuzz_workspace/
 fuzzer-libfuzzer/crash-*
+
+/target
+**/*.rs.bk


these seem redundant with lines 1 and 3, no?

Ah yes, cargo init probably added thos

frewsxcv · 2018-04-26T02:55:11Z

README.md

+- `cargo run -- list-targets` gives you a list of all fuzz targets
+- `cargo run -- run pulldown_cmark_read` runs the `pulldown_cmark_read` target with the default fuzzer
+- `cargo run -- run pulldown_cmark_read --fuzzer libfuzzer` runs the `pulldown_cmark_read` target with `libfuzzer`
+- `cargo run -- continuous` runs all targets (you can overwrite timeout per target and change the fuzzer)


should we encourage cargo run --release? there might be a way to specify opt = 3 or whatever for the profile in cargo.toml, not sure

I don't think the performance of the CLI tool is that important, but this reminds me:

@PaulGrandperrin do the fuzzer's cargo subcommands have a --release flag? Should we pass it? Previously, I've always invoked libfuzzer with --release and debug assertions on.

frewsxcv · 2018-04-26T03:29:10Z

cli.rs

+fn run_honggfuzz(target: &str, timeout: Option<i32>) -> Result<(), Error> {
+    let fuzzer = Fuzzer::Honggfuzz;
+    write_fuzzer_target(fuzzer, target)?;
+    let dir = fuzzer.dir()?;


i tried checking out your branch and running cargo run -- run:

> cargo run -- run pulldown_cmark_read Finished dev [unoptimized + debuginfo] target(s) in 0.91 secs Running `target/debug/cli run pulldown_cmark_read` No such file or directory (os error 2)

one of the downsides of the ? approach here is that it doesn't indicate what operation we tried to do when the error happened. i haven't done any investigation yet, so it's not on this line in particular, just thought i'd mention my thoughts

did a little investigating. this is the line that was returning the error:

https://github.com/rust-fuzz/targets/pull/103/files#diff-eaa2729b20be051a8cb0a93eb8dd2d9fR262

the bin directory isn't being created automatically, so i had to do mkdir fuzzer-honggfuzz/src/bin

also, related to my first point, i learned you can do this with failure:

diff --git a/cli.rs b/cli.rs index cf48dc4..90ef062 100644 --- a/cli.rs +++ b/cli.rs @@ -13,7 +13,7 @@ use std::fs; use std::path::{Path, PathBuf}; use std::process::Command; -use failure::Error; +use failure::{Error, ResultExt}; use regex::Regex; use structopt::StructOpt; @@ -259,7 +259,8 @@ fn write_fuzzer_target(fuzzer: Fuzzer, target: &str) -> Result<(), Error> { .write(true) .create(true) .truncate(true) - .open(&path)?; + .open(&path) + .context("Could not write fuzzer target")?; let source = template.replace("###TARGET###", &target); file.write_all(source.as_bytes())?;

not suggesting we should do that everywhere, just wanted to share

I've just sprinkled a bunch of error context's all over the place and also added automatically creating this */src/bin/ directories

Wow, I didn't about failure::ErrorExt, that's really neat!

killercup · 2018-04-26T10:32:17Z

Oh: We could rename the run subcommands a bit (mainly "run" to "target") and get rid of the -- in the docs (cargo supports this):

$ cargo run target
$ cargo run continuously
$ cargo run list-targets

Yeah, I'll go ahead with this.

`cargo run target pulldown_cmark_read` -- How nice is this?

Copied from clap -- thanks, Kevin!

PaulGrandperrin · 2018-04-26T18:04:28Z

targets.txt

@@ -1,60 +0,0 @@
-brotli_read 1


I like the fact that now this list is automatically extracted from the code (less duplication = fewer potentials errors).
However, this file was also meant to be useful to specify a fuzzing duration multiplicator.
This is not important for this PR, but I just wanted to say somewhere that it's, I think, something important to have eventually.

PaulGrandperrin · 2018-04-26T18:14:59Z

I can't really review the structopt DSL as I have no experience with it but from a user perspective I think it works great!
I think the only feature "lost in translation" is the ability to put weights on each targets, but it can be done later.
You can merge as soon as you want, I'd like to start doing something about the "compile all targets at once" problem.

PaulGrandperrin · 2018-04-26T18:22:08Z

cli.rs

+                .filter(|x| filter.as_ref().map(|f| x.contains(f)).unwrap_or(true));
+
+            if infinite {
+                for target in targets.cycle() {


In the old script I was doing a cargo update at the end of each cycle. I think it can be useful as continuous runs can be long-lived.

killercup · 2018-04-26T21:24:52Z

Either of you: Feel free to merge, we can easily add more stuff in follow up PRs.

I think the only feature "lost in translation" is the ability to put weights on each targets, but it can be done later.

Ah yes, I explicitly skipped this feature for now. I thought I had commented on that in a commit message, but it seems I forgot. Sorry about that.

What was you main goal with that? To make sure expensive fuzzers get some more time to get through a useful number of iterations?

I like the fact that now this list is automatically extracted from the code

It's currently done in quite a hack-y way (regex), and I'm not sure how best to add a multiplier there.

By the way, I was surprised to see there was no timeout to stop after ≥x seconds of no new branch found, by the way.

frewsxcv · 2018-04-27T05:17:48Z

bors r+

🎉🎉🎉🎉🎉🎉

103: Add a Rust CLI r=frewsxcv a=killercup Co-authored-by: Pascal Hertleif <killercup@gmail.com>

bors · 2018-04-27T05:27:00Z

Build succeeded

continuous-integration/travis-ci/push

killercup added 4 commits April 25, 2018 18:48

Add a Rust CLI

a2a3cbb

Allow filtering fuzz targets in continuous mode

5147d99

Auto-generate fuzz target binaries from template

9716f68

Add timeout mode to libfuzzer

4517d6f

killercup added 4 commits April 25, 2018 19:46

Remove now obsolete files

33b8e2e

Update Readme to show new CLI tool

df0c695

Default to non-infinite fuzzing

6d6ee1b

Update CI

74b189d

killercup force-pushed the feature/rust-cli branch from 8000091 to 74b189d Compare April 25, 2018 17:46

Make Cargo happy

54c27c5

killercup force-pushed the feature/rust-cli branch from a557e7c to 54c27c5 Compare April 25, 2018 17:51

frewsxcv reviewed Apr 26, 2018

View reviewed changes

frewsxcv mentioned this pull request Apr 26, 2018

Extend cargo run -- continuous optionally to output HTML with stats #104

Open

killercup added 3 commits April 26, 2018 12:09

Add more context to errors

ccbf716

Automatically create src/bin/ dirs

24529d4

Clean up .gitignore

6155e24

killercup added 2 commits April 26, 2018 13:00

Rename subcommand run to target, update Readme

c3c6cb2

`cargo run target pulldown_cmark_read` -- How nice is this?

Add simple "did you mean" suggestions for targets

a29e7ef

Copied from clap -- thanks, Kevin!

killercup force-pushed the feature/rust-cli branch from 15b879f to a29e7ef Compare April 26, 2018 11:11

Fix libfuzzer timeout

7b1c464

killercup mentioned this pull request Apr 26, 2018

"Cargo has produced more than one binary artifact" error even though only one kind: "bin" target is logged vadimcn/codelldb#104

Closed

PaulGrandperrin reviewed Apr 26, 2018

View reviewed changes

bors bot added a commit that referenced this pull request Apr 27, 2018

Merge #103

fb6e0b5

103: Add a Rust CLI r=frewsxcv a=killercup Co-authored-by: Pascal Hertleif <killercup@gmail.com>

frewsxcv approved these changes Apr 27, 2018

View reviewed changes

bors bot merged commit 7b1c464 into rust-fuzz:master Apr 27, 2018

This was referenced Apr 27, 2018

Add 'weight' to fuzzer when fuzzing continuously #105

Open

Add option to call cargo update between continuous loops #106

Closed

continuous honggfuzz leaves zombie processes behind (on macos) #111

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a Rust CLI #103

Add a Rust CLI #103

killercup commented Apr 25, 2018

killercup commented Apr 25, 2018

killercup commented Apr 25, 2018

frewsxcv Apr 26, 2018

killercup Apr 26, 2018

frewsxcv Apr 26, 2018

killercup Apr 26, 2018

frewsxcv Apr 26, 2018

frewsxcv Apr 26, 2018

frewsxcv Apr 26, 2018

killercup Apr 26, 2018

PaulGrandperrin Apr 26, 2018

killercup commented Apr 26, 2018 •

edited

Loading

PaulGrandperrin Apr 26, 2018

PaulGrandperrin commented Apr 26, 2018 •

edited

Loading

PaulGrandperrin Apr 26, 2018

killercup commented Apr 26, 2018

frewsxcv commented Apr 27, 2018

bors bot commented Apr 27, 2018

Add a Rust CLI #103

Add a Rust CLI #103

Conversation

killercup commented Apr 25, 2018

killercup commented Apr 25, 2018

killercup commented Apr 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

killercup commented Apr 26, 2018 • edited Loading

Choose a reason for hiding this comment

PaulGrandperrin commented Apr 26, 2018 • edited Loading

Choose a reason for hiding this comment

killercup commented Apr 26, 2018

frewsxcv commented Apr 27, 2018

bors bot commented Apr 27, 2018

Build succeeded

killercup commented Apr 26, 2018 •

edited

Loading

PaulGrandperrin commented Apr 26, 2018 •

edited

Loading