Tree-shaped multi-machine fuzzing #2302

rmalmain · 2024-06-11T13:43:06Z

another attempt to get multi-machine fuzzing working, in addition to #66 and #1302.
this one aims to act at broker level to be resilient against restarts.
draft state for now, the code is messy.
it does some refactoring here and there to have things working nicely.

tokatoka · 2024-06-12T09:02:20Z

libafl/src/events/broker_hooks/centralized_multi_machine.rs

+    inputs::Input,
+};
+
+/// The Receiving side of the multi-machine architecture


sending side?

tokatoka · 2024-06-12T09:05:06Z

libafl/src/events/broker_hooks/centralized_multi_machine.rs

+    SP: ShMemProvider,
+    I: Input + Send + Sync + 'static,
+{
+    /// check for received messages, and forward them alongside the incoming message to inner.


maybe update comment?

libafl/src/events/broker_hooks/centralized_multi_machine.rs

tokatoka · 2024-06-12T09:46:02Z

libafl/src/events/launcher.rs

        if self.spawn_broker {
            log::info!("I am broker!!.");

-            // TODO we don't want always a broker here, think about using different laucher process to spawn different configurations


this is much cleaner now 😀

tokatoka · 2024-06-12T09:52:46Z

libafl/src/events/multi_machine.rs

+        let serialized_msg = bitcode::serialize(msg)?;
+        let msg_len = u32::to_le_bytes(serialized_msg.len() as u32);
+
+        // 0. Write the dummy byte


why this is needed? to write dummy bytes?

tokatoka · 2024-06-12T10:03:00Z

libafl/src/events/multi_machine.rs

+
+                parent_lock.parent = loop {
+                    info!("Trying to connect to parent @ {}..", parent_addr);
+                    match TcpStream::connect(parent_addr).await {


shall we use set_keepalive() on this connection?

thread safety stuff

mv to sqlite less warnings

domenukk · 2024-06-18T00:35:40Z

libafl/src/events/broker_hooks/centralized_multi_machine.rs

+
+        // Here, we suppose msg will never be written again and will always be available.
+        // Thus, it is safe to handle this in a separate thread.
+        let msg_lock = unsafe { NullLock::new((msg.as_ptr(), msg.len())) };


I feel like this is wrong on a rust level, the mutable borrow only lives as long as on_new_message, right? And here you spawn an async function that runs after the function returns, using the msg.

at least the TcpMultiMachineLlmpReceiverHook new function should be unsafe, and this side effect ("on_new_message needs msg to have a quasi-static lifetime) should be documented there

the msg_lock will leave beyond the call to on_new_message. it's valid with 2 hypothesis:

messages, once sent, are never written again

once a message is posted, it is never deallocated.

this patch is very experimental in general, we merged for practical reasons (it's part of the reason why it's hidden behind an undocumented feature for now)

I'll open the follow-up patch

domenukk · 2024-06-18T00:39:52Z

libafl/src/events/llmp/mgr.rs


                if self.always_interesting {
                    let item = fuzzer.add_input(state, executor, self, input)?;
-                    log::info!("Added received Testcase as item #{item}");
+                    debug!("Added received Testcase as item #{item}");


Why debug? This feels pretty confusing, I'd prefer to prepend log
(There is the non-log debug macro already)

i tried to keep the existing as log and the new one as debug to avoid polluting the existing logs, otherwise it becomes very verbous. this one is my mistake, i forgot to revert it to info

domenukk · 2024-06-18T00:41:25Z

libafl/src/events/llmp/mod.rs

@@ -13,6 +13,7 @@ use libafl_bolts::{
    shmem::{NopShMemProvider, ShMemProvider},
    ClientId,
 };
+use log::debug;


This shadows the debug macro, which is pretty handy

domenukk · 2024-06-18T00:43:18Z

libafl/src/events/tcp.rs

@@ -1,1369 +0,0 @@
-//! TCP-backed event manager for scalable multi-processed fuzzing


Why remove this? It's good to have

not sure, i'll ask to toka

domenukk · 2024-06-18T00:43:34Z

libafl/src/lib.rs

@@ -32,6 +32,7 @@ Welcome to `LibAFL`
    clippy::similar_names,
    clippy::too_many_lines,
    clippy::into_iter_without_iter, // broken
+    clippy::type_complexity,


@tokatoka why allow this?

domenukk · 2024-06-18T00:52:12Z

Why does this remove the tcp event manager?

domenukk · 2024-06-18T01:01:53Z

fuzzers/libfuzzer_libpng_tcp_manager/Cargo.toml

@@ -18,7 +18,7 @@ debug = true

 [build-dependencies]
 cc = { version = "1.0", features = ["parallel"] }
-which = "4.4"
+which = "6.0"

 [dependencies]
 libafl = { path = "../../libafl/", features = ["default", "tcp_manager"] }


How does that (still) work?

* Address comments from #2302 * secure? * cleanup * early exit ftw * address clippy * Fix all the things

* tree-shaped multi-machine fuzzing * forgot main file * aaa * moving things around * fix * working? * remove debug panic * aaa * aaa * fmt * normal centralized adapted * removed old useless code * cleanup * llmp hooks * working multi machine apparently? * aaa * cleanup (AFLplusplus#2305) * added old message dispatch. thread safety stuff * testing things around * opti opti opti * :) * fuzz * limit the amound received at once to avoid congestion * remove useless corpus mv to sqlite less warnings * aaa * ; * big opti * adding cfgs * fix * fixer * fix * s * clippy and reduce generics * debugging * fix * more robust disconnection * aaa * aaa * aaa * nostd * more nostd * clippy * not in ci * unused * aaa * doc * clippy * clippy * clippy * no crash in libpng * aaa * aaa * aaa * aaa * graph generator * fix * fix * windows fix all --------- Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>

* Address comments from AFLplusplus#2302 * secure? * cleanup * early exit ftw * address clippy * Fix all the things

rmalmain added 18 commits June 7, 2024 15:09

tree-shaped multi-machine fuzzing

6991513

forgot main file

79c238e

aaa

c74ff4d

moving things around

1b5b0bb

fix

f963cda

Merge branch 'main' into multi_machine

031adec

working?

2e682e4

remove debug panic

a750b28

aaa

06e6520

aaa

7ed03d7

Merge branch 'main' into multi_machine

c7e66e0

fmt

5633bcb

normal centralized adapted

393dde2

removed old useless code

05c2a95

cleanup

8844258

llmp hooks

54d2242

working multi machine apparently?

2e81f71

aaa

cb0378d

tokatoka reviewed Jun 12, 2024

View reviewed changes

tokatoka and others added 9 commits June 12, 2024 16:07

cleanup (#2305)

09b3608

added old message dispatch.

cb47ee6

thread safety stuff

testing things around

8591e0f

opti opti opti

101175f

Merge branch 'main' into multi_machine

660c3ec

:)

0703000

fuzz

a8165a1

limit the amound received at once to avoid congestion

a6fb497

remove useless corpus

f69454a

mv to sqlite less warnings

rmalmain and others added 16 commits June 17, 2024 18:30

not in ci

9bf7aa9

unused

9265e3e

aaa

c67e873

doc

a6ab6db

clippy

d9f0683

clippy

8054d80

clippy

250276f

no crash in libpng

76128a5

aaa

872190c

aaa

c7e41ab

aaa

a883729

aaa

56c8b55

graph generator

9d7d6bd

fix

2c0c8eb

fix

e159c62

windows fix all

431f953

tokatoka merged commit fa17f47 into main Jun 17, 2024
7 checks passed

tokatoka deleted the multi_machine branch June 17, 2024 21:23

domenukk reviewed Jun 18, 2024

View reviewed changes

domenukk added a commit that referenced this pull request Jun 18, 2024

Address comments from #2302

8ed3f16

domenukk mentioned this pull request Jun 18, 2024

Address comments from #2302 #2322

Merged

domenukk added a commit that referenced this pull request Jun 18, 2024

Address comments from #2302 (#2322)

e64f0fb

* Address comments from #2302 * secure? * cleanup * early exit ftw * address clippy * Fix all the things

R9295 pushed a commit to R9295/LibAFL that referenced this pull request Jun 21, 2024

Address comments from AFLplusplus#2302 (AFLplusplus#2322)

113da5a

* Address comments from AFLplusplus#2302 * secure? * cleanup * early exit ftw * address clippy * Fix all the things

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tree-shaped multi-machine fuzzing #2302

Tree-shaped multi-machine fuzzing #2302

rmalmain commented Jun 11, 2024

tokatoka Jun 12, 2024

tokatoka Jun 12, 2024

tokatoka Jun 12, 2024

tokatoka Jun 12, 2024

tokatoka Jun 12, 2024

domenukk Jun 18, 2024

domenukk Jun 18, 2024

rmalmain Jun 18, 2024

domenukk Jun 18, 2024 •

edited

Loading

rmalmain Jun 18, 2024

domenukk Jun 18, 2024

domenukk Jun 18, 2024

rmalmain Jun 18, 2024

domenukk Jun 18, 2024

rmalmain Jun 18, 2024

domenukk commented Jun 18, 2024

domenukk Jun 18, 2024

		@@ -1,1369 +0,0 @@
		//! TCP-backed event manager for scalable multi-processed fuzzing

Tree-shaped multi-machine fuzzing #2302

Tree-shaped multi-machine fuzzing #2302

Conversation

rmalmain commented Jun 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk commented Jun 18, 2024

Choose a reason for hiding this comment

domenukk Jun 18, 2024 •

edited

Loading