Skip to content

Testing

Jump to bottom
Matthieu Baerts edited this page Dec 19, 2024 · 13 revisions

MPTCP Linux Testing

The MPTCP Linux community uses three main tools for test purposes: kselftest, syzkaller, and packetdrill.

For kernel development, we recommend using the MPTCP Virtme Docker image

kselftest

A set of test scripts and associated programs are included in the kernel source tree: tools/testing/selftests/net/mptcp

Make sure the kernel you build for use with these tests is built with the config options listed in tools/testing/selftests/net/mptcp/config.

To build and run the tests:

$ cd $KERNELGITREPO
$ cd tools/testing/selftests/net/mptcp/
$ make
$ sudo ./mptcp_connect.sh
$ sudo ./mptcp_join.sh
$ sudo ... # and so on for each .sh file in that directory

Note that a docker image can be used to prepare a dev environment ready to run tests and more. For more details, see here.

syzkaller

syzkaller is a kernel fuzzing tool for Linux and other kernels. Refer to the syzkaller Linux setup instructions.

The syzkaller project has a list of kernel config options that allow syzkaller to work best. Some options may slow down the test kernel quite a bit, if performance is an issue you might experiment with removing some of the debugging options in the "the more the better" section.

Here's a bash script that will generate an MPTCP / syzkaller kernel config with all relevant options enabled:

#!/bin/bash

make defconfig
make kvm_guest.config

scripts/config --enable CONFIG_MPTCP

scripts/config --enable CONFIG_IPV6
scripts/config --enable CONFIG_MPTCP_IPV6
scripts/config --enable CONFIG_INET_DIAG
scripts/config --enable CONFIG_INET_MPTCP_DIAG
scripts/config --enable CONFIG_VETH
scripts/config --enable CONFIG_NET_SCH_NETEM
scripts/config --enable CONFIG_NETFILTER
scripts/config --enable CONFIG_NETFILTER_ADVANCED
scripts/config --enable CONFIG_NETFILTER_NETLINK
scripts/config --enable CONFIG_NF_TABLES
scripts/config --enable CONFIG_NFT_COUNTER
scripts/config --enable CONFIG_NFT_COMPAT
scripts/config --enable CONFIG_NETFILTER_XTABLES
scripts/config --enable CONFIG_NETFILTER_XT_MATCH_BPF
scripts/config --enable CONFIG_NF_TABLES_IPV4
scripts/config --enable CONFIG_NF_TABLES_IPV6

scripts/config --enable CONFIG_NETLINK_DIAG
scripts/config --enable CONFIG_FUSE_FS
scripts/config --enable CONFIG_DYNAMIC_DEBUG
scripts/config --enable CONFIG_BTF
scripts/config --enable CONFIG_NET_NS_REFCNT_TRACKER

#from syzkaller kernel_configs.md
scripts/config --enable CONFIG_KCOV
scripts/config --enable CONFIG_KCOV_INSTRUMENT_ALL
scripts/config --enable CONFIG_KCOV_ENABLE_COMPARISONS
scripts/config --enable CONFIG_DEBUG_FS

scripts/config --enable CONFIG_DEBUG_INFO

scripts/config --enable CONFIG_KALLSYMS
scripts/config --enable CONFIG_KALLSYMS_ALL

scripts/config --enable CONFIG_NAMESPACES
scripts/config --enable CONFIG_UTS_NS
scripts/config --enable CONFIG_IPC_NS
scripts/config --enable CONFIG_PID_NS
scripts/config --enable CONFIG_NET_NS
scripts/config --enable CONFIG_CGROUP_PIDS
scripts/config --enable CONFIG_MEMCG

scripts/config --enable CONFIG_USER_NS

scripts/config --disable CONFIG_RANDOMIZE_BASE

scripts/config --enable CONFIG_CONFIGFS_FS
scripts/config --enable CONFIG_SECURITYFS

scripts/config --set-val CONFIG_DEFAULT_HUNG_TASK_TIMEOUT 140
scripts/config --set-val CONFIG_RCU_CPU_STALL_TIMEOUT 100
scripts/config --enable CONFIG_SOFTLOCKUP_DETECTOR
scripts/config --enable CONFIG_HARDLOCKUP_DETECTOR
scripts/config --enable CONFIG_BOOTPARAM_HARDLOCKUP_PANIC
scripts/config --enable CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC
scripts/config --enable CONFIG_DETECT_HUNG_TASK
scripts/config --enable CONFIG_WQ_WATCHDOG

scripts/config --enable CONFIG_TUN

make olddefconfig

More details in syzkaller doc.

The best is to use the above config for a bit of time, to build up the corpus.

Then you can add KASAN:

scripts/config --enable CONFIG_KASAN
scripts/config --enable CONFIG_KASAN_INLINE

make olddefconfig

Then when the KASAN-kernel stops finding interesting things, progressively add one of these, then a mix:

scripts/config --enable CONFIG_DEBUG_KMEMLEAK

make olddefconfig

scripts/config --enable CONFIG_FAULT_INJECTION
scripts/config --enable CONFIG_FAULT_INJECTION_DEBUG_FS
scripts/config --enable CONFIG_FAULT_INJECTION_USERCOPY
scripts/config --enable CONFIG_FAILSLAB
scripts/config --enable CONFIG_FAIL_PAGE_ALLOC
scripts/config --enable CONFIG_FAIL_MAKE_REQUEST
scripts/config --enable CONFIG_FAIL_IO_TIMEOUT
scripts/config --enable CONFIG_FAIL_FUTEX

make olddefconfig

scripts/config --enable CONFIG_LOCKDEP
scripts/config --enable CONFIG_PROVE_LOCKING
scripts/config --enable CONFIG_DEBUG_ATOMIC_SLEEP
scripts/config --enable CONFIG_PROVE_RCU
scripts/config --enable CONFIG_DEBUG_VM
scripts/config --enable CONFIG_FORTIFY_SOURCE
scripts/config --enable CONFIG_HARDENED_USERCOPY
scripts/config --enable CONFIG_LOCKUP_DETECTOR

make olddefconfig

Before running syzkaller you will need to create a config file. Here is a config.cfg template, with some exclusions for system calls that can consume significant test time but are not relevant for MPTCP. Be sure to substitute the directory names as needed for your system anywhere you see $ENVIRONMENT_VARIABLE-style names (except the disable_syscalls list). You can also increase the vm count variable depending on your available CPU cores and memory.

{
        "target": "linux/amd64",
        "http": "127.0.0.1:56741",
        "workdir": "/home/$USER/gopath/src/github.com/google/syzkaller/$YOUR_WORK_DIR",
        "kernel_obj": "$YOUR_BUILD_DIR",
        "kernel_src": "$YOUR_SRC_DIR",
        "image": "$YOUR_SYZ_IMAGE_DIR/stretch.img",
        "sshkey": "$YOUR_SYZ_IMAGE_DIR/stretch.id_rsa",
        "syzkaller": "/home/$USER/gopath/src/github.com/google/syzkaller",
        "disable_syscalls": ["perf_event_open", "syz_mount_image", "syz_read_part_table", "openat$ttyprintk", "mount", "mkdir",
"openat$ptmx", "mq_open", "fsetxattr", "rt_tgsigqueueinfo", "ioctl$VT_RESIZE", "ioctl$TIOCVHANGUP", "get_robust_list",
"openat$nullb", "ioctl$SCSI_IOCTL_SEND_COMMAND", "lremovexattr", "mknod$loop", "write$binfmt_script", "syz_open_dev$sg", "write$nbd",
"prlimit64", "write$P9_RRENAMEAT", "fcntl$addseals", "finit_module", "ioctl$KDSETMODE", "write$FUSE_NOTIFY_STORE",
"ioctl$TIOCL_SETVESABLANK", "fsmount", "socket$vsock_stream", "socketpair$unix", "socket$nl_audit", "connect$unix", "bind$unix"],
        "procs": 24,
        "type": "qemu",
        "vm": {
                "count": 12,
                "kernel": "$YOUR_BUILD_DIR/arch/x86/boot/bzImage",
                "cpu": 2,
                "mem": 2048
        }
}

(feel free to change the number of procs, vm count, cpu and mem)

In addition to the syzkaller configuration, your system will need to have QEMU installed. Before running syzkaller the first time, you will need to create a vm image with the create-image.sh tool that's part of syzkaller and is mentioned in the VM setup section of the syzkaller docs. That script depends on the debootstrap tool, which is available on many Linux distros (Debian-based or not).

Once you have golang, syzkaller, your config, the VM image, and your kernel build all in place, you run syzkaller with:

$ cd $SYZPATH
$ bin/syz-manager -config config.cfg

syzkaller can generate reproducer programs. Sometimes these reproducers are included in MPTCP bug reports. See the syzkaller "reproducing crashes" docs for information about running these.

packetdrill

The MPTCP project has a MPTCP fork of packetdrill. That github project page has instructions for building packetdrill. To run the MPTCP tests, use:

$ cd $PACKETDRILL_WORKING_DIR/gtests/net
$ sudo ./packetdrill/run_all.py -vslS mptcp/

Note that a docker image can be used to prepare a dev environment ready to run tests and more. For more details, see here.

Decoding a stacktrace

In some cases, the kernel will print a stacktrace -- you should see Call Trace: in the middle in the kernel logs. Running the decode_stacktrace.sh script will resolve addresses, and improve the output:

./scripts/decode_stacktrace.sh /PATH/TO/vmlinux /PATH/TO/kernel-source-dir

This script is not specific to MPTCP, you can find more examples on how to use it elsewhere on the Internet.

The vmlinux image had to be built with debugging info. If the kernel was installed via a package manager -- e.g. default installation on most GNU/Linux distribution like Debian, Ubuntu, RHEL, Fedora, CentOS, etc. -- you probably need to install an extra package like linux-image-$(uname -r)-dbgsym, kernel-(...)-debug or kernel-(...)-dbg. Refer to you distribution's documentation to find out what to do in your case.

If you compiled your kernel yourself, you can get more details there for example.

If you use the docker image, the stacktrace is automatically decoded in "auto" mode. In manual mode, you can use:

cd <kernel source dir>
cat stacktrace.txt | ./scripts/decode_stacktrace.sh .virtme/build/vmlinux $PWD .virtme/build/.virtme_mods

Launch a syzrepro

Get syzkaller ready: (more details here)

cd <Workspace>
git clone git@github.com:google/syzkaller.git
cd syzkaller
make

Launch syz-execprog (adapt the links, the timeout, the number of CPUs, and the path to syzkaller on your host):

cd <kernel source code>
mkdir -p patches
curl -o patches/MY_ISSUE.config <CONFIG_LINK>
curl -o patches/MY_ISSUE.syz <SYZ REPRO LINK>
echo "cd /opt/syzkaller/bin/linux_amd64/ &&
      timeout 30m ./syz-execprog -repeat=0 -procs=\$((\$(nproc) * 4)) \
      $PWD/patches/MY_ISSUE.syz" > .virtme-exec-run

docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" \
      -e INPUT_CPUS=14 \
      -v "/PATH/TO/SYZKALLER:/opt/syzkaller:rw" \
      -v "${PWD}/.home:/root:rw" --rm \
      -it --privileged --pull always \
      mptcp/mptcp-upstream-virtme-docker:latest \
      auto-debug patches/MY_ISSUE.config
Clone this wiki locally