-
-
Notifications
You must be signed in to change notification settings - Fork 43
Testing
The MPTCP Linux community uses three main tools for test purposes: kselftest, syzkaller, and packetdrill.
For kernel development, we recommend using the MPTCP Virtme Docker image
A set of test scripts and associated programs are included in the kernel source tree: tools/testing/selftests/net/mptcp
Make sure the kernel you build for use with these tests is built with the config options listed in tools/testing/selftests/net/mptcp/config.
To build and run the tests:
$ cd $KERNELGITREPO
$ cd tools/testing/selftests/net/mptcp/
$ make
$ sudo ./mptcp_connect.sh
$ sudo ./mptcp_join.sh
$ sudo ... # and so on for each .sh file in that directory
Note that a docker image can be used to prepare a dev environment ready to run tests and more. For more details, see here.
syzkaller is a kernel fuzzing tool for Linux and other kernels. Refer to the syzkaller Linux setup instructions.
The syzkaller project has a list of kernel config options that allow syzkaller to work best. Some options may slow down the test kernel quite a bit, if performance is an issue you might experiment with removing some of the debugging options in the "the more the better" section.
Here's a bash script that will generate an MPTCP / syzkaller kernel config with all relevant options enabled:
#!/bin/bash
make defconfig
make kvm_guest.config
scripts/config --enable CONFIG_MPTCP
scripts/config --enable CONFIG_IPV6
scripts/config --enable CONFIG_MPTCP_IPV6
scripts/config --enable CONFIG_INET_DIAG
scripts/config --enable CONFIG_INET_MPTCP_DIAG
scripts/config --enable CONFIG_VETH
scripts/config --enable CONFIG_NET_SCH_NETEM
scripts/config --enable CONFIG_NETFILTER
scripts/config --enable CONFIG_NETFILTER_ADVANCED
scripts/config --enable CONFIG_NETFILTER_NETLINK
scripts/config --enable CONFIG_NF_TABLES
scripts/config --enable CONFIG_NFT_COUNTER
scripts/config --enable CONFIG_NFT_COMPAT
scripts/config --enable CONFIG_NETFILTER_XTABLES
scripts/config --enable CONFIG_NETFILTER_XT_MATCH_BPF
scripts/config --enable CONFIG_NF_TABLES_IPV4
scripts/config --enable CONFIG_NF_TABLES_IPV6
scripts/config --enable CONFIG_NETLINK_DIAG
scripts/config --enable CONFIG_FUSE_FS
scripts/config --enable CONFIG_DYNAMIC_DEBUG
scripts/config --enable CONFIG_BTF
scripts/config --enable CONFIG_NET_NS_REFCNT_TRACKER
#from syzkaller kernel_configs.md
scripts/config --enable CONFIG_KCOV
scripts/config --enable CONFIG_KCOV_INSTRUMENT_ALL
scripts/config --enable CONFIG_KCOV_ENABLE_COMPARISONS
scripts/config --enable CONFIG_DEBUG_FS
scripts/config --enable CONFIG_DEBUG_INFO
scripts/config --enable CONFIG_KALLSYMS
scripts/config --enable CONFIG_KALLSYMS_ALL
scripts/config --enable CONFIG_NAMESPACES
scripts/config --enable CONFIG_UTS_NS
scripts/config --enable CONFIG_IPC_NS
scripts/config --enable CONFIG_PID_NS
scripts/config --enable CONFIG_NET_NS
scripts/config --enable CONFIG_CGROUP_PIDS
scripts/config --enable CONFIG_MEMCG
scripts/config --enable CONFIG_USER_NS
scripts/config --disable CONFIG_RANDOMIZE_BASE
scripts/config --enable CONFIG_CONFIGFS_FS
scripts/config --enable CONFIG_SECURITYFS
scripts/config --set-val CONFIG_DEFAULT_HUNG_TASK_TIMEOUT 140
scripts/config --set-val CONFIG_RCU_CPU_STALL_TIMEOUT 100
scripts/config --enable CONFIG_SOFTLOCKUP_DETECTOR
scripts/config --enable CONFIG_HARDLOCKUP_DETECTOR
scripts/config --enable CONFIG_BOOTPARAM_HARDLOCKUP_PANIC
scripts/config --enable CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC
scripts/config --enable CONFIG_DETECT_HUNG_TASK
scripts/config --enable CONFIG_WQ_WATCHDOG
scripts/config --enable CONFIG_TUN
make olddefconfig
More details in syzkaller doc.
The best is to use the above config for a bit of time, to build up the corpus.
Then you can add KASAN
:
scripts/config --enable CONFIG_KASAN
scripts/config --enable CONFIG_KASAN_INLINE
make olddefconfig
Then when the KASAN-kernel stops finding interesting things, progressively add one of these, then a mix:
scripts/config --enable CONFIG_DEBUG_KMEMLEAK
make olddefconfig
scripts/config --enable CONFIG_FAULT_INJECTION
scripts/config --enable CONFIG_FAULT_INJECTION_DEBUG_FS
scripts/config --enable CONFIG_FAULT_INJECTION_USERCOPY
scripts/config --enable CONFIG_FAILSLAB
scripts/config --enable CONFIG_FAIL_PAGE_ALLOC
scripts/config --enable CONFIG_FAIL_MAKE_REQUEST
scripts/config --enable CONFIG_FAIL_IO_TIMEOUT
scripts/config --enable CONFIG_FAIL_FUTEX
make olddefconfig
scripts/config --enable CONFIG_LOCKDEP
scripts/config --enable CONFIG_PROVE_LOCKING
scripts/config --enable CONFIG_DEBUG_ATOMIC_SLEEP
scripts/config --enable CONFIG_PROVE_RCU
scripts/config --enable CONFIG_DEBUG_VM
scripts/config --enable CONFIG_FORTIFY_SOURCE
scripts/config --enable CONFIG_HARDENED_USERCOPY
scripts/config --enable CONFIG_LOCKUP_DETECTOR
make olddefconfig
Before running syzkaller you will need to create a config file. Here is a config.cfg
template, with some exclusions for system calls that can consume significant test time but are not relevant for MPTCP. Be sure to substitute the directory names as needed for your system anywhere you see $ENVIRONMENT_VARIABLE-style
names (except the disable_syscalls
list). You can also increase the vm count variable depending on your available CPU cores and memory.
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "/home/$USER/gopath/src/github.com/google/syzkaller/$YOUR_WORK_DIR",
"kernel_obj": "$YOUR_BUILD_DIR",
"kernel_src": "$YOUR_SRC_DIR",
"image": "$YOUR_SYZ_IMAGE_DIR/stretch.img",
"sshkey": "$YOUR_SYZ_IMAGE_DIR/stretch.id_rsa",
"syzkaller": "/home/$USER/gopath/src/github.com/google/syzkaller",
"disable_syscalls": ["perf_event_open", "syz_mount_image", "syz_read_part_table", "openat$ttyprintk", "mount", "mkdir",
"openat$ptmx", "mq_open", "fsetxattr", "rt_tgsigqueueinfo", "ioctl$VT_RESIZE", "ioctl$TIOCVHANGUP", "get_robust_list",
"openat$nullb", "ioctl$SCSI_IOCTL_SEND_COMMAND", "lremovexattr", "mknod$loop", "write$binfmt_script", "syz_open_dev$sg", "write$nbd",
"prlimit64", "write$P9_RRENAMEAT", "fcntl$addseals", "finit_module", "ioctl$KDSETMODE", "write$FUSE_NOTIFY_STORE",
"ioctl$TIOCL_SETVESABLANK", "fsmount", "socket$vsock_stream", "socketpair$unix", "socket$nl_audit", "connect$unix", "bind$unix"],
"procs": 24,
"type": "qemu",
"vm": {
"count": 12,
"kernel": "$YOUR_BUILD_DIR/arch/x86/boot/bzImage",
"cpu": 2,
"mem": 2048
}
}
(feel free to change the number of procs
, vm count
, cpu
and mem
)
In addition to the syzkaller configuration, your system will need to have QEMU installed. Before running syzkaller the first time, you will need to create a vm image with the create-image.sh
tool that's part of syzkaller and is mentioned in the VM setup section of the syzkaller docs. That script depends on the debootstrap
tool, which is available on many Linux distros (Debian-based or not).
Once you have golang, syzkaller, your config, the VM image, and your kernel build all in place, you run syzkaller with:
$ cd $SYZPATH
$ bin/syz-manager -config config.cfg
syzkaller can generate reproducer programs. Sometimes these reproducers are included in MPTCP bug reports. See the syzkaller "reproducing crashes" docs for information about running these.
The MPTCP project has a MPTCP fork of packetdrill. That github project page has instructions for building packetdrill. To run the MPTCP tests, use:
$ cd $PACKETDRILL_WORKING_DIR/gtests/net
$ sudo ./packetdrill/run_all.py -vslS mptcp/
Note that a docker image can be used to prepare a dev environment ready to run tests and more. For more details, see here.
In some cases, the kernel will print a stacktrace -- you should see Call Trace:
in the middle in the kernel logs. Running the decode_stacktrace.sh
script will resolve addresses, and improve the output:
./scripts/decode_stacktrace.sh /PATH/TO/vmlinux /PATH/TO/kernel-source-dir
This script is not specific to MPTCP, you can find more examples on how to use it elsewhere on the Internet.
The vmlinux
image had to be built with debugging info. If the kernel was installed via a package manager -- e.g. default installation on most GNU/Linux distribution like Debian, Ubuntu, RHEL, Fedora, CentOS, etc. -- you probably need to install an extra package like linux-image-$(uname -r)-dbgsym
, kernel-(...)-debug
or kernel-(...)-dbg
. Refer to you distribution's documentation to find out what to do in your case.
If you compiled your kernel yourself, you can get more details there for example.
If you use the docker image, the stacktrace is automatically decoded in "auto" mode. In manual mode, you can use:
cd <kernel source dir>
cat stacktrace.txt | ./scripts/decode_stacktrace.sh .virtme/build/vmlinux $PWD .virtme/build/.virtme_mods