CI: test on arm64

[shrkamat]

I see failure in running kprobe example for aarch64 kernel version 4.19.113

GOOS=linux GOARCH=arm64 go build -o ebpf-kprobe

./ebpf-kprobe 
2021/03/25 22:03:40 error while loading objects: field KprobeExecve: program kprobe_execve: load program without BTF: invalid argument

looks like BPF_PROG_LOAD is failing

[lmb]

We don't have arm64 on CI, so the support is best effort. I think the real cause of the error is that loading BTF is bailing out for some reason. Without access to arm64 it'll be hard to debug.

[ti-mo]

Thanks for reporting. I have some Pi4's laying around that I still need to set up, but we'll likely only be able to support this properly once we can build and test on arm64 CI kernels. There's some experimental arm64 support in virtme, but haven't tried it yet. Might be on the roadmap at some point, will leave this open to keep track.

[lmb]

FWIW I suspect that this particular error is because ./ebpf-kprobe was run without root privileged.

[ti-mo]

Reopening as #304 was only kprobe-related.

[ti-mo]

Finally had the chance to look at this. Have ALARM set up on a rpi4, and this is the test output: https://gist.github.com/ti-mo/033e7de02d58da9abc6a2bac91458291

Everything seems to work for the most part, but it seems like the following map/prog types just don't work on this kernel build:

--- FAIL: TestHaveMapType (0.10s)
    --- FAIL: TestHaveMapType/XSKMap (0.00s)
        map_test.go:64: Map type XSKMap isn't supported even though kernel is at least 4.18: not supported
    --- FAIL: TestHaveMapType/InodeStorage (0.00s)
        map_test.go:64: Map type InodeStorage isn't supported even though kernel is at least 5.10: not supported
    --- FAIL: TestHaveMapType/TaskStorage (0.00s)
        map_test.go:64: Map type TaskStorage isn't supported even though kernel is at least 5.11: not supported
--- FAIL: TestHaveProgType (0.00s)
    --- FAIL: TestHaveProgType/RawTracepointWritable (0.00s)
        prog_test.go:76: Program type RawTracepointWritable isn't supported even though kernel is at least 5.2: not supported
    --- FAIL: TestHaveProgType/RawTracepoint (0.00s)
        prog_test.go:76: Program type RawTracepoint isn't supported even though kernel is at least 4.17: not supported
    --- FAIL: TestHaveProgType/PerfEvent (0.00s)
        prog_test.go:76: Program type PerfEvent isn't supported even though kernel is at least 4.9: not supported
    --- FAIL: TestHaveProgType/Kprobe (0.00s)
        prog_test.go:76: Program type Kprobe isn't supported even though kernel is at least 4.1: not supported
    --- FAIL: TestHaveProgType/TracePoint (0.00s)
        prog_test.go:76: Program type TracePoint isn't supported even though kernel is at least 4.7: not supported

This matches bpftool output:

[timo@alarm-rpi4 ebpf]$ sudo bpftool feature probe | grep NOT
eBPF program_type kprobe is NOT available
eBPF program_type tracepoint is NOT available
eBPF program_type perf_event is NOT available
eBPF program_type raw_tracepoint is NOT available
eBPF program_type lirc_mode2 is NOT available
eBPF program_type raw_tracepoint_writable is NOT available
eBPF program_type cgroup_sockopt is NOT available
eBPF program_type tracing is NOT available
eBPF program_type ext is NOT available
eBPF program_type lsm is NOT available
eBPF map_type xskmap is NOT available
eBPF map_type struct_ops is NOT available
eBPF map_type inode_storage is NOT available
eBPF map_type task_storage is NOT available

It seems like, for now, the error reported here is expected. I've played around with arm32 on an older Pi before, and attempting to use kprobes/tracepoints there resulted in consistent kernel panics. I think it's just been disabled in recent builds, or the kernel support is just not ready.

I'll slowly investigate some of the other failures too, but they all look related to either u/kprobes, perf, or missing vmlinux BTF somehow.

[lmb]

I'm now developing on an m1 in a VM, and I see a bunch of errno 524 for unimplemented stuff. Is that the case on your rpi as well?

[ti-mo]

Yes, this was the case back when I was briefly testing this. Don't have the Pi's handy atm, and iirc the one running on it didn't have a bunch of kernel options enabled.

My idea was to (eventually) build kernel and userspace images for something like https://github.com/weaveworks/ignite, but QEMU-driven instead of Firecracker, since it doesn't support 9pfs/virtiofs. Not sure if we can get more mileage out of virtme, at least not with arm guests on a x86 host.

[florianl]

I don't have access to ARM64 hardware either and so fall back to virtual environments for testing low level Kernel APIs like netlink or ebpf.
This is the approach I use so far:

# Cross compile a Linux kernel for arm64
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
$ ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j $(nproc)


# Create a statically linked executable from the tests in github.com/cilium/ebpf
$ GOARCH=arm64 GOOS=linux go test -ldflags='-extldflags=-static' -trimpath -tags 'osusergo netgo static_build linux' -c
$ file ebpf.test 
ebpf.test: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=WY90hJx458GNGAub3oIg/dTyiwLPimJ2BRRTReGBd/iVHqsVdn5p0icxIQlPF9/auBWq5LKg0Fr0xpEELk_, not stripped

# Embedd the statically linked executable with the tests into a minimal initramfs
$ bluebox -a arm64 -e ebpf.test:"-test.v"

# Run the tests in a virtual environment
$ qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 2 -m 4096 -kernel ./src/git.kernel.org/torvalds/linux/arch/arm64/boot/Image -initrd ./go/src/github.com/cilium/ebpf/initramfs.cpio

The limitation of bluebox is that one can execute/test only statically linked executables as it does not provide a full Unix environment. On the other hand it can be integrated into a CI/CD pipeline quite easy and comes without dependencies (except for Go) that are hard to manage.

[lmb]

Thanks for the link to bluebox, that is a cool idea!