Slim down of libbcc.so

[joelagnel]

For Android usecases, I am trying to bring down the size of libbcc.so. Currently it is at 55MB.

Disassembling it, I see that there are a lot of references to the x86 LLVM backend. This seems a bit useless since libbcc.so will only generate eBPF backend machine code.

I built my own LLVM without the x86 backend and only with eBPF backend.

git clone http://llvm.org/git/llvm.git
cd llvm/tools; git clone http://llvm.org/git/clang.git
cd ..; mkdir -p build/install; cd build
cmake -G "Unix Makefiles" -DLLVM_TARGETS_TO_BUILD="BPF" \
  -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/install ..
make
make install
export PATH=$PWD/install/bin:$PATH

Now I am trying to build BCC with that LLVM I just built using:

rm -rf build && mkdir -p build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=./install -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
make -j90 VERBOSE=1
make install

However that fails because the "clang" I built obviously doesn't have the x86 backend so it cannot compile BCC itself.

What I want is to use the "stock" LLVM in my distro to build BCC, however for libbcc itself, I want it to link to the LLVM I just built with only the eBPF backend. So basically that looks like:

LLVM libs with only BPF backend         BCC Sources
                  |                        /
                  |                      /
Compile/Link with LLVM compiler with x86 backend (in distro)
                                     |
                              Compact libbcc.so with only eBPF backend

This seems a bit tricky, but I'll keep poking. Any thoughts / guidance on how I can accomplish this @drzaeus77 @yonghong-song ?

[joelagnel]

Also if we can get rid of exceptions, that brings down the size of libbcc.so from 55MB to 46MB as .eh_frame section will not be generated and those are huge. How important are C++ exceptions for BCC? Can we use any alternatives to exceptions, such as maybe logging errors?

[brendangregg]

Maybe you already thought of it, but you should be able to use objcopy(1) to remove the .eh_frame section, and other unnecessary sections (-R .ehframe -R ...). That way you can take the packaged libbcc.so and apply a trimming script.

[joelagnel]

Yes, I can use strip also to get rid of eh_frame:

strip --remove-section=.eh_frame build/src/cc/libbcc.so.0.8.0

But I am not sure if it will break something (only build tested cos I'm on a plane :D) related to exceptions. I'll test it more. I thought a cleaner solution was if we could make BCC not use exceptions and build it with -fno-exceptions. But I was not fully sure if BCC project is Ok with that.

[yonghong-song]

To compile a tracing program, the llvm needs to support two targets: native host (e.g, arm64) and BPF. The arm64 is needed so all the kernel headers linked into bpf program are properly processed so that bpf program has the same view for the kernel data structures as the kernel itself. In your above, your llvm only builds with BPF backend, which is not sufficiently. llvm majority code is not in target. Maybe adding arm64 target should not increase the its size too big?

For exceptions in bcc. yes, currently bcc uses it. I am okay to replace exceptions with proper error output.