Building and profiling with gperftools

Setting up gperftools

• Change to the directory where you want to build gperftools.
• Download and run the build script found here: https://gist.github.com/e-harvey/ac98dea070fec4a73ef9c48434e23c9c#file-install_gperftools-sh.

Building libfabric for profiling with gperftools

We need to build libfabric and the application with the gcc flag -fno-omit-frame-pointer. See https://github.com/gperftools/gperftools/blob/master/INSTALL for more information.

• Building libfabric ./autogen.sh && ./configure CFLAGS="-fno-omit-frame-pointer" --prefix="$PWD/libfabric-test" && make clean && make -j install

Building cray-tests for profiling with gperftools

• Use the --with-profiler option during the cray-tests configure step ./autogen.sh && ./configure --prefix=$PWD/../cray-tests-build --with-libfabric=$PWD/../libfabric-test --with-pmi=/opt/cray/pmi/default --with-profiler=/path/to/gperftools-build && make -j install

Configuring an autotools application for profiling with gperftools

./configure CFLAGS=-fno-omit-frame-pointer CPPFLAGS=-I"$PWD/gperftools-build/include/" LDFLAGS=-L"$PWD/lib/" LIBS=-lprofiler

Creating an executable for the CPU profiler from gperftools

• We need to manually compile a separate source file with a main function since pprof uses the binary to generate the profiling output. Using gnitest as the binary does not work. Here is an example of how the source code should look https://github.com/ofi-cray/cray-tests/blob/master/contrib/buddy_test.c. Note that you need to include <gperftools/profiler.h>.
• Once you have your source code and your calls to ProfilerStart("name.prof") and ProfilerStop() where you want them and have built libfabric in $PWD/libfabric-build compile it like this on tiger: gcc -fno-omit-frame-pointer -Wall `pkg-config cray-ugni --cflags --libs` `pkg-config cray-gni-headers --cflags --libs` `pkg-config cray-udreg --cflags --libs` -I $PWD/gperftools-build/include/ -I $PWD/libfabric-cray/include/ -I $PWD/prov/gni/include/ -L$PWD/libfabric-build/lib/ -L$PWD/gperftools-build/lib/ -I $PWD/libfabric-cray -lfabric -lprofiler buddy_test.c -o buddy_test $PWD/libfabric-build/lib/libfabric.a

Running the executable

• To profile the entire application: export CPUPROFILE=ProfileName.prof
• export CPUPROFILE_FREQUENCY=1000000 (this can be increased to have the profiler peek at the top of the call stack more frequently)
• Run it manually or with a job launcher

Using pprof

pprof --text a.out buddy.test

Gotchas

• Don't forget to include gperftools/gperftools.h.
• Wrap the section of code you're interested in in ProfilerStart("ProfileName.prof") and ProfilerStop().
• Remember that if the job runs multiple ranks/threads/messageSizes and the ProfileName.prof is named the same for each rank/thread/messageSize the last instance to run will stomp on all the other profiles.

Notes

• The cpu profiler should work with multiple threads, according to the gperf documentation. However, sometimes the profiles show calls to pthread routines even if the ProfileStart and ProfileStop calls are wrapped around code that doesn't call into pthread.
• There are many more output types and additional options for using the profiler that can be found here: http://htmlpreview.github.com/?https://github.com/gperftools/gperftools/blob/master/docs/cpuprofile.html.