Build

Introduction

Likwid is built using GNU make and Perl. Besides the Linux kernel and the standard C library, all required dependencies are shipped with the archive (Lua and hwloc) and GOTCHA (for NVIDIA GPU support, see here). It should build on any Linux distribution with a recent GCC or CLANG compiler and 2.6 or newer Linux kernel without any changes.

There is one generic top level Makefile and one .mk configuration file for each compiler (at the moment GCC, CLANG and ICC). Please note that we test LIKWID only with GCC. CLANG and ICC is only tested for basic functionality.

There is one exception: If you want to use LIKWID on a Intel Xeon Phi card (KNC) you have to choose the MIC as Compiler in config.mk, which is based on Intel ICC compiler.

Directory structure

All source files are in the src directory. All header files are located in src/includes . Lua application source files are in src/applications. All external tools, namely HWLOC and Lua, are located in ext/. The bench/ folder contains all files of the benchmarking suite of LIKWID.

All build products are generated in the directory ./TAG, where TAG is the compiler configuration, default ./GCC.

Configuration

Usually the only thing you have to configure is the PREFIX install path in the build config file config.mk in the top directory. (Note: No + signs in the prefix path and it has to be an absolute path)

Changing color of likwid-pin output

Depending on the background of your terminal window you can choose a color for likwid-pin output. It can be set in the build config file config.mk at the variable COLOR. Possible colors are: NONE, BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN or WHITE.

Usage of access daemon likwid-accessD

Usually on your own system, you can use LIKWID with direct access to the MSR files. If you install LIKWID on a shared system as a HPC compute cluster you may consider to use the access daemon likwid-accessD. This is a proxy application which was implemented with security in mind and performs address checks for allowed access. Using the access daemon, the measurements involve more overhead, especially if you use likwid-perfctr in timeline mode or with the marker API.

To enable using the access daemon, configure in config.mk:

1. Set BUILDDAEMON to true
2. Configure the path to the access daemon binary at ACCESSDAEMON
3. Set the ACCESSMODE to accessdaemon

ACCESSMODE can be direct, accessdaemon, perf_event and sysdaemon (not yet officially supported). You can overwrite the default setting on the command line with the -M commandline option.

If you want to access Uncore performance counters that are located in the PCI memory range, like they are implemented in Intel SandyBridge EP and IvyBridge EP and newer architectures, you have to use the access daemon or have root privileges because access to the PCI space is only permitted for high privileged users.

Usage of frequency daemon likwid-setFreq

The application likwid-setFrequencies uses another daemon to modify the frequency of CPUs. The daemon is build and later installed if BUILDFREQ is set to true in config.mk.

Build LIKWID as shared library

Per default the LIKWID library is build as a shared library. You need the library if you want to use the Marker API. You can also use the LIKWID modules like perfmon directly. In some settings it is necessary to build LIKWID as a shared library. To do so set SHARED_LIBRARY to true in config.mk.

Instrument likwid-bench for usage with likwid-perfctr

likwid-bench is instrumented for use with likwid-perfctr. This allows you to measure various metrics of your likwid-bench kernels. Enable instrumentation by setting INSTRUMENT_BENCH to true in config.mk.

Build LIKWID with perf_event backend instead of native access

LIKWID can be built to run on top of perf_event (config.mk: USE_PERF_EVENT in 4.2 and ACCESSMODE=perf_event in 4.3), an interface of the Linux kernel to the hardware performance monitors. The events and counters are the same but not all might be supported. LIKWID supports the Uncore and RAPL (energy) units provided by perf. The thermal module is currently not supported with perf_event as perf_event does not provide thermal information. For core-local measurements the paranoid level of 1 is enough to measure applications. To get the same behavior as native access, the paranoid level must be 0 or less. 0 or less is also required for Uncore measurements.

Be aware that LIKWID reads information out of registers that is not provided by any other source like procfs and sysfs. When switching to perf_event backend, these registers cannot be accessed and less information is printed. The different CPU core frequencies in turbo mode (likwid-powermeter -i) serve as an example. If you want to use perf_event for measurements and the access daemon for other operations, install LIKWID first with ACCESSMODE=accessdaemon and followed by make distclean, change ACCESSMODE=perf_event in config.mk and then build and install LIKWID again.

If you have no root permissions for make install, please set BUILDFREQ=false in config.mk. The build of the access daemon is disabled when ACCESSMODE=perf_event. If you encounter problems, you can manually disable it by setting BUILDDAEMON=false in config.mk

Enabling Fortran interface for Marker API

If you want to use the Marker API in Fortran programs, LIKWID offers a native Fortran90 interface. To enable it, set FORTRAN_INTERFACE to true in config.mk. More information using LIKWID with Fortran look at Marker API with Fortran90.

Use system Lua instead of internal Lua

By setting the variables LUA_INCLUDE_DIR, LUA_LIB_DIR, LUA_LIB_NAME (Lua interpreter has to use the same name, e.g. lua5.1) and LUA_BIN (directory of the Lua interpreter) in config.mk, the internal Lua library and interpreter are deactivated. The LIKWID library is linked against the provided Lua library and the Shebang of all Lua scripts is changed to the system Lua interpreter. LIKWID's Lua interface was patched to support Lua 5.1, 5.2 and 5.3. It is still recommended to use the internal Lua and only basic tests are done before a release. Lua 5.1 was not usable with LIKWID 5.0.x, fixed in 5.1.x.

Dependencies

Although we tried to minimize the external dependencies of LIKWID, some advanced tools or only specific tool options require external packages.

• likwid-perfscope uses the Perl script feedGnuplot to forward the real-time data to gnuplot. feedGnuplot is included into LIKWID, but gnuplot itself is not.
• likwid-agent provided multiple backends to output the periodically measured data. The syslog backend requires the shell tool logger to be installed. The RRD backend requires rrdtool and the GMetric backend the gmetric tool, part of the Ganglia Monitoring System.
• In order to create the HTML documentation of LIKWID, the tool Doxygen is required.

Build targets

You have to edit config.mk to configure your build and install path.

The following make targets are available:

• make - Build everything
• make likwid-bench - Build likwid-bench
• make likwid-accessD - Build likwid-accessD
• make likwid-setFreq - Build likwid-setFreq
• make docs - Create HTML documentation using doxygen
• make clean - Remove the object file directory ./GCC, keep the executables. If you encounter problems after make clean, please try make distclean
• make distclean - Remove all generated files
• make local - Adjust paths in Lua scripts to work from the build directory. Requires the daemons to be already installed. Mainly used for testing.

The build system has a working dependency tracking, therefore make clean is only needed if you change the Makefile configuration.

Installing

NOTE: The pinning functionality and the daemons only work if configured in config.mk and installed with make install. If you do not use the pinning functionality the tools can be used without installation.

• make install - Installs the executables, libraries, man pages and headers to the PREFIX path you configured in config.mk.
• make uninstall - Delete all installed files. If you changed the INSTALLED_PREFIX in config.mk you can use:
• make move - Copy all files from the PREFIX path to the INSTALLED_PREFIX path defined in config.mk
• make uninstall_moved - Delete all installed and moved files from the INSTALLED_PREFIX path

Normal configuration in config.mk sets INSTALLED_PREFIX = PREFIX, so in most cases, make && (sudo) make install is sufficient to build and install LIKWID.

Setting up access for hardware performance monitoring

Hardware performance monitoring on x86 is enabled using model-specific registers (MSR). MSR registers are special registers not part of the instruction set architecture. To read and write to these registers the x86 ISA provides special instructions. These instructions can only be executed in protected mode or in other words only kernel code can execute these instructions. Fortunately, any Linux kernel 2.6 or newer provides access to these registers via a set of device files. This allows to implement all of the functionality in user space. Still it does not allow to use those more advanced features of hardware performance monitoring which require to setup interrupt service routines or kernel located memory.

Per default only root has read/write access to these MSR device files. In order to use the LIKWID tools, which need access to these files (likwid-perfctr, likwid-powermeter and likwid-agent) as standard user, you need to setup access rights to these files.

likwid-perfctr, likwid-powermeter and likwid-agent require the Linux msr kernel module. This module is part of most standard distro kernels. You have to be root to do the initial setup.

- Check if the `msr` module is loaded with `lsmod | grep msr`. There should be an output.
- It the module is not loaded, load it with `modprobe msr`. For automatic loading at startup consult your distros documentation how to do so.
- Adopt access rights on the MSR device files for normal user. To grant access to anyone,  you can use `chmod o+rw /dev/cpu/*/msr`. This is only recommended on single user desktop systems.
- For Uncore, make sure to grant access rights to `/dev/mem` as well. Be aware that this let's the user easily exploit the entire machine. Use the access daemon for higher security.

As in general access to MSRs is not desired on security sensitive systems, you can either implement a more sophisticated access rights settings with e.g. setgid. A common solution used on many other device files, e.g. for audio, is to introduce a group and make a chown on the MSR device files to that group. Now if you execute likwid-perfctr with setgid on that group, the executing user can use the tool but cannot directly write or read the MSR device files.

Some distributions backported the capabilities check for the MSR device to older kernels. If there are problems with accessing the MSR devices for older kernels with file system permissions set to read&write, please check your kernel code (arch/x86/kernel/msr.c) for the backport and set the MSR capabilities in case.

A secure solution is to use the access daemon likwid-accessD, which encapsulates the access to the MSR device files and performs a address check for allowed registers.

Some newer kernels implement the so-called capabilities, a fine-grained permission system that can allow access to the MSR files for common users. On the downside, it may be not enough anymore to set the suid-root flag for the access daemon, the executable must be registerd at the libcap.

sudo setcap cap_sys_rawio+ep EXECUTABLE

This is only possible on local file systems. A feasible way is to use the likwid-accessD for all accesses and just enable the capabilities for this one binary. This will enable the usage for all LIKWID tools and also for all instrumented binaries. If the likwid-perfctr utility should only be used in wrapper mode, it is suitable to set the capabilities for likwid-perfctr only. Please remember to set the file permission of the MSR device file to read/write for all users, even if capabilites are configured correctly.

If you are using cgroups and access is not allowed although you tried all the steps above, check whether access to the msr device files is denied by cgroup's devices.deny or not explicitly allowed in devices.allow.

Update for Linux kernel 5.9 and newer: With Linux 5.9, the msr kernel module got some security fixes. The major change for LIKWID is, that now all MSR are non-writable by default. In order to change that, you have to change the boot options of your operating system to contain msr.allow_writes=on to enable writes again. This affects only ACCESSMODE=direct and ACCESSMODE=accessdaemon. If you use the perf_event backend, you don't have to change anything.

Update for Linux kernel 5.10 and newer: We got reports, that with Linux 5.10 the PCI accesses are also restricted by security mechanisms. In order to fix this, the access daemon requires an additinal capabilities flag:

sudo setcap cap_sys_admin,cap_sys_rawio=ep EXECUTABLE

Installation on Intel Xeon Phi (KNC)

In order to build LIKWID for the Xeon Phi, set the COMPILER option in config.mk to MIC and build it on a host that has a MIC-aware Intel compiler and the MPSS installed. Moreover, there might be the problem that the required Intel libraries are not present on the Xeon Phi and you need to copy them. Commonly the libraries are libimf.so, libsvml.so, libirng.so and libintlc.so.5. LIKWID assumes that they are copied in the same path as the LIKWID library liblikwid.so ($INSTALLED_PREFIX/lib). If you install them somewhere different, please set the RPATHS variable in config.mk or make/include_MIC.mk to point to the location.

Moreover, the Intel Xeon Phi has some limit of running processes and since every CPU needs its access daemon, LIKWID would start many processes. Therefore, LIKWID allows only direct access to the MSR device files on Xeon Phi. In order to allow this, set the suid-root bit for the Lua interpreter (sudo chmod u+s $INSTALLED_PREFIX/bin/likwid-lua).

Installation on Intel Xeon Phi (KNL)

The successor of the KNC has more similarities with normal system CPUs. Therefore, no special compiler flags are needed during the build process, the default compiler GCC can be used. The only thing that needs to be adjusted is the number of supported hardware threads. In config.mk the variable MAX_NUM_THREADS must be at least the number of hardware threads, so for a 72 core KNL you have to set the variable to minimum 288. Moreover, there might be the problem that the required Intel libraries are not present on the Xeon Phi and you need to copy them. Commonly the libraries are libimf.so, libsvml.so, libirng.so and libintlc.so.5. LIKWID assumes that they are copied in the same path as the LIKWID library liblikwid.so ($INSTALLED_PREFIX/lib). If you install them somewhere different, please set the RPATHS variable in config.mk or make/include_.mk to point to the location.

Installation on ARM-based systems

ARM support was added in January. The main switch is the COMPILER setting in config.mk. There are two possibilities: GCCARMv7 and GCCARMv8. For build flags changes, please use the appropriate file make/include_<COMPILER>.mk. The backend for ARM is perf_event. There is a native backend as well but it is currently not usable as the user would need to measure multiple times per second to catch all register overflows. As soon as LIKWID starts a management thread to read the registers in the background, I will publish this backend as well.

Installation on POWER-based systems

The main switch is the COMPILER setting in config.mk. The change to GCCPOWER configures the build to work on POWER and switch to the proper ACCESSMODE.

LIKWID and SecureBoot

LIKWID in direct or accessdaemon mode uses the common MSR and PCI interfaces from the Linux kernel. Unfortunately, some recent operating systems (e.g. Ubuntu 18.04) patches the Linux kernel so that these interfaces (at least MSR) are not allowed to use anymore. You probably just get read/write errors because LIKWID currently has no check of the SecureBoot state. If you want to use LIKWID, you have to disable SecureBoot.

LIKWID as static library

If you want to compile LIKWID as static library, you have to set SHARED_LIBRARY in config.mk to false. You can integrate LIKWID in your application but you cannot use the command line tools (except likwid-bench). I havn't found a way to make the tools work because they are written in Lua and in order to load a C-library at runtime, Lua needs dlopen which only works with shared libraries.

LIKWID with NVIDIA GPU support

In order to build LIKWID with support for NVIDIA GPUs, you have to change NVIDIA_INTERFACE in config.mkto true. The Makefile uses the environment variable CUDA_HOME to determine the paths for the compilations but if needed, you can adjust the paths manually in CUDAINCLUDE and CUPTIINCLUDE. In case of problems, rebuild (make distclean && make) with BUILDAPPDAEMON set to true. At runtime, LIKWID uses dlopen to load the required functions, so the library paths to CUDA and CUPTI should be in LD_LIBRARY_PATH.

LIKWID with AMD GPU support

For LIKWID with support for AMD GPUs, you have the turn on the ROCM_INTERFACE with true in config.mk. The Makefile uses the environment variable ROCM_HOME to determine the paths for the compilation but if needed, the paths can be set manually in config.mk (HSAINCLUDE, ROCPROFILERINCLUDE, HIPINCLUDE, RSMIINCLUDE). At runtime, LIKWID uses dlopen to load the required functions, so the various ROCm related library paths should be in LD_LIBRARY_PATH.

LIKWID build fails with Perl errors in likwid-bench

likwid-bench uses the Perl Template Toolkit to generate the assembly files that are compiled into. This Perl module is outdated and might cause problems. We have seen errors like Can't call method "process" on an undefined value at ./perl/generatePas.pl line 202. The workaround is to delete the included version rm -r bench/perl/Template* and install it with the distribution's package manager. For Fedora the package is called perl-Template-Toolkit.

Setting LIKWID build options on the command line

If you want to minimize changes to the config.mk file but still configure the build, most configuration options in config.mk can also be set on the command line. This is quite helpful for package managers' build recipes or other scripted installation.

Example build with configuration options on the command line:

$ make PREFIX=/usr/mylocal COMPILER=GCC ACCESSMODE=perf_event

Make sure that you use the same options when performing other calls to make:

$ make PREFIX=/usr/mylocal COMPILER=GCC ACCESSMODE=perf_event install