Installation

The LTTng libraries that ship with Ubuntu 12.04 have been very buggy, and the generated header files using lttng-gen-tp have needed to be fixed just to compile in the Ceph tree. The packages available in Ubuntu 14.04 seem to work alright, and for older versions please install LTTng from the LTTng PPA.

https://launchpad.net/~lttng/+archive/ppa

Then install as normal

apt-get install lttng-tools liblttng-ust-dev

Add/Update Provider

Create tracepoint definition file

Add tracepoint definitions for the provider into a .tp file. Documentation on defining a tracepoint can be found in man lttng-ust. By convention files are named according to the logical sub-system they correspond to (e.g. mutex.tp, pg.tp). And add a C source file to be compiled into the tracepoint provider shared object, in which TRACEPOINT_DEFINE should be defined. See LTTng document for details. Place the .tp and the .c files into the src/tracing directory and modify the CMake file src/tracing/CMakeLists.txt accordingly.

Function Instrumentation

Ceph supports instrumentation using GCC's -finstrument-functions flag. Supported CMake flags are:

• -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON: instrument OSD code

Note that this instrumentation adds an extra function call on each function entry and exit of Ceph code. This option is currently only supported with GCC. Using it with Clang has no effect.

The only function tracing implementation at the moment is done using LTTng UST. In order to use it, Ceph needs to be configured with LTTng using -DWITH_LTTNG=ON. TraceCompass can be used to generate flame charts/graphs and other metrics.

It is also possible to use libbabeltrace to write custom analysis. The entry and exit tracepoints are called lttng_ust_cyg_profile:func_enter and lttng_ust_cyg_profile:func_exit respectively. The payload variable addr holds the address of the function called and the payload variable call_site holds the address where it is called. nm can be used to resolve function addresses (addr to function name).