22. Develop Icinga 2

This chapter provides hints on Icinga 2 development especially for debugging purposes.


If you are planning to build your own development environment, please consult the INSTALL.md file from the source tree.

22.1. Debug Requirements

Make sure that the debug symbols are available for Icinga 2. The Icinga 2 packages provide a debug package which must be installed separately for all involved binaries, like icinga2-bin or icinga2-ido-mysql.


# apt-get install icinga2-dbg


# yum install icinga2-debuginfo


# zypper install icinga2-bin-debuginfo icinga2-ido-mysql-debuginfo

Furthermore, you may also have to install debug symbols for Boost and your C library.

If you're building your own binaries, you should use the -DCMAKE_BUILD_TYPE=Debug cmake build flag for debug builds.

22.2. GDB

Install gdb:


# apt-get install gdb


# yum install gdb


# zypper install gdb

Install the boost, python and icinga2 pretty printers. Absolute paths are required, so please make sure to update the installation paths accordingly (pwd).

$ mkdir -p ~/.gdb_printers && cd ~/.gdb_printers

Boost Pretty Printers compatible with Python 3:

$ git clone https://github.com/mateidavid/Boost-Pretty-Printer.git && cd Boost-Pretty-Printer
$ git checkout python-3
$ pwd

Python Pretty Printers:

$ cd ~/.gdb_printers
$ svn co svn://gcc.gnu.org/svn/gcc/trunk/libstdc++-v3/python

Icinga 2 Pretty Printers:

$ mkdir -p ~/.gdb_printers/icinga2 && cd ~/.gdb_printers/icinga2
$ wget https://raw.githubusercontent.com/Icinga/icinga2/master/tools/debug/gdb/icingadbg.py

Now you'll need to modify/setup your ~/.gdbinit configuration file. You can download the one from Icinga 2 and modify all paths.

Example on Fedora 22:

$ wget https://raw.githubusercontent.com/Icinga/icinga2/master/tools/debug/gdb/gdbinit -O ~/.gdbinit
$ vim ~/.gdbinit

set print pretty on

import sys
sys.path.insert(0, '/home/michi/.gdb_printers/icinga2')
from icingadbg import register_icinga_printers

import sys
sys.path.insert(0, '/home/michi/.gdb_printers/python')
from libstdcxx.v6.printers import register_libstdcxx_printers

import sys
sys.path.insert(0, '/home/michi/.gdb_printers/Boost-Pretty-Printer')
import boost_print

If you are getting the following error when running gdb, the libstdcxx printers are already preloaded in your environment and you can remove the duplicate import in your ~/.gdbinit file.

RuntimeError: pretty-printer already registered: libstdc++-v6

22.2.1. GDB Run

Call GDB with the binary (/usr/sbin/icinga2 is a wrapper script calling /usr/lib64/icinga2/sbin/icinga2 since 2.4) and all arguments and run it in foreground.

# gdb --args /usr/lib64/icinga2/sbin/icinga2 daemon -x debug --no-stack-rlimit

The exact path to the Icinga 2 binary differs on each distribution. On Ubuntu it is installed into /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 on 64-bit systems for example.


If gdb tells you it's missing debug symbols, quit gdb and install them: Missing separate debuginfos, use: debuginfo-install ...

Run the application.

(gdb) r

Kill the running application.

(gdb) k

Continue after breakpoint.

(gdb) c

22.2.2. GDB Core Dump

Either attach to the running process using gdb -p PID or start a new gdb run.

(gdb) r
(gdb) generate-core-file

22.2.3. GDB Backtrace

If Icinga 2 aborted its operation abnormally, generate a backtrace.

(gdb) bt
(gdb) thread apply all bt full

If gdb stops at a SIGPIPE signal please disable the signal before running Icinga 2.

(gdb) handle SIGPIPE nostop noprint pass
(gdb) r

If you create a bug report, make sure to attach as much detail as possible.

22.2.4. GDB Backtrace from Running Process

If Icinga 2 is still running, generate a full backtrace from the running process and store it into a new file (e.g. for debugging dead locks):

# gdb -p $(pidof icinga2) -batch -ex "thread apply all bt full" -ex "detach" -ex "q" > gdb_bt.log

22.2.5. GDB Backtrace Stepping

Identifying the problem may require stepping into the backtrace, analysing the current scope, attributes, and possible unmet requirements. p prints the value of the selected variable or function call result.

(gdb) up
(gdb) down
(gdb) p checkable
(gdb) p checkable.px->m_Name

22.2.6. GDB Breakpoints

To set a breakpoint to a specific function call, or file specific line.

(gdb) b checkable.cpp:125
(gdb) b icinga::Checkable::SetEnablePerfdata

GDB will ask about loading the required symbols later, select yes instead of no.

Then run Icinga 2 until it reaches the first breakpoint. Continue with c afterwards.

(gdb) run
(gdb) c

If you want to delete all breakpoints, use d and select yes.

(gdb) d


When debugging exceptions, set your breakpoint like this: b __cxa_throw.

Breakpoint Example:

(gdb) b __cxa_throw
(gdb) r
(gdb) up
(gdb) up
#11 0x00007ffff7cbf9ff in icinga::Utility::GlobRecursive(icinga::String const&, icinga::String const&, boost::function<void (icinga::String const&)> const&, int) (path=..., pattern=..., callback=..., type=1)
    at /home/michi/coding/icinga/icinga2/lib/base/utility.cpp:609
609         callback(cpath);
(gdb) l
605 #endif /* _WIN32 */
607     std::sort(files.begin(), files.end());
608     BOOST_FOREACH(const String& cpath, files) {
609         callback(cpath);
610     }
612     std::sort(dirs.begin(), dirs.end());
613     BOOST_FOREACH(const String& cpath, dirs) {
(gdb) p files
$3 = std::vector of length 11, capacity 16 = {{static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/agent.conf"}, {static NPos = 18446744073709551615,
    m_Data = "/etc/icinga2/conf.d/commands.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/downtimes.conf"}, {static NPos = 18446744073709551615,
    m_Data = "/etc/icinga2/conf.d/groups.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/notifications.conf"}, {static NPos = 18446744073709551615,
    m_Data = "/etc/icinga2/conf.d/satellite.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/services.conf"}, {static NPos = 18446744073709551615,
    m_Data = "/etc/icinga2/conf.d/templates.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/test.conf"}, {static NPos = 18446744073709551615,
    m_Data = "/etc/icinga2/conf.d/timeperiods.conf"}, {static NPos = 18446744073709551615, m_Data = "/etc/icinga2/conf.d/users.conf"}}

22.3. Core Dump

When the Icinga 2 daemon crashes with a SIGSEGV signal a core dump file should be written. This will help developers to analyze and fix the problem.

22.3.1. Core Dump File Size Limit

This requires setting the core dump file size to unlimited.

Example for Systemd:

vim /usr/lib/systemd/system/icinga2.service


systemctl daemon-reload

systemctl restart icinga2

Example for init script:

vim /etc/init.d/icinga2
ulimit -c unlimited

service icinga2 restart

Verify that the Icinga 2 process core file size limit is set to unlimited.

cat /proc/`pidof icinga2`/limits
Max core file size        unlimited            unlimited            bytes

22.3.2. Core Dump Kernel Format

The Icinga 2 daemon runs with the SUID bit set. Therefore you need to explicitly enable core dumps for SUID on Linux.

sysctl -w fs.suid_dumpable=1

Adjust the coredump kernel format and file location on Linux:

sysctl -w kernel.core_pattern=/var/lib/cores/core.%e.%p

install -m 1777 -d /var/lib/cores


sysctl -w kern.corefile=/cores/core.%P

chmod 777 /cores

22.3.3. Core Dump Analysis

Once Icinga 2 crashes again a new coredump file will be written. Please attach this file to your bug report in addition to the general details.

Simple test case for a SIGSEGV simulation with sleep:

ulimit -c unlimited
sleep 1800&
[1] <PID>
kill -SEGV <PID>
gdb `which sleep` /var/lib/cores/core.sleep.<PID>
(gdb) bt
rm /var/lib/cores/core.sleep.*

Analyzing Icinga 2:

gdb /usr/lib64/icinga2/sbin/icinga2 core.icinga2.<PID>
(gdb) bt