Icinga Version 1.14 Documentation

Table of Contents

1. About

1.1. About Icinga

1.1.1. What is Icinga?

1.1.2. System requirements

1.1.3. Licensing

1.1.4. Acknowledgements

1.1.5. Downloading The Latest Version

1.1.6. Compatibility

1.2. What's New in Icinga 1.14

1.2.1.

1.2.2. More information

2. Getting Started

2.1. Advice for beginners

2.1.1. Icinga Overview

2.1.2. Icinga web GUI comparison

2.2. I'm a newbie to ...

2.2.1. Linux

2.2.2. Mail

2.2.3. SNMP

2.2.4. Packages

2.2.5. Monitoring

2.2.6. Help!

2.3. Quickstart Installation Guides

2.4. Icinga Quickstart

2.4.1. Introduction

2.4.2. Prerequisites

2.4.3. Install packages

2.4.4. Create Account Information

2.4.5. Download Icinga and the Plugins

2.4.6. Compile and install Icinga

2.4.7. Customise Configuration

2.4.8. Configure the Classic Web Interface

2.4.9. Compile and Install the Monitoring Plugins

2.4.10. Adjusting the SELinux settings

2.4.11. Start Icinga

2.4.12. Login to the Classic Web Interface

2.4.13. Other Modifications

2.4.14. You're Done

2.5. Icinga Quickstart FreeBSD

2.6. Icinga with IDOUtils Quickstart

2.6.1. Introduction

2.6.2. Prerequisites

2.6.3. New features for the IDOUtils:

2.6.4. Install Packages

2.6.5. Create Account Information

2.6.6. Download Icinga and the Plugins

2.6.7. Compile and Install Icinga with IDOUtils

2.6.8. Compile and Install

2.6.9. Customise Configuration

2.6.10. Enable the idomod event broker module

2.6.11. Creation of Database and IDOUtils

2.6.12. Configure the Classic Web Interface

2.6.13. Compile and Install the Monitoring Plugins

2.6.14. Adjusting the SELinux settings

2.6.15. Start IDOUtils and Icinga

2.6.16. Configure Icinga Startup

2.6.17. Login to the Classic Web Interface

2.6.18. Other Modifications

2.6.19. You're Done

2.7. Icinga and IDOUtils Quickstart on FreeBSD

2.8. Icinga Packages for Linux Distributions

2.9. Links to other published Howtos

2.10. Upgrading (to) Icinga

2.10.1. Upgrading from previous Icinga Releases

2.10.2. Upgrading from Nagios 3.x Releases

2.10.3. Upgrading from Nagios 2.x Releases

2.10.4. Upgrading from an RPM Installation

2.11. Upgrading IDOUtils Database

2.11.1. Upgrading IDOUtils to 1.13

2.12. Monitoring Overview

2.13. Monitoring Windows Machines

2.13.1. Introduction

2.13.2. Overview

2.13.3. Steps

2.13.4. What's Already Done For You

2.13.5. Prerequisites

2.13.6. Installing the Windows Agent

2.13.7. Configuring Icinga

2.13.8. Password Protection

2.13.9. Restarting Icinga

2.13.10. Troubleshooting

2.14. Monitoring Linux/Unix Machines

2.14.1. Introduction

2.14.2. Overview

2.15. Monitoring Netware Servers

2.16. Monitoring Network Printers

2.16.1. Introduction

2.16.2. Overview

2.16.3. Steps

2.16.4. What's Already Done For You

2.16.5. Prerequisites

2.16.6. Configuring Icinga

2.16.7. Restarting Icinga

2.17. Monitoring Routers and Switches

2.17.1. Introduction

2.17.2. Overview

2.17.3. Steps

2.17.4. What's Already Done For You

2.17.5. Prerequisites

2.17.6. Configuring Icinga

2.17.7. Restarting Icinga

2.18. Monitoring Publicly Available Services

2.18.1. Introduction

2.18.2. Plugins for Monitoring Services

2.18.3. Creating a Host Definition

2.18.4. Creating Service Definitions

2.18.5. Restarting Icinga

3. Configuring Icinga

3.1. Configuration Overview

3.1.1. Introduction

3.1.2. File names

3.1.3. Main Configuration File

3.1.4. Resource File(s)

3.1.5. Object Definition Files

3.1.6. CGI Configuration File

3.2. Main Configuration File Options

3.3. Object Configuration Overview

3.4. Object Definitions

3.4.1. Host definition

3.4.2. Hostgroup Definition

3.4.3. Service Definition

3.4.4. Servicegroup Definition

3.4.5. Contact Definition

3.4.6. Contactgroup Definition

3.4.7. Timeperiod Definition

3.4.8. Command Definition

3.4.9. Servicedependency Definition

3.4.10. Serviceescalation Definition

3.4.11. Hostdependency Definition

3.4.12. Hostescalation Definition

3.4.13. Hostextinfo Definition

3.4.14. Serviceextinfo Definition

3.4.15. Module Definition

3.5. Custom Object Variables

3.5.1. Introduction

3.5.2. Custom Variable Basics

3.5.3. Examples

3.5.4. Custom Variables As Macros

3.5.5. Custom Variables And Inheritance

3.6. CGI Configuration File Options

4. Running Icinga

4.1. Verifying Your Configuration

4.2. Starting and Stopping Icinga

5. The Basics

5.1. Icinga Plugins

5.1.1. Introduction

5.1.2. What Are Plugins?

5.1.3. Plugins As An Abstraction Layer

5.1.4. What Plugins Are Available?

5.1.5. Obtaining Plugins

5.1.6. Switch to the Icinga user

5.1.7. Extending the environment

5.1.8. How Do I Use Plugin X?

5.1.9. Integrating a new plugin

5.1.10. Raw command line

5.1.11. Threshold and ranges

5.1.12. Activating the definition

5.1.13. Plugin API

5.2. Understanding Macros and How They Work

5.2.1. Macros

5.2.2. Macro Substitution - How Macros Work

5.2.3. Example 1: Host Address Macro

5.2.4. Example 2: Command Argument Macros

5.2.5. On-Demand Macros

5.2.6. On-Demand Group Macros

5.2.7. Custom Variable Macros

5.2.8. Macro Cleaning

5.2.9. Macros as Environment Variables

5.2.10. Available Macros

5.3. Standard Macros in Icinga

5.3.1. Macro Validity

5.3.2. Macro Availability Chart

5.4. Host Checks

5.4.1. Introduction

5.4.2. When Are Host Checks Performed?

5.4.3. Cached Host Checks

5.4.4. Dependencies and Checks

5.4.5. Parallelization of Host Checks

5.4.6. Host States

5.4.7. Host State Determination

5.4.8. Host State Changes

5.5. Service Checks

5.5.1. Introduction

5.5.2. When Are Service Checks Performed?

5.5.3. Cached Service Checks

5.5.4. Dependencies and Checks

5.5.5. Parallelization of Service Checks

5.5.6. Service States

5.5.7. Service State Determination

5.5.8. Service State Changes

5.5.9. Service State is "stale"

5.6. Active Checks

5.6.1. Introduction

5.6.2. How Are Active Checks Performed?

5.6.3. When Are Active Checks Executed?

5.7. Passive Checks

5.7.1. Introduction

5.7.2. Uses For Passive Checks

5.7.3. How Passive Checks Work

5.7.4. Enabling Passive Checks

5.7.5. Submitting Passive Service Check Results

5.7.6. Submitting Passive Host Check Results

5.7.7. Passive Checks and Host States

5.7.8. Submitting Passive Check Results From Remote Hosts

5.8. State Types

5.8.1. Introduction

5.8.2. Service and Host Check Retries

5.8.3. Soft States

5.8.4. Hard States

5.8.5. Example

5.9. Time Periods

5.9.1. Introduction

5.9.2. Precedence in Time Periods

5.9.3. How Time Periods Work With Host and Service Checks

5.9.4. How Time Periods Work With Contact Notifications

5.9.5. How Time Periods Work With Notification Escalations

5.9.6. How Time Periods Work With Dependencies

5.10. Determining Status and Reachability of Network Hosts

5.10.1. Introduction

5.10.2. Example Network

5.10.3. Defining Parent/Child Relationships

5.10.4. Reachability Logic in Action

5.10.5. UNREACHABLE States and Notifications

5.11. Notifications

5.11.1. Introduction

5.11.2. When Do Notifications Occur?

5.11.3. Who Gets Notified?

5.11.4. What Filters Must Be Passed In Order For Notifications To Be Sent?

5.11.5. Program-Wide Filter:

5.11.6. Service and Host Filters:

5.11.7. Contact Filters:

5.11.8. Notification Methods

5.11.9. Notification Type Macro

5.11.10. Helpful Resources

5.12. Notifications: Examples and troubleshooting

5.12.1. Introduction

5.12.2. Some examples

5.12.3. Troubleshooting

6. User Interfaces

6.1. Icinga Classic UI: Information On The Modules

6.2. Authentication And Authorization In The Classic UI

6.2.1. Introduction

6.2.2. Definitions

6.2.3. Setting Up Authenticated Users

6.2.4. Enabling Authentication/Authorization Functionality In The Classic UI

6.2.5. Default Permissions To Classic UI Information

6.2.6. Granting Additional Permissions To Classic UI Information

6.2.7. Classic UI Authorization Requirements

6.2.8. Authentication On Secured Web Servers

6.3. Information On Classic UI parameters

6.3.1. Introduction

6.3.2. Executing Classic UI modules on the command line

6.3.3. Properties / Statustypes

6.3.4. Matrix Parameters / Classic UI modules

6.3.5. Parameter Details

6.3.6. Abbreviations, Classic UI modules, Links

6.3.7. Host and Service Filter Properties

6.3.8. Host and Service Status Types

6.3.9. Commands

6.4. Executing Classic UI modules (CGIs) on the command line

6.4.1. Introduction

6.4.2. Prerequisites

6.4.3. Examples

6.5. Installation of the Icinga Web Frontend

6.5.1. Introduction

6.5.2. Prerequisites

6.5.3. The installation

6.5.4. Settings

6.5.5. Use It!

6.5.6. Test & Errors?

6.6. Upgrading Icinga Web and Icinga Web Database

6.6.1. Upgrading Icinga Web

6.6.2. The Upgrade

6.6.3. Upgrading the Icinga Web Database

6.7. Configuration Overview of Icinga Web

6.7.1. Where are my config files?

6.7.2. Index

6.7.3. Global Configuration Section

6.7.4. Troubleshooting

6.7.5. Module Configuration

6.7.6. Customised Configuration

6.8. Introduction to Icinga Web

6.8.1. Introduction to Icinga Web (>= 1.14)

6.9. Integration of PNP4Nagios into Icinga Web

6.9.1. Install PNP4Nagios

6.9.2. Templates Extension

6.9.3. Clearing the cache

7. Advanced Topics

7.1. External Commands

7.1.1. Introduction

7.1.2. Enabling External Commands

7.1.3. When Does Icinga Check For External Commands?

7.1.4. Using External Commands

7.1.5. Command Format

7.1.6. External Commands List

7.2. List of External Commands

7.3. Event Handlers

7.3.1. Introduction

7.3.2. When Are Event Handlers Executed?

7.3.3. Event Handler Types

7.3.4. Enabling Event Handlers

7.3.5. Event Handler Execution Order

7.3.6. Writing Event Handler Commands

7.3.7. Permissions For Event Handler Commands

7.3.8. Service Event Handler Example

7.4. Volatile Services

7.4.1. Introduction

7.4.2. What Are They Useful For?

7.4.3. What's So Special About Volatile Services?

7.4.4. The Power Of Two

7.4.5. Icinga Configuration:

7.4.6. PortSentry Configuration:

7.4.7. Port Scan Script:

7.5. Service and Host Freshness Checks

7.5.1. Introduction

7.5.2. How Does Freshness Checking Work?

7.5.3. Enabling Freshness Checking

7.5.4. Example

7.5.5. "Check results ... are stale. Forcing an immediate check ...

7.6. Distributed Monitoring

7.6.1. Introduction

7.6.2. Goals

7.6.3. Reference Diagram

7.6.4. Central Server vs. Distributed Servers

7.6.5. Obtaining Service Check Information From Distributed Monitors

7.6.6. Distributed Server Configuration

7.6.7. ocsp_command=submit_check_result

7.6.8. Central Server Configuration

7.6.9. Problems With Passive Checks

7.6.10. Freshness Checking

7.6.11. Performing Host Checks

7.7. Redundant and Failover Network Monitoring

7.7.1. Introduction

7.7.2. Prerequisites

7.7.3. Sample Scripts

7.7.4. Scenario 1 - Redundant Monitoring

7.7.5. Scenario 2 - Failover Monitoring

7.8. Detection and Handling of State Flapping

7.8.1. Introduction

7.8.2. How Flap Detection Works

7.8.3. Example

7.8.4. Flap Detection for Services

7.8.5. Flap Detection for Hosts

7.8.6. Flap Detection Thresholds

7.8.7. States Used For Flap Detection

7.8.8. Flap Handling

7.8.9. Enabling Flap Detection

7.9. Notification Escalations

7.9.1. Introduction

7.9.2. When Are Notifications Escalated?

7.9.3. Contact Groups

7.9.4. Overlapping Escalation Ranges

7.9.5. Recovery Notifications

7.9.6. Notification Intervals

7.9.7. Time Period Restrictions

7.9.8. State Restrictions

7.10. Escalation Condition

7.10.1. Introduction

7.10.2. Syntax

7.10.3. Example

7.11. On-Call Rotations

7.11.1. Introduction

7.11.2. Scenario 1: Holidays and Weekends

7.11.3. Scenario 2: Alternating Days

7.11.4. Scenario 3: Alternating Weeks

7.11.5. Scenario 4: Vacation Days

7.11.6. Other Scenarios

7.12. Monitoring Service and Host Clusters

7.12.1. Introduction

7.12.2. Plan of Attack

7.12.3. Using the check_cluster Plugin

7.12.4. Monitoring Service Clusters

7.12.5. Monitoring Host Clusters

7.13. Host and Service Dependencies

7.13.1. Introduction

7.13.2. Service Dependencies Overview

7.13.3. Defining Service Dependencies

7.13.4. Example Service Dependencies

7.13.5. How Service Dependencies Are Tested

7.13.6. Execution Dependencies

7.13.7. Notification Dependencies

7.13.8. Dependency Inheritance

7.13.9. Host Dependencies

7.13.10. Example Host Dependencies

7.14. State Stalking

7.14.1. Introduction

7.14.2. How Does It Work?

7.14.3. Should I Enable Stalking?

7.14.4. How Do I Enable Stalking?

7.14.5. How Does Stalking Differ From Volatile Services?

7.14.6. Caveats

7.15. Performance Data

7.15.1. Introduction

7.15.2. Types of Performance Data

7.15.3. Plugin Performance Data

7.15.4. Processing Performance Data

7.15.5. Processing Performance Data Using Commands

7.15.6. Writing Performance Data To Files

7.15.7. Performance Data Format

7.16. Scheduled Downtime

7.16.1. Introduction

7.16.2. Scheduling Downtime

7.16.3. Fixed vs. Flexible Downtime

7.16.4. Triggered Downtime

7.16.5. How Scheduled Downtime Affects Notifications

7.16.6. Overlapping Scheduled Downtime

7.17. Recurring Downtimes

7.17.1. Downtime definition

7.17.2. Variable dates

7.17.3. Local holidays

7.17.4. Entering downtimes

7.17.5. Planning downtimes

7.17.6. Converting downtime definitions

7.17.7. Files

7.18. Using The Embedded Perl Interpreter

7.18.1. Advantages

7.18.2. Disadvantages

7.18.3. Using The Embedded Perl Interpreter

7.18.4. Compiling Icinga With Embedded Perl

7.18.5. Plugin-Specific Use of the Perl Interpreter

7.18.6. Developing Plugins For Use With Embedded Perl

7.19. Adaptive Monitoring

7.19.1. Introduction

7.19.2. What Can Be Changed?

7.19.3. External Commands For Adaptive Monitoring

7.20. Predictive Dependency Checks

7.20.1. Introduction

7.20.2. How Do Predictive Checks Work?

7.20.3. Enabling Predictive Checks

7.20.4. Cached Checks

7.21. Cached Checks

7.21.1. Introduction

7.21.2. For On-Demand Checks Only

7.21.3. How Caching Works

7.21.4. What This Really Means

7.21.5. Configuration Variables

7.21.6. Optimizing Cache Effectiveness

7.22. Passive Host State Translation

7.22.1. Introduction

7.22.2. Different World Views

7.22.3. Enabling State Translation

7.23. Service and Host Check Scheduling

7.23.1. Introduction

7.23.2. Configuration Options

7.23.3. Initial Scheduling

7.23.4. Inter-Check Delay

7.23.5. Service Interleaving

7.23.6. Maximum Concurrent Service Checks

7.23.7. Time Restraints

7.23.8. Normal Scheduling

7.23.9. Scheduling During Problems

7.23.10. Host Checks

7.23.11. Scheduling Delays

7.23.12. Scheduling Example

7.23.13. Service Definition Options That Affect Scheduling

7.23.14. TODO

7.24. Custom CGI Headers and Footers (Classic UI)

7.24.1. Introduction

7.24.2. How Does It Work?

7.25. Modified attributes

7.25.1. Introduction

7.25.2. Example

7.26. Object Inheritance

7.26.1. Introduction

7.26.2. Basics

7.26.3. Local Variables vs. Inherited Variables

7.26.4. Inheritance Chaining

7.26.5. Using Incomplete Object Definitions as Templates

7.26.6. Custom Object Variables

7.26.7. Cancelling Inheritance of String Values

7.26.8. Additive Inheritance of String Values

7.26.9. Implied Inheritance

7.26.10. Implied/Additive Inheritance in Escalations

7.26.11. Important values

7.26.12. Multiple Inheritance Sources

7.26.13. Precedence With Multiple Inheritance Sources

7.27. Time-Saving Tricks For Object Definitions

8. Security and Performance Tuning

8.1. Security Considerations

8.1.1. Introduction

8.1.2. Best Practices

8.2. Enhanced Classic UI Security and Authentication

8.2.1. Introduction

8.2.2. Additional Techniques

8.2.3. Implementing Digest Authentication

8.2.4. Implementing Forced TLS/SSL

8.2.5. Implementing IP subnet lockdown

8.2.6. Important Notes

8.3. Tuning Icinga For Maximum Performance

8.3.1. Introduction

8.3.2. Optimization Tips

8.4. Fast Startup Options

8.4.1. Introduction

8.4.2. Background

8.4.3. Evaluating Startup Times

8.4.4. Pre-Caching Object Configuration

8.4.5. Skipping Circular Path Tests

8.4.6. Putting It All Together

8.5. Large Installation Tweaks

8.5.1. Introduction

8.5.2. Effects

8.6. Using The Icingastats Utility

8.6.1. Introduction

8.6.2. Usage Information

8.6.3. Human-Readable Output

8.6.4. PNP4Nagios Integration

8.7. Graphing Performance Info With PNP4Nagios

8.7.1. Introduction

8.7.2. Prerequisites

8.7.3. Average Host / Service Check Latency

8.7.4. Service Statistics

8.7.5. Host Statistics

8.7.6. Average Execution Times

8.7.7. External Commands

8.7.8. External Command Buffers

8.7.9. Cached Host and Service Checks

8.7.10. Average State Changes

8.8. Temporary Data

9. Integration With Other Software

9.1. Integration Overview

9.1.1. Introduction

9.1.2. Integration Points

9.1.3. Integration Examples

9.2. SNMP Trap Integration

9.3. TCP Wrapper Integration

9.3.1. Introduction

9.3.2. Defining A Service

9.3.3. Configuring TCP Wrappers

9.3.4. Writing The Script

9.3.5. Finishing Up

9.4. MKLiveStatus Integration

9.4.1. Introduction

9.4.2. Installation / Configuration

9.5. Installation of the Icinga Reporting with JasperReports Server

9.5.1. Introduction

9.5.2. Official Docs

9.5.3. Requirements

9.5.4. Icinga Reporting

9.5.5. Configuration

9.5.6. Reports

9.5.7. Errors

10. Additional software

10.1. Icinga Addons

10.1.1. Introduction

10.2. NRPE

10.2.1. Introduction

10.2.2. Prerequisites

10.2.3. Download

10.2.4. Optional changes

10.2.5. Compile on the Icinga server

10.2.6. First test

10.2.7. Remote system(s)

10.2.8. Prerequisites on the remote host(s)

10.2.9. Second test

10.2.10. Installation on the remote host

10.2.11. Third test

10.2.12. Troubleshooting

10.2.13. Security

10.2.14. Definition of local checks

10.2.15. Definitions on the Icinga server

10.2.16. More Troubleshooting

10.2.17. Upgrading

10.3. NSCA

10.3.1. Introduction

10.3.2. Prerequisites

10.3.3. Download and compile

10.3.4. Customise

10.3.5. First test

10.3.6. Install

10.3.7. Remote system(s)

10.3.8. Copy files

10.3.9. Second test

10.3.10. Troubleshooting

10.3.11. Security

10.3.12. Operation

10.3.13. Integration into Icinga

11. Development

11.1. Icinga Plugin API

11.1.1. Other Resources

11.1.2. Plugin Overview

11.1.3. Return Code

11.1.4. Plugin Output Spec

11.1.5. Plugin Output Examples

11.1.6. Plugin Output Length Restrictions

11.1.7. Examples

11.1.8. Perl Plugins

11.2. Developing Plugins For Use With Embedded Perl

11.2.1. Introduction

11.2.2. Target Audience

11.2.3. Things you should do when developing a Perl Plugin (ePN or not)

11.2.4. Things you must do to develop a Perl plugin for ePN

11.3. No Icinga API to install anymore

11.3.1. Installation and use of the Icinga API

11.4. The Icinga Web REST API

11.4.1. Why should I use the API?

11.4.2. Features of the Icinga Web REST API

11.4.3. What's the difference between the Icinga API and the Icinga Web REST API?

11.4.4. Prerequisites

11.4.5. Reference

11.4.6. GET

11.4.7. The structure of the URL:

11.4.8. The Parameters en detail:

11.4.9. Example for GET

11.4.10. POST

11.4.11. Parameters en detail

11.4.12. Example for POST

12. IDOUtils

12.1. Introduction

12.1.1. Purpose

12.1.2. Design Overview

12.1.3. Instances

12.1.4. Installation

12.2. Components

12.2.1. Overview

12.2.2. IDOMOD

12.2.3. LOG2IDO

12.2.4. FILE2SOCK

12.2.5. IDO2DB

12.3. Configuration of IDOUtils

12.3.1. IDOMOD config options

12.3.2. IDO2DB config options

12.4. Example Configurations

12.4.1. Single Server, Single Instance Setup

12.4.2. Single Server, Multiple Instance Setup

12.4.3. Single Server, Single Instance Log File Import

12.5. IDOUtils Database Model

12.5.1. Central Tables

12.5.2. Debugging Tables

12.5.3. Historical Tables

12.5.4. Current Status Tables

12.5.5. Configuration Tables

12.6. Database changes/alterations

13. Icinga Samples

13.1. Sample configuration files and definitions

13.2. icinga.cfg

13.3. cgi.cfg

13.4. resource.cfg

13.5. httpd.conf

13.6. commands.cfg

13.7. commands.cfg

13.8. localhost.cfg

13.9. notifications.cfg

13.10. printer.cfg

13.11. switch.cfg

13.12. templates.cfg

13.13. timeperiods.cfg

13.14. windows.cfg

Index