25. 09. 2023
Icinga Web 2, NetEye, Unified Monitoring
05. 05. 2020
Patrick Zambelli
NetEye, Unified Monitoring
Import Data Correlation and Automation with Icinga2 Director
A common scenario during the setup of a monitoring environment represents the import of Host and Service definitions from external databases or existing monitoring systems. This task can become quite complex when there are multiple external sources, and the data needs to be correlated.
This was the challenge we faced during a recent project where we needed to integrate important attributes for host objects from a third party source.
Let me introduce you to the scenario:
- Monitoring with Icinga2 / NetEye 4 during project deployment at about 50% coverage.
- Need to enrich host objects with the required attributes for organizing notification management. Data to write:
- Team(s) responsible for host object: Name of the team(s) as a string
- Operating system: OS description as string
- Whether a host requires 24×7 support
Sample of the data provided in CSV format:
| Hostname | FQDN | Responsibility1 | Responsibility2 | Responsibility3 | OperatingSystem | provide 24×7 |
| Server1 | Server1.mydoman.lan | Server Admins | Windows Server 2012 R2 | NO | ||
| Appliance1 | Appliance1. mydoman.lan | Network Admins | Gerneral Admins | CentosOS 7 | YES |
The Problem:
We had already created hosts in monitoring via Director Self-service API, and future hosts should be auto-registered during agent setup. The provided list of hosts covers hosts from the entire infrastructure and therefore exceeds the needs for the current import.
Step 1: Clean hosts from the list not yet in monitoring
Another reason for avoiding the import of new hosts from the list is that the creation of host objects without an IP address, template, etc. is not valid. Director “import automation” allows choosing only among “update modes”, but no option exists to avoid the creation of new hosts. Therefore, I need to arrange this myself: The Director DB is a good starting point!
First, in MySQL I create a new empty database and a suitable table for my CSV file:
> create table hosts (
Hostname varchar(255),
Responsibility1 varchar(255),
Responsibility2 varchar(255),
Responsibility3 varchar(255),
OperatingSystem varchar(255),
`provide 24×7` varchar(255)
);
In the next step, I load the CSV into this table:
> LOAD DATA LOCAL INFILE ‘/neteye/shared/httpd/file-import/01_host_notification_data.csv’
INTO TABLE `hosts`
FIELDS TRMINATED BY ‘;’
ENCLOSED BY ‘”‘
LINES TERMINATED BY ‘\n’
IGNORE 1 ROWS;
Now I fetch all hosts from CSV-List where “Hostname” exists in Director:
> SELECT h.*
FROM hosts_fqdn h
WHERE h.fqdn in (select object_name from director.icinga_host where object_type = “object”);
And I write the results to a file:
> SELECT h.*
FROM hosts_fqdn h
WHERE h.fqdn IN (select object_name from director.icinga_host where object_type = “object”)
INTO OUTFILE ‘/neteye/shared/ httpd/file-import/01_hosts2update_notification_data.csv’ FIELDS TERMINATED BY ‘;’ ENCLOSED BY ‘”‘ LINES TERMINATED BY ‘\n’;
Having done this, I obtain a list of only those hosts already registered in Director. Now let’s go on to…
Step 2: Configure the Icinga2 Director Import automation import source and sync rules
Definition of “Import Source”:
The import source is required to organize:
- Translation of Hostname/FQDN to lower case
- Sanitizing of values for Responsibility and whether 24×7 is required (i.e. set “n/a” or “none”to NULL)
- Normalization of OperatingSystem
- Organize Responsibility roles as an Array
Example of a regular expression base replacement
Here we need to normalize OperatingSystem from “Windows Server 2008 R2 SP1” or “Windows Server 2008 R2 SP2” to “Windows Server 2008 R2”:
Example of Regular expression base replacement to generate a value interpreted by Director as Boolean
Now I need to arrange the associated teams with its 3 columns into an array. Director’s Import source provides modifiers to combine multiple values into a list by referring to the columns of the CSV data by ${col_name}. Think about introducing a separator!
Having a separator and a next modifier “split” translates the list into an array – magic!
Here comes a preview of the import source:
Step 3: Mapping of source values into Director fields
Now we’re at the easy step where we create a suitable mapping of the provided values into custom fields in Icinga2 Director.
The properties mapping of the Sync rule is organized in this way:
- The host OS is mapped into a field of type string, with values organized within a datalist to provide a user-friendly drop-down box
- The provide 24×7 value is organized as a Boolean
- The array Responsibility fits into a field of type Array
Before running the final import of my approximately 500 hosts, and running the risk of overriding valuable data already defined by colleagues, it is a good moment to check the changes to be made in the “Preview” tab!
HINT: The list is limited to show only 50 elements in preview. What is good for performance is not so great for the user, however, because I have no option besides “sync only the first 50 elements of list”. This trick is not nice but it helps.
Change file providing “Sync” preview:
File: /usr/share/icingaweb2/modules/director/application/controllers/SyncruleController.php
diff -ruN SyncruleController.php.orig SyncruleController.php
— SyncruleController.php.orig 2020-04-08 17:36:51.034028512 +0200
+++ SyncruleController.php 2020-04-08 17:37:03.750027732 +0200
@@ -298,7 +298,7 @@
* @return string
* @throws \Icinga\Exception\NotFoundError
*/
– protected function firstNames($objects, $max = 50)
+ protected function firstNames($objects, $max = 700)
{
$names = [];
$list = new UnorderedList();
Enjoy after importing the compiled field on the host:
Patrick Zambelli
Project Manager at Würth Phoenix
After my graduation in Applied Computer Science at the Free University of Bolzano I decided to start my professional career outside the province. With a bit of good timing and good luck I went into the booming IT-Dept. of Geox in the shoe district of Montebelluna, where I realized how a big IT infrastructure has to grow and adapt to quickly changing requirements. During this experience I had also the nice possibility to travel the world, while setting up the various production and retail areas of this company. Arrived at Würth Phoenix I started developing on our monitoring solution NetEye. Today, in my position as Consulting an Project Manager I am continuously heading to implement our solutions to meet the expectation of your enterprise customers.
Author
Latest posts by Patrick Zambelli
16. 01. 2024
NetEye, Unified Monitoring
Icinga 2 DSL for Defining the Monitoring Status of Objects with Director
23. 12. 2020
NetEye
Tornado – Getting in Action with Sample Rules
20. 12. 2019
Icinga News, NetEye
One Year of Community for NetEye 4 Projects