How to Use Tornado for event based monitoring with Icinga 2 (Part 1/2)
Tornado is the spiritual successor of the NetEye EventHandler. Tornado is an open-source, rule based Event Processing engine designed to handle up to millions of events per second. We can leverage this capability to ingest all possibly interesting events of our entire infrastructure, and react to the effectively interesting ones.
In this blog post we will explore the possibility to use tornado to build a good starting point for event based monitoring. In this first part we will see how to teach Tornado to react to incoming events. The second part will explore how to test this configuration from both GUI and CLI, and put tornado under a little stress test.
For the sake of simplicity let’s leave out the first stage “collectors” and assume that your devices send their events directly to tornado. In the real world you will have your tornado collectors taking care of this step, and translate e.g. all Webhooks, Snmptraps, Emails or Syslogs sent to NetEye into Tornado Events.
Approach
We will now create a simple RuleSet that will react to events with the following characteristics:
- Event
Typeequalsblog IP Addressis in the range of our subnet10.23.24.0/24- Whenever the payload starts with the word
CRITICALa passive check result for the serviceblogon the host with the sameAddressshould be set - Whenever the payload starts with the word
OKthe same should happen with anOKstate
The cleanest approach is to add a branch to out Processing tree, for only blog_event events. The content of our rules.d folder will look like this:
rules.d/ rules.d/blog rules.d/blog/filter.json rules.d/blog/blog_subnet rules.d/blog/blog_subnet/filter.json rules.d/blog/blog_subnet/blog_set_service rules.d/blog/blog_subnet/blog_set_service/020_service_blog_ok.json rules.d/blog/blog_subnet/blog_set_service/030_service_blog_critical.json
The rules.d/blog/filter.json will contain a simple filter, letting through only those events which have the blog_event type. The same way for rules.d/blog/blog_subnet/filter.json which will let through only events coming from the subnet 10.23.24.0/24 In this way the rest of the branch will not be traversed by all other events of tornado.
rules.d/blog
rules.d/blog/filter.json
rules.d/blog/blog_subnet
rules.d/blog/blog_subnet/filter.json
rules.d/blog/blog_subnet/blog_set_service
rules.d/blog/blog_subnet/blog_set_service/020_service_blog_ok.json
rules.d/blog/blog_subnet/blog_set_service/030_service_blog_critical.json
The rules.d/blog/filter.json will contain a simple filter, letting through only those events which have the blog_event type. The same way for rules.d/blog/blog_subnet/filter.json which will let through only events coming from the subnet 10.23.24.0/24 In this way the rest of the branch will not be traversed by all other events of tornado.
The next stage rules.d/blog/blog_subnet/blog_set_service is a RuleSet containing all Rules to be applied sequentially to events passing through. Using the icinga2 action, it will send a passive-check-result to Icinga2.
If you’re using the Tornado GUI it should look similar to this:
Configuration
For readability reasons I’ll show only a piece of the configuration in the following paragraphs. You can find a complete example, together with a script for replicating the setup on the bottom of this blog post.
Filter
For the first step of filtering, we’ll simply check the type of the event:
"filter": {
"type": "equals",
"first": "${event.type}",
"second": "blog"
}
In the second step we’ll dynamically match the subnet we’re looking for on the address field:
"filter": {
"type": "regex",
"regex": "^10.23.24.",
"target": "${event.payload.address}"
}
Rules
In the RuleSet we have two almost identical rules, differing only in the constraint.WHERE.regex and in the actions.payload.exit_code field. First we match the output, and check for either ^OK or ^CRITICAL
"WHERE": {
"type": "regex",
"target": "${event.payload.output}",
"regex": "^OK\s"
}
When this is matched we want to trigger an icinga2 action, which sets the service state correctly:
"actions": [
{
"id": "icinga2",
"payload": {
"icinga2_action_name": "process-check-result",
"icinga2_action_payload": {
"exit_status": "0",
"plugin_output": "${event.payload.output}",
"filter": "host.address==\"${event.payload.address}\" && service.name==\"blog\"",
"type": "Service"
}
}
}
]
This is a little more verbose, but basically, we are calling the process-check-result endpoint of icinga2, with the given parameters, dynamically specifying the host, on which the blog service shall be affected.
As soon as the events start rolling your event log of the blog services will show activity which should look similar to the following:
If you want to try it out for yourself, you can build Tornado from the Tornado GitHub repository.
Resources
Rules, Setup Script and Stress Test: Download
Benjamin Gröber
Author
