How to Configure Slack Notification on NetEye
Notifications for service and host problems are an integral part of your monitoring setup, representing the responsive components of your monitoring system. You can configure actions based on changes in the status of monitored objects.
While monitoring systems are incredibly useful for active interpretation and investigation, one of the primary benefits of a complete monitoring system is letting administrators disengage from the system. Alerts allow you to define situations where it makes sense to manage actively, while relying on the passive monitoring of the software to watch for changing conditions.
There are many ways of sending notifications, for instance by email, XMPP, IRC, Twitter, etc. Out of the box, NetEye (i.e., Icinga 2) does not know how to send notifications. Here I will show you how to correctly configure Slack notifications on NetEye.
The first step is: create a Slack app.
- Create an app at api.slack.com/apps
- Activate Incoming Webhooks from Features > Incoming Webhooks, then click “Add New Webhook to Workspace”. Install the app and select a channel.
- Navigate to the OAuth & Permissions page and add the following bot scopes:
incoming-webhook(This should be pre-selected)chat:write(to send messages)user:read(to read user messages on the channel)
Now you will be able to send a message in JSON format using the Slack app, by directly sending a curl command to the specified endpoint. You can easily find the reference to the endpoint in the app’s Incoming Webhook section.
Once the app configuration is complete and works correctly, you can begin with the configuration of the NetEye notifications.
Let’s start by defining the Notification command. NotificationCommand objects define how notifications are delivered to external interfaces. A fresh NetEye install comes with two example scripts for host and service notifications by email.
We need to create a NotificationCommand that will send the curl request to our previously defined endpoint, and that includes a JSON object containing the object status information.
Let’s create a Command template in NetEye’s Director module:
template NotificationCommand "slack-notification-command" {
import "plugin-notification-command"
command = [ "/usr/bin/curl" ]
arguments += {
"(no key)" = {
skip_key = true
value = "<WebHook endpoint>"
}
"-H" = "Content-type: application/json"
"-X" = "POST"
}
}
Once the command template is ready, we have to use it in the host and service NotificationCommands respectively.
Here are two examples, one for Host and one for Service:
object NotificationCommand "slack-host-notification" {
import "plugin-notification-command"
import "slack-notification-command"
arguments += {
"--data" = "{\"text\":\"[$notification.type$] Host $host.display_name$ is $host.state$!\"}"
}
}
object NotificationCommand "slack-service-notification" {
import "plugin-notification-command"
import "slack-notification-command"
arguments += {
"--data" = "{\"text\":\"[$notification.type$] Service $service.display_name$ is $service.state$!\"}"
}
}
}
A notification specification requires one or more users (and/or user groups) who will be notified in case of problems. These users must have all those custom attributes defined which will take part in the NotificationCommand when executed.
Let’s consider the following example:
object User "r&d-team" {
display_name = "R&D Team"
email = "email@wuerth-phoenix.com"
enable_notifications = true
period = "24x7"
states = [ Critical, Down, OK, Unknown, Up, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
}
The R&D Team in the example will get notified on Critical, Down, OK, Unknown, Up and Warning problems. In addition to that, Recovery notifications are sent (they require the OK state).
Once NotificationCommand(s) and users are configured, we can set up the Notification itself.
In the NetEye Director module under Notification, let’s create a template for both Hosts and for Services:
template Notification "Slack Host Down" {
command = "slack-host-notification"
interval = 0s
period = "24x7"
states = [ Down, Up ]
types = [ DowntimeEnd, DowntimeStart, Problem, Recovery ]
}
template Notification "Slack Service Down" {
command = "slack-service-notification"
interval = 0s
period = "24x7"
states = [ Critical, OK, Warning ]
types = [ DowntimeEnd, DowntimeStart, Problem, Recovery ]
users = [ "r&d-team" ]
}
By setting interval=0s we prevent re-notification for the same host/service status change.
After the Notification template has been created you have to create the Notification itself, where you specify the users who will be notified, the template to use, and the condition that will trigger the sending of a Notification.
apply Notification "slack-host" to Host {
import "Slack Host Down"
assign where host.name //this means for all hosts
users = [ "r&d-team" ]
}
apply Notification "slack-service" to Service {
import "Slack Service Down"
assign where service.name // for all services
users = [ "r&d-team" ]
}
This guide will help you configure very simple notifications for both hosts and services and, for all state changes. You can now customize the command and the notification definition at will to better satisfy your needs.
Valentina Da Rold
Author
