06. 11. 2024
AI, Log-SIEM, Machine Learning, NetEye
28. 10. 2020
Nicola Degara
Business Service Monitoring, Cloud, Icinga Web 2, NetEye, Service Management, SLM, Unified Monitoring
NetEye Incident Response with Atlassian Opsgenie Integration
As an Atlassian partner we are working in this period on the preparation of an online demo system in parallel with the delivery of our new NetEye demo online service. For the occasion I decided to expand our already existing NetEye integration with the Atlassian ecosystem that you will find described in other previous blogs.
With this article, I would like to introduce the new integration between NetEye and the Opsgenie Cloud Solution.
Atlassian defines Opsgenie as “a modern incident management platform that ensures critical incidents are never missed, and actions are taken by the right people in the shortest possible time. Opsgenie receives alerts from your monitoring systems and custom applications and categorizes each alert based on importance and timing. On-call schedules ensure the right people are notified through multiple communication channels including voice calls, email, SMS, and push messages on mobile devices. If an alert is not acknowledged, Opsgenie automatically escalates it, ensuring the incident gets the needed attention.”
The meaning of this integration
I personally think that Opsgenie is Atlassian’s answer to the Incident Response concept: it will help you to organize teams and resources for major incident resolution and stakeholder engagement!
With this new integration, you can now:
- Send notification alarms from NetEye to the Opsgenie Incident Dashboard
- Ack and un-Ack an Incident from Opsgenie back to NetEye
- Add comments to NetEye directly from the Opsgenie Incident Dashboard
- Synchronize NetEye Business Process with Opsgenie Team Services
- … and of course use all the feature available in Opsgenie to manage the events lifecycle received from NetEye!
One more thing: the integration between NetEye and Opsgenie runs completely over port 443!
A sample Use Case
Thanks to this integration, it is possible to implement a common, extended Incident Response process from NetEye Monitoring Events generation up to eventual incident resolution and stockholder engagement.
Thanks to the functionalities available for Stakeholder engagement, I believe that one concrete value is related to the possibility of sending real time custom messages to end users regarding a degraded service reported by NetEye: here is an example of the first message that I, as a stakeholder in my test, received on my smartwatch:
Pretty cool, right?
Integration results screenshots
Interface installation notes
In order to implement the integration we can easily follow this online Icinga2/Opsgenie how-to guide: https://docs.Opsgenie.com/docs/icinga2-integration, but be careful because some custom configurations are required since you are installing the integration inside a NetEye environment.
Here I would like to list some important differences and sections you need to be aware of:
- API
Depending on the data location you’ve selected during the Opsgenie’s cloud instance creation be sure you are using the right API server address: if you created an instance in Europe, the correct API address is https://api.eu.Opsgenie.com. For a U.S. instance you can use the default API address (https://api.Opsgenie.com). This particular condition cost me a lot of time debugging J
The head of your file /home/Opsgenie/oec/conf/config.json should look like:
{
"apiKey": "[OPSGENIE ICINGA2 INTEGRATION KEY]",
"baseUrl": "https://api.eu.Opsgenie.com",
"logLevel": "DEBUG",
"globalArgs": [],
"globalFlags": {
"graphite_url": "http://localhost:5003",
"api_url": "https://localhost:5665",
"user": "root",
"password": "[Icinga2 API root password]",
"insecure": "false"
- Configuration notes
The file /home/Opsgenie/oec/Opsgenie-icinga2/Opsgenie.conf that contains Notification Commands should be placed in /neteye/shared/icinga2/conf/icinga2/conf.d/Opsgenie.conf
This will let you use the NetEye Director KickStart Wizard tool (https://YOUR-NETEYE-FQDN/neteye/director/dashboard?name=infrastructure#!/neteye/director/kickstart) and automatically import the command into External Commands.
Before you run the Wizard, assign the correct Icinga permissions to this file, and I also suggest that you comment out the user creation node in the Opsgenie.conf file as you may have problems with the import. So comment out the following lines and create the User Opsgenie manually in NetEye.
# object User "Opsgenie" {
# import "generic-user"
# display_name = "Opsgenie Contact"
# }
Now you can run the wizard and deploy your changes!
- NetEye External commands
Once you’ve checked that there are two new External Commands in your NetEye:
you should create two related notifications. Here are the two notifications to be imported with the icingacli using this command (with json below):
icingacli director notification create –key <notification-name>’ –json ‘<json>’
template Notification "Opsgenie-host-notification" {
import "generic notify all host events"
command = "Opsgenie-host-notification"
period = "24x7"
}
template Notification "Opsgenie-service-notification" {
import "generic notify all events service"
command = "Opsgenie-service-notification"
period = "24x7"
}
- NetEye Business Process – OpsGenie Team’s Service synchronization
In order to synchronize the Business Process with NetEye we have check ready script… Here an example of my quick – but working- bash test:
#!/bin/sh
#
# NETEYEDEMO
# DENI 28.10.2020
#
# this script is just a test .. it currently does not handle BP with spaces
#
opsgenieAPIURL="https://api.eu.opsgenie.com/v1/services"
opsgenieapitoken="(OpsGenie-Token)"
TeamId="(Team's id)" # it can be retrieved with API but this is faster :-)
## GET BP
echo " "
echo " Get BP processes from NetEye"
echo " "
hosts=$(icingacli businessprocess process list )
neteyearray=()
for row in $hosts; do
if [[ $row =~ ^\( ]];then
continue
fi
echo "NetEye Business Process: " $row
neteyearray+=("$row")
done
echo " "
echo " Get services from OpsGenie"
echo " "
## GET SERVICES FROM OPSGENIE
opsgeniehosts=$(curl -s -X GET "$opsgenieAPIURL" -H "Authorization: GenieKey $opsgenieapitoken" -H "Content-Type: application/json" | jq '.data[].name' | sed "s/\"//g" )
opsgeniearray=()
for opsgenierow in $opsgeniehosts; do
echo "OpsGenie Service: " $opsgenierow
opsgeniearray+=("$opsgenierow")
done
## GET DIFFERENCES BETWEEN TWO ARRAYS TO AVOID DUPLICATES
HostDiff=()
for i in "${neteyearray[@]}"; do
skip=
for j in "${opsgeniearray[@]}"; do
[[ $i == $j ]] && { skip=1; break; }
done
[[ -n $skip ]] || HostDiff+=("$i")
done
# CREATE ONLY NEW SERVICE IN OPSGENIE
hostlist=""
for i in "${HostDiff[@]}"
do
json='{"teamId": "'$TeamId'","name": "'$i'", "tags": ["neteye","business process"] }'
curl -X POST "$opsgenieAPIURL" -H "Authorization: GenieKey $opsgenieapitoken" -H "Content-Type: application/json" --data "$json" -v
hostlist+=" ( $i ) "
done
if [ -n "$hostlist" ]
then
echo "OK - syncronization succesfully executed: created or updated objects: [$hostlist] "
else
echo "OK - syncronization succesfully executed: no new hosts have been created or updated"
fi
exit 0
# you can schedule this check once a day and let NetEye handle the sync
And this is the result!
- Running oec
As described in the online guide, please check that the oec service is correctly working on your NetEye:- systemctl status oec (if stopped, start it!)
- journalctl -u oec -f
- tail -f /home/Opsgenie/oec/output/output.txt
(during some ack-unack operations from Opsgenie you should see some debug information, as by default in Opsgenie.conf the log level is debug mode.)
Nicola Degara
Technical Consulting and Delivery Team Manager at @ Würth Phoenix
My name is Nicola Degara and I work as Consulting and Delivery Team Leader. With a technical developer background, I am in the IT field since the nineties. After more than 6 years’ experience as director of an international software development company in Shanghai I have been embracing, once back to Europe, the dynamic NetEye experience and philosophy in combination with international Atlassian product projects.
My strong conviction towards the Open Source supports and continues to influence any reasons of my daily choices and future visions
Author
Latest posts by Nicola Degara
21. 08. 2020
NetEye, Service Management, SLM, Unified Monitoring
The NetEye and Jira Cloud Ecosystem Is Even More Integrated
19. 05. 2020
Anomaly Detection, Log Management, Machine Learning, NetEye, Unified Monitoring, Webinar
Neteye SIEM: processes and tools for IT Services Security 🇮🇹
24. 04. 2020
Events, NetEye, Unified Monitoring, Webinar
NetEye 4: novità nel monitoraggio unificato per l’infrastruttura IT 🇮🇹