20. 08. 2020 Rocco Pezzani NetEye, Unified Monitoring

Understand Which Agents you Need to Upgrade

As you may already know, starting with version 4.12 NetEye 4 upgraded its Icinga 2 engine from version 2.10.5 to version 2.11.3. At every training session, I always emphasize this rule:

Icinga 2 Master Instance version >= Icinga 2 Satellite Instance version >= Icinga 2 Agent Instance version

And thus everyone expects that his or her own monitoring infrastructure will continue to behave correctly even after upgrading. This is almost correct: beginning with version 2.11, Icinga 2 required TLS v2, so some older agents might not work correctly. However, if all of your Icinga 2 Agents are version 2.10.5, the statement above is still correct.

So if you want to take advantage of all the new Icinga2 features, you might want to update your entire monitoring infrastructure. Besides upgrading your satellites, which is a rather “small” activity in comparison, upgrading your agents might involve a lot of time and effort not only for the upgrades themselves, but also for understanding what exactly you need to upgrade and for keeping track of all the upgrades done until a given time.

I’m afraid Icinga 2 (and also NetEye 4) doesn’t provide any built-in support for assessing all running Icinga2 instances (meaning both satellites and agents), but there is something that can help you partially overcome this limitation. I say partially because nothing beats having a good Asset Management system, and if you don’t have one yet you should consider it; but I’m not saying that gathering the minimum required information to assess your situation would be impossible.

Let’s dig into what NetEye 4 and Icinga 2 can do for you.

Preparation: Force NetEye to Gather Agent Data

The first step is to force NetEye 4 to tell us (whenever possible) the version of every connected agent. To do so, you need to monitor the performance of all the Icinga2 instances you have. To do this we can take advantage of the embedded icinga command (https://icinga.com/docs/icinga2/latest/doc/10-icinga-template-library/#icinga). This command is provided directly via the ITL. The documentation has some gaps, but as you can guess this command can retrieve the version of every Icinga Instance you have. You only need to run it on the right instance (i.e., run it on the agent).

You’ll have to wrap this command within a new Service Template and force it to be executed by the Icinga 2 Agent itself. So, inside Director, you have to create a new Service Template with these characteristics:

Its name could be agent-instance-performance, or whatever you feel is right, just ensure that you use a meaningful name
If you already have either the generic-agent-service (preferred) or generic-agent Service Template, your new Service Template can inherit from one of those: it is not required, but highly recommended
Check command must be icinga
Under “Icinga Agent and zone settings”, the “Run on agent” value must be set to Yes

You can see an example in this screenshot:

Now that you have the Service Template, you need to distribute it to all Agent-monitored hosts. Just add a new service to all the Host Templates on which you want to deploy the new configuration. You can name the service something like Agent instance Performance, or alternatively something more meaningful to you.

After you deploy, your new services will start to report for every Icinga2 Agent instance, whether the instance is running, its version and several other statistics.

Processing: Assess the Current Situation

Now that your NetEye 4 is collecting data, you can create a report of your current situation just by looking at the Services list:

On the Overview menu item, you will find the Services submenu item
Click on it, then filter the list by providing the name of the service you created earlier (e.g.: Agent instance Performance)
Export the list in CSV format

If you open the CSV file you just downloaded inExcel, you’ll find the column service_output reporting the full output of the command, and inside of that you’ll find the current Agent version. You can use this column to do any filtering, searching and sorting you want.

Just a note: Older agents might not be able to report their version. If you don’t see any version number, just assume the agent is really old and needs to be upgraded as soon as possible.

Tracking: Keep up with the Upgraded Agents

When you start upgrading your Agents, you might want to be updated about the current status of the whole process. At any point you can simply redo the search described above, export the CSV file and recreate the Excel file anew. But you have another option: although it might be a bit invasive, you can let NetEye 4 keep track of this for you. The icinga command can in fact check the version of the current Icinga 2 instance and check if it is equal to or above a certain value, as described by the official documentation. The invasive part is that the command will fire a Critical alert if the instance’s version is below the value you specified, so you might start to receive extra notifications and some noise in your Problems list.

If you want to do this, you’ll have to create and use the right datafield.