08. 07. 2024
Unified Monitoring
20. 08. 2024
Franco Federico
APM, Log-SIEM, Unified Monitoring
A Journey through Elastic Integrations
At the beginning of the month we released NetEye version 4.37 that contains Elastic Stack 8.14.3.
Every version update of Elastic has both improvements and additions. To see all available integrations in NetEye, click on the screenshot here:
As you can see the changes range from cloud integration, to ticketing, to not forgetting security, networking, etc.
As I write, there are currently 384 possible integrations for the latest version of NetEye. The integrations that we have used most often and with which we have experience are the following:
Here’s a couple of tips we’ve learned along the way….
Elastic Agent or Beat?
When approaching a new integration, check the documentation provided to see whether it’s supported with Elastic Agents or Beats or both. Please note that with Elastic Agent as we’ve configured it in NetEye, you can update the Elastic Agent directly from the Fleet console.
The Elastic lines say that Elastic Agent is a single binary that provides the same functionality as Beats, but it’s still possible to find gaps in functionality.
To correctly choose which approach to use, follow these guidelines:
- Check whether the integrations you need are supported and marked as GA (General Availability)
- If the integration is available, check the supported outputs
- Review the capability comparison to check for any required features
If you’re satisfied with all three of these steps, choose Elastic Agent. If not, keep the previous Beats while keeping an eye on future updates.
Keeping Integrations Up-to-date
Always remember to update when there are update notices in the integrations we are using and have configured (they are released much more often than the product itself).
How do we check for updates? Go to the Default space of your Elastic Stack and check if there are any installed integrations that need to be updated using this link (don’t forget to insert your address):
https://<NetEye FQDN>/neteye/kibana/app/integrations/installed
Once the page loads you’ll see this screen:
Clicking on “Updates available” you’ll see the integrations to be updated:
In this case click on Custom Logs and then on Settings, and finally click on “Upgrade to latest version”:
After a few seconds, it will be updated to the latest version. Repeat this procedure for all integrations that need to be updated.
Custom Ingest Pipeline for Integrations
Each integration provides an ingest pipeline and its mapping, which are defined as managed. The corresponding ingest pipeline in turn calls up one or more ingest pipelines. This allows you to add your own pipelines to the bottom of the pipeline and/or allows you to modify <pipeline-name>@custom pipelines. Managed ingest pipelines do not need to be modified since they are system pipelines that are overwritten when fixes and/or new features are released. Custom ingest pipelines, on the other hand, can be modified.
Let’s look at an example to understand this better. We’ll add a new integration to a policy for Windows servers that will collect Windows events.
Here I decide that I want to make a custom filter for PowerShell events I collect, so I scroll down and find the pipeline name:
Then I go to edit it by clicking on the button next to it:
Enter the pipeline by scrolling down until you see the call to other pipelines.
As you can see from the description, we can decide to intervene at different levels. To stay more relevant to our purpose, we’ll use the last pipeline callback which refers to the pipeline logs-windows.powershell@custom.
I then create a new pipeline with this name and insert a drop processor to remove all events with a certain event_id (this is a simple example):
As you can see, there’s a button for testing pipelines by inserting a test document.
Once created and operational, they’ll be called from the parent pipeline logs-windows.powershell-1.47.0.
The world of ingest pipelines is fascinating maybe I will do a blog about it later.
With these tips you’ll be able to use as many integrations as you want, so enjoy Elastic Integration! And obviously, if you prefer, you can also contact us and we’ll be very happy to consult with you and help you configure and/or modify them.
Franco Federico
Hi, I’m Franco and I was born in Monza. For 20 years I worked for IBM in various roles. I started as a customer service representative (help desk operator), then I was promoted to Windows expert. In 2004 I changed again and was promoted to consultant, business analyst, then Java developer, and finally technical support and system integrator for Enterprise Content Management (FileNet). Several years ago I became fascinated by the Open Source world, the GNU\Linux operating system, and security in general. So for 4 years during my free time I studied security systems and computer networks in order to extend my knowledge. I came across several open source technologies including the Elastic stack (formerly ELK), and started to explore them and other similar ones like Grafana, Greylog, Snort, Grok, etc. I like to script in Python, too. Then I started to work in Würth Phoenix like consultant. Two years ago I moved with my family in Berlin to work for a startup in fintech(Nuri), but the startup went bankrupt due to insolvency. No problem, Berlin offered many other opportunities and I started working for Helios IT Service as an infrastructure monitoring expert with Icinga and Elastic, but after another year I preferred to return to Italy for various reasons that we can go into in person 🙂 In my free time I continue to dedicate myself to my family(especially my daughter) and I like walking, reading, dancing and making pizza for friends and relatives.
Author
Latest posts by Franco Federico
25. 03. 2024
APM, NetEye, Visual Synthetic Monitoring
Migration from Alyvix Server to Alyvix Service
10. 12. 2021
APM, NetEye, Real User Experience, Visual Synthetic Monitoring
NEP Alyvix – Ready for Use