Some time ago I published an article about how to store the NetEye SMS Protocol log into an ELK environment. Now, after using it some times, I discovered that it was not completely correct as the time/date functions for the Logstash filters are a bit more complicated. In particular, it was that the date was written in the SMS protocol file in this way:
June 29th 2016, 10:30:22 CEST 2016
And we used this Logstash date filter to convert it:
date {
locale = "en"
match = [ "sms_timestamp_text", "EEE MMM dd HH:mm:ss" ]
}
Now it seemed that it would work, but after some time (some days until the start of the next month) we discovered that the date in the first days of the month would look like:
July 1th 2016, 10:30:22 CEST 2016
As we had a textual timezone and date filters do not support this, in the first draft we had this rule to be able to parse the sms_timestamp_text:
match =>[ "message", "%{SMS_TIMESTAMP_SHORT:sms_timestamp_text}
%{WORD:timezone} %{YEAR}:%{INT:sms_phonenumber}:%{GREEDYDATA:sms_text}"
We discovered that our filter would not work with that as we had “dd” for 2 digit day. Now, how would we do this as the date cannot be matched by “d”, neither by “dd”? After studying the filter rules I discovered the solution. It is possible to have “or” rules inside the date and so being able to match more than one date format. So we changed the filter in this way:
date {
locale = "en"
match => [ "sms_timestamp_text", "EEE MMM dd HH:mm:ss Z yyyy", "EEE MMM d HH:mm:ss Z yyyy" ]
}
You see the new Z and yyyy parameters because if we do not match a complete date it will not work correctly. To be able to parse this correctly now I discovered that the pattern match had to change in this way:
match => [ "message", "%{SMS_TIMESTAMP:sms_timestamp_text}:%{INT:sms_phonenumber}:%{GREEDYDATA:sms_text}" ]
As I told earlier, Logstash cannot parse textual time zones, but this is what we have here. What should we do? We know that our date is in Western Europe so we have a solution for this mutate tag in this way:
I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.
Author
Juergen Vigna
I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.
Icinga Director is a powerful configuration tool included in NetEye that streamlines the process of defining and managing monitoring configurations. However, you may encounter issues with duplicate addresses and spaces in hostnames, which can lead to confusion and monitoring inaccuracies. Read More
In today’s digital landscape, ensuring optimal performance and availability of applications is critical. Monitoring tools like Elastic Synthetics Journey Monitor and Alyvix offer unique capabilities tailored to diverse needs. Understanding where each tool excels can help you choose the best Read More
In this blog post, we'll explore the power of custom branding in NetEye reports. The reporting module offers the ability to filter specific hosts and services while also enabling the configuration of custom thresholds to visually emphasize unmet SLAs. Users Read More
Scenario NetEye 4 is a comprehensive monitoring platform which natively supports Business Processes. A Business Process is an abstract view of a customer’s business from the Application point of view. Usually, it’s a collection of Icinga 2 checks aggregated by Read More
On February 3rd and 4th, 2024, we attended FOSDEM, a major event where thousands of free and open-source software developers from around the world gather to exchange ideas and collaborate. This year I dedicated much of the second day to Read More