20. 12. 2024
Atlassian, NetEye, Service Management
The Italian Data Protection Authority requires the management and storage of millions of logs, but also outside Italy, a good log management strategy can provide several advantages.
Some years ago, the Italian Data Protection Authority defined that the companies must register and store all data related to the system accesses performed by the administrators. This should facilitate the audit of the administrator’s activities. (Here you can find the official contents) Therefore, every company has to implement a reliable Log Management system, to be able to archive all the necessary logs.
Well, log management isn’t a new issue. Actually, it became the basis for a successful IT management and all further monitoring processes. Logs can be generated for a variety of processes and purposes. The key for a reliable and useful Log Management is the awareness of the activities which have to be actually monitored and, obviously, to use the right tools to analyze these mountains of logs to benefit from valuable information.
Why the management and analysis of logs is such important? What are the main benefits? Actually, there are multiple action areas.
Productive Monitoring: To control the efficiency and effectiveness of your systems in production. It is possible to configure the log management, to monitor and trace events that are out of normal operation.
Troubleshooting: Quick problem analysis is the key for reliable problem solving. Log management is often the most convenient way to identify the origin of disservices. Having all data on hand and using suitable analysis tools allows identifying the root cause of most anomalies. The possibility to compare data and to identify the precise moment in time when the problem occurred is therefore fundamental in terms of troubleshooting.
Debugging: The purpose of log generation by applications, compilers, debuggers etc. is the creation of alerts in case of exceptions or particular events. Thanks to the multitude of different logs, the log management is able to trace events on applications and processes. On application level, this data is used to identify possible bottlenecks within the data flow.
Analysis on business level: Log data is fundamental also in terms of business analysis. Very specific information can be generally retrieved. For example the most visited area of a website can be compared with more specific data regarding a single page or product. Analyzing the load time of a webpage and measuring the time users stay on a single webpage are used to determine the user experience. Specific information regarding your business can be collected through the analysis of obtained log data. Additionally, individual alerts can be configured to control certain accesses and operations.
Data Security: Ensuring data security is one of the most critical tasks of a reliable log management. Certain events, as the tracing of single login and non-authorized login attempts as well as attempts to modify existing data, can be identified and controlled by log management instruments. The possibility to define conditions, which trigger alarms (for example for repeated failed login attempts or the creation of a new account) provides a comprehensive security system.
Obviously, it’s not easy to deal with the obtained mountain of logs, but nevertheless it is important to define a good monitoring strategy. An appropriate monitoring system, allows fulfilling all prior mentioned necessities. Analyzing and visualizing logs in real time will be very easy. So, what does our solution consist of? To make sense of the huge amount of collected data, we have created the new Log Management module for NetEye. To get the most out of the new module, the open source projects Logstash, Elasticsearch and Kibana have been integrated.
Logstash parses logs (deriving from the Safed Agent and sent to rsyslog, see image above) and submits them to Elasticsearch, which saves them in a structured way. Finally, Kibana displays all the collected data through intuitive graphics. Kibana is a highly scalable interface, which allows a selective search and easy information visualization to make sense of the obtained logs.
The collected information are shown in real time, to provide all the required information at a glance. Moreover, individual dashboard can be created to meet specific needs.
Andrea di Lernia
Profit Center Manager at Würth Phoenix
Hi everybody, I’m Andrea and my contribution to this blog is to give hints of the monitoring issue from an IT manager point of view. I was born in Bolzano in 1965 and my professional path started 25 years ago operating on the technical field as programmer, system/database administrator, network engineer, consultancy and so on. I’ve been living in Milan for 10 years working for multinational IT companies and I decided to return to Bolzano after my marriage and the birth of my daughter.
I love sailing and diving in the summer, skiing in the winter and travelling off-road with my Landcruiser anytime
Author
Latest posts by Andrea di Lernia
02. 09. 2015
NetEye
VoIP-Monitoring mit IP SLA
06. 05. 2015
Garante della Privacy, NetEye
Archiviazione dei log e poi?
06. 05. 2015
Log Auditing, NetEye
Was tun mit den ganzen Logs?