NetEye 4 Log Management: Rsyslog and the Elastic Stack
When you need to manage and collect large amounts of data, there can be a lot of hard tasks to do. So we decided to take some of the best Open Source tools to help us do it in the best possible way. Let me introduce you to Rsyslog and the Elastic Stack implementation for NetEye 4 Log Management.
Rsyslog (Version 8)
As the official website says, Rsyslog is the rocket-fast system for log processing. It manages huge amounts of incoming log data, pre-processes it, and delivers it to local destinations at a rate of over one million messages per second.
The version we decided to adopt is version 8, the latest stable release. The main features for this version are:
Multi-threading
TCP, SSL, TLS, and RELP ready
Whole syslog message filtering
A fully configurable output format
We use it to collect data from multiple agents, and we configured it to write the logs into the file system. The configuration includes the rules to allow the agents to send log messages to the Rsyslog service, along with the directives to write the log files into the file system.
The rules for the agents are written in a fully automated way by the Icinga Web 2 Log Manager module, but they are placed in a conf.d/ directory so that when the Rsyslog service is started, manual configuration files can also be loaded by placing them in that same folder (/neteye/shared/rsyslog/conf/rsyslog.d).
The Elastic Stack (Version 6)
Built on an open source foundation, the Elastic Stack lets you reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time. It is made of three main products: Elasticsearch, Logstash and Kibana.
Logstash
Logstash is a server-side data processing pipeline that ingests data from many different sources and delivers it to a data storage (“stash”).
We use it to read the log files written by Rsyslog in real time and to send the content to Elasticsearch.
Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine. It is the heart of the Elastic Stack and the core of the NetEye 4 log storage system.
Kibana
Kibana is a render engine for Elasticsearch data. It allows you to visualize and understand large amounts of data by aggregating them logically and graphically.
We created filters to tell it what data to organize for rendering graphs and charts.
Full Stack Developer at Wuerth Phoenix. I love questioning myself, find new challenges to learn and new adventures to grow up. PHP lover trying to expand my skills studying new languages and tools to improve my professional life.
Author
Gianluca Piccolo
Full Stack Developer at Wuerth Phoenix. I love questioning myself, find new challenges to learn and new adventures to grow up. PHP lover trying to expand my skills studying new languages and tools to improve my professional life.
In order to be able to carry out detailed network monitoring, an IT administrator naturally wants to know what is happening in his or her network. To obtain this information, the network flows must of course be analyzed. Many network Read More
Icinga Director is a powerful configuration tool included in NetEye that streamlines the process of defining and managing monitoring configurations. However, you may encounter issues with duplicate addresses and spaces in hostnames, which can lead to confusion and monitoring inaccuracies. Read More
In today’s digital landscape, ensuring optimal performance and availability of applications is critical. Monitoring tools like Elastic Synthetics Journey Monitor and Alyvix offer unique capabilities tailored to diverse needs. Understanding where each tool excels can help you choose the best Read More
In this blog, I'll propose and describe a solution for detecting potential infostealers targeting Chromium-based browsers, taking a cue from the research exposed by Google's Chrome Security Team (Detecting browser data theft using Windows Event Logs). Obviously a solution using Read More
In this blog post, we'll explore the power of custom branding in NetEye reports. The reporting module offers the ability to filter specific hosts and services while also enabling the configuration of custom thresholds to visually emphasize unmet SLAs. Users Read More