If you’re familiar with the NetEye SIEM module you probably also know El Proxy, the solution integrated into NetEye to ensure the integrity and inalterability of the logs produced by the SIEM module.
Since its introduction in NetEye, the only way to understand what El Proxy was doing was to inspect its logs, but as we know this is not an ideal solution for getting an overview of the behavior of any piece of software. This means that until now, El Proxy has been like a black box for most users, who may be have been wondering for example:
Is El Proxy signing and processing all logs correctly? Or is it perhaps encountering some error?
What is the workload in El Proxy? Are El Proxy and Elasticsearch keeping up with all the logs produced by the SIEM module?
To answer these types of questions we started introducing observability into El Proxy. In particular, we started with metrics, which will allow users to easily spot anomalies in the infrastructure and analyze the behavior of El Proxy over time.
The technologies involved in the process of exposing and visualizing El Proxy metrics in NetEye are:
OpenTelemetry: used by El Proxy to generate the metrics and expose them via an HTTP endpoint using the Prometheus format
Telegraf: polls the metrics from the HTTP endpoint and writes them to InfluxDB
Grafana: visualizes the metrics via multiple dashboards installed in NetEye
To design the metrics and the visualizations, we divided the metrics into two main topics. The first one is troubleshooting. For which users may ask: Did El Proxy fail to process some logs? If so, for what reason? Did it store logs in DLQ? If so, when?
To answer these questions we created the “Troubleshooting” dashboard, based on metrics constructed from these use cases.
El Proxy Troubleshooting dashboard. In the first 2 panels, we can see that around 15:28 El Proxy had an error contacting Elasticsearch, probably due to an infrastructure incident, which led to 2 logs not being processed and sent back to Logstash. On the bottom half instead, we gain insights into the single requests performed by El Proxy to Elasticsearch. In particular, the 1st panel shows all the failed requests to Elasticsearch, while the 2nd and 3rd ones show the number of “retries”, i.e., the ones that failed and the ones that were successful, respectively.
Another topic of interest is the performance metrics of El Proxy and Elasticsearch. Hence NetEye also provides a dedicated dashboard for this:
El Proxy Performance dashboard. The 2 top-left panels give an overview of how many requests and logs El Proxy receives over time, while the third panel on the left shows how many logs are inside El Proxy at any moment, which can help you understand if El Proxy is managing to process and write logs at a faster rate than the rate of logs received as input. On the top right instead we have information on the timing of the requests to El Proxy and to Elasticsearch over time. This lets you understand for example if Elasticsearch is slowing down when it’s under heavy load, which may be a sign of a lack of resources.
Finally, a third dashboard gives an overview of the number of logs generated by each Tenant present in the infrastructure:
El Proxy Tenant Performance dashboard. In this dashboard, a panel is generated for each tenant present in your environment. Each panel displays the number of logs received from each Tenant, and also gives an insight into each of the Tenant’s blockchains if they has multiple ones.
We hope this first improvement on the observability of El Proxy will enable users to better and more easily get a grasp on the behavior of El Proxy. Any feedback is appreciated, please report it through the Wuerth Phoenix channels!
These Solutions are Engineered by Humans
Are you passionate about performance metrics or other modern IT challenges? Do you have the experience to drive solutions like the one above? Our customers often present us with problems that need customized solutions. In fact, we’re currently hiring for roles just like this as well as other roles here at Würth Phoenix.ext
As you may have noticed from recent posts from some of my colleagues about "Boosting NetEye CI Speed: Test Parallelization" or "Speeding up the NetEye CI Testing Phase", we are improving our CI in order to have a reliable, fast Read More
Icinga Director is a powerful configuration tool included in NetEye that streamlines the process of defining and managing monitoring configurations. However, you may encounter issues with duplicate addresses and spaces in hostnames, which can lead to confusion and monitoring inaccuracies. Read More
In today’s digital landscape, ensuring optimal performance and availability of applications is critical. Monitoring tools like Elastic Synthetics Journey Monitor and Alyvix offer unique capabilities tailored to diverse needs. Understanding where each tool excels can help you choose the best Read More
In this blog post, we'll explore the power of custom branding in NetEye reports. The reporting module offers the ability to filter specific hosts and services while also enabling the configuration of custom thresholds to visually emphasize unmet SLAs. Users Read More
Scenario NetEye 4 is a comprehensive monitoring platform which natively supports Business Processes. A Business Process is an abstract view of a customer’s business from the Application point of view. Usually, it’s a collection of Icinga 2 checks aggregated by Read More