When designing an Elasticsearch architecture, choosing the right storage is crucial. While NFS might seem like a convenient and flexible option, it comes with several pitfalls when used for hosting live Elasticsearch data (hot, warm, cold, and frozen nodes). However, NFS proves to be an excellent choice for storing snapshots and searchable snapshots. Here’s why….
Read MoreIn a previous blog post by one of my colleagues, we shared how we developed a powerful semantic search engine for our NetEye User Guide. This solution uses Elasticsearch in combination with machine learning models like ELSER to index and query our documentation. While the proof of concept (POC) worked great, there was a challenge…
Read MoreHello everyone! As you may remember, a topic I like to discuss a lot on this blog is the Proof of Concept (POC) about how we could enhance search within our online NetEye User Guide. Well, we’re happy to share with you how we’ve further developed the POC, specifically the process outlined in the first…
Read MoreWe all know that NetEye Upgrades are boring activities. Upgrading is important and useful because it brings you bug fixes and new features, but nonetheless it’s extremely expensive in terms of time. The most boring, tiring and lengthy part is when you restart NetEye Services; if you have the SIEM Module installed, the time spent…
Read MoreHey everyone! We played around a bit last time with our radar data to build a model that we could train outside Elasticsearch, loading it through Eland and then applying it using an ingest pipeline. But since our data is in the form of vectors, could we actually exploit Elasticsearch vector database functionality and perform…
Read MoreHi all, it’s been a while. I’m deeply sorry not to have sent out some blog posts lately, so now I’ll try to get back your trust by providing some useful information. Not only that, I’ll even go out of my comfort zone: instead of NetEye Core and monitoring strategies, I’ll talk about NetEye SIEM…
Read MoreElasticsearch limits the number of open shards per node with the max_shards_per_node cluster setting, which defaults to 1000. The limit on the total number of shards is then calculated from this setting with this formula: total_max_number_of_shards = cluster.max_shards_per_node * number of non-frozen data nodes If the total number of shards is reached either by a…
Read MoreSay you want to monitor logs coming into your Elasticsearch instance, and have it send data to your Monitoring Dashboard. I’ll show you how to do this with a practical example, in particular for an event coming from the Active Directory where a user is locked out, and the associated Domain Controller sends the event…
Read MoreFirst of all, I’ll briefly explain what the “Tornado” in NetEye actually is. Tornado is a Complex Event Processor that receives reports of events from data sources such as monitoring, email, and SNMP Traps, matches them against rules you’ve configured, and executes the actions associated with those rules, which can include sending notifications, logging to…
Read MoreIndex Lifecycle Management (ILM) policies constitute a fundamental component in Elasticsearch index management. They enable users to define the life stages of an index, determining when and how specific actions, such as transitioning from a “hot” to a “cold” state or deleting obsolete indices, should occur. ILM policies empower users to ensure the optimal distribution…
Read More
In my previous blog post, we saw how it’s possible to index some documents that we created by crawling our NetEye User Guide, then applying the ELSER model in Elasticsearch to create a bag of words for searching that takes into account the context of the various documents. Moreover, we also performed a simple query…
Read More
Once upon a time (in fact it was just a month ago, but it sounds more dramatic this way) I had the opportunity to attend a webinar about Vector Search, Generative AI, and modern NLP by the Elastic Team. One of the topics that was touched on during the webinar was ELSER , Elastic’s new…
Read MoreWith the introduction of the Composable Index Templates in Elastic, we at NetEye had to redesign the way index settings and mappings are applied to the indices generated by El Proxy. In this post I’ll explain: The solution explained in the remainder of the post was designed for El Proxy, but it may apply also…
Read MoreIn a previous article, we used NetEye and Elasticsearch to train a machine learning model able to classify documents about some collected radar signals, separating them into two categories (good vs bad), starting from an existing dataset. Afterwards, we applied it to new incoming documents using an Ingest Pipeline and the Inference Processor. Taking as…
Read MoreIn a previous article, we explored the Machine Learning capabilities of Elasticsearch, which allowed us to apply anomaly detection techniques to our data, and helped us discover some really interesting facts as a result of our analysis. But can we take that idea even further? For instance, could we use data we’ve already collected to…
Read More