NFS and Elasticsearch: A Storage Disaster for Data but a Lifesaver for Snapshots
When designing an Elasticsearch architecture, choosing the right storage is crucial. While NFS might seem like a convenient and flexible option, it comes with several pitfalls when used for hosting live Elasticsearch data (hot, warm, cold, and frozen nodes). However, NFS proves to be an excellent choice for storing snapshots and searchable snapshots. Here’s why.
Why NFS is a Bad Choice for Elasticsearch Data
1. Poor Performance and High Latency
NFS is a network protocol, introducing additional latency compared to local storage (NVMe, SSD, or HDD). Elasticsearch is highly sensitive to disk latency, as it performs numerous real-time I/O operations, such as:
Writing and updating indices
Searching and aggregating
Replica and segment rebalancing across nodes
The latency introduced by NFS can severely degrade cluster performance, slowing down queries and indexing operations (check the Elastic discussion forum here).
2. Locking and Concurrency Issues
Elasticsearch relies on advanced locking mechanisms to ensure data consistency. However, NFS does not handle file system locks well, especially when multiple nodes access the same segments simultaneously. This can lead to:
Index corruption
Lock errors on shards
Failures during shard rebalancing or recovery
3. Weak Consistency and Crash Recovery Problems
NFS does not provide the same level of consistency and persistence guarantees as local file systems like XFS or ext4. In the event of a node crash or loss of connection to the NFS server, Elasticsearch might end up in an inconsistent state, resulting in hard-to-diagnose errors.
4. Scalability Bottlenecks
Elasticsearch is designed to scale by distributing the load across multiple nodes, each with its own local storage. Using NFS as shared storage for multiple nodes introduces contention, becoming a bottleneck that limits the cluster’s ability to scale efficiently.
Why NFS is Perfect for Snapshots (and Searchable Snapshots)
1. Snapshots: Reliable and Scalable Backups
Elasticsearch snapshots are point-in-time backups of indices, used for disaster recovery or data migration. In this case, NFS is an excellent choice because:
Snapshots are sequential operations, and thus not dependent on low-latency disk performance
NFS is easily expandable, allowing you to store large numbers of snapshots without impacting cluster performance
Snapshot recovery is asynchronous, so network latency does not affect Elasticsearch’s real-time operations
2. Searchable Snapshots: Cost Optimization and Efficient Archiving
For the cold and frozen tiers, Elasticsearch introduces searchable snapshots, allowing searches to be performed directly on snapshots without restoring them first. Here, NFS provides significant advantages:
Searchable snapshots are accessed in read-only mode, avoiding the locking and consistency issues of using NFS for live data
Local storage savings: searchable snapshots eliminate the need to keep full local copies of indices, reducing storage costs
On-demand access: searchable snapshot data is read-only when needed, making NFS latency acceptable for data that’s accessed less frequently
Want to know more about snapshots? Check-out the Elastic Documentation here:
Using NFS as primary storage for Elasticsearch is highly discouraged due to latency, locking, consistency, and scalability issues. However, NFS is an excellent solution for managing snapshots and searchable snapshots, providing a reliable backup strategy and efficient long-term data management in cold and frozen tiers.
If you’re deciding where and how to use NFS in your Elasticsearch cluster, use it for snapshots—but never for live indices!
I'm an IT professional with a strong knowledge of Security Information and Event Management solutions.
I have proven experience in multiple Enterprise contexts with managing, designing, and administering Security Information and Event Management (SIEM) solutions (including log source management, parsing, alerting and data visualizations), its related processes and on-premises and cloud architectures, as well as implementing Use Cases and Correlation Rules to enable SOC teams to detect and respond to cyber threats.
Author
Matteo Cipolletta
I'm an IT professional with a strong knowledge of Security Information and Event Management solutions.
I have proven experience in multiple Enterprise contexts with managing, designing, and administering Security Information and Event Management (SIEM) solutions (including log source management, parsing, alerting and data visualizations), its related processes and on-premises and cloud architectures, as well as implementing Use Cases and Correlation Rules to enable SOC teams to detect and respond to cyber threats.
Just like last year, we had the wonderful opportunity to attend FOSDEM, the most important open source conference in Europe. This year was no exception, and among the many exciting talks, one that particularly caught my attention was Alex Stefanini’s Read More
In a previous blog post by one of my colleagues, we shared how we developed a powerful semantic search engine for our NetEye User Guide. This solution uses Elasticsearch in combination with machine learning models like ELSER to index and Read More
When using Kibana in environments that require a proxy to reach external services, you might encounter issues with unrecognized SSL certificates. Specifically, if the proxy is exposed with its own certificate and acts as an SSL terminator, requests made by Read More
In a previous post we went through the configuration of Elastic Universal Profiling in NetEye, seeing how we can profile applications written in programming languages that do not compile to native code (for example Python, PHP, Perl, etc.) But what Read More
Elastic 8.16, which comes with NetEye 4.39, made Elastic Universal Profiling generally available for self-hosted installations. This means that NetEye SIEM installations will now be able to take advantage of the continuous profiling solution by Elastic. In this blog post Read More