Elasticsearch limits the number of open shards per node with the max_shards_per_node cluster setting, which defaults to 1000. The limit on the total number of shards is then calculated from this setting with this formula:
total_max_number_of_shards = cluster.max_shards_per_node * number of non-frozen data nodes
If the total number of shards is reached either by a single node or multiple nodes, then no new shards are written. To avoid this behavior you should monitor these values and take action ahead of time, by setting a higher per node value or by cleaning up (e.g., deleting) shards/indexes.
To help you do this I wrote a plugin, which you can download here. You should add a service to your monitoring setup like Elasticsearch_Shards_Status on any one of your Elasticsearch cluster nodes, which calls a command like this:
This will then check if you reach the warning or critical value for the possible total number of shards and/or for those on the Node-Name you specified. Please note that the Node-Name has to be the name with which the node is registered in the Elasticsearch cluster.
I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.
Author
Juergen Vigna
I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.
Say you want to monitor logs coming into your Elasticsearch instance, and have it send data to your Monitoring Dashboard. I'll show you how to do this with a practical example, in particular for an event coming from the Active Read More
First of all, I'll briefly explain what the "Tornado" in NetEye actually is. Tornado is a Complex Event Processor that receives reports of events from data sources such as monitoring, email, and SNMP Traps, matches them against rules you've configured, Read More
Index Lifecycle Management (ILM) policies constitute a fundamental component in Elasticsearch index management. They enable users to define the life stages of an index, determining when and how specific actions, such as transitioning from a "hot" to a "cold" state Read More
In my previous blog post, we saw how it's possible to index some documents that we created by crawling our NetEye User Guide, then applying the ELSER model in Elasticsearch to create a bag of words for searching that takes Read More
Once upon a time (in fact it was just a month ago, but it sounds more dramatic this way) I had the opportunity to attend a webinar about Vector Search, Generative AI, and modern NLP by the Elastic Team. One Read More