30. 12. 2024 Alessandro Taufer DevOps, Log-SIEM

Configure kubernetes index lifecycle policies in Elastic Stack

If you are monitoring an Openshift or a Kubernetes cluster with Elastic Stack you might have noticed that the kubernetes integration uses the default Index Lifecycle Policy. It means that those logs and metrics have an unlimited retention. If the volume of logs is high, and for kubernetes clusters it usually is, it won’t be sustainable.

Unfortunately due to the design of the integration, configuring them it’s a bit trickier than it might seem.

Where the problem stands

A quick search in the data streams section will reveal that the integration by default uses 22 datastreams.

Screenshot containing the list of datastreams

Configuring them manually is something that should absolutely be avoided for multiple reasons:

It is error prone
The slow manual operation will need to be repeated every time you edit them
None of your colleagues will be sure about how they are configured without checking each one of them

Luckily, I already created a small script that you can adapt to your use cases. It performs mainly two operations:

Edits the custom component template
Forces the rollover and removes legacy indexes

Editing the custom component template

The first step of the procedure consists into assigning the ILP to each datastream. The right way to do it, is by editing the custom component template, this will prevent that future updates of the integration reset back the default setting.

# The name of the ILP you want to use
index_lifecycle_policy_name="kubernetes-metrics"

# Authentication credentials of elastic
ELASTICSEARCH_MONITORING_CERTS_PATH="/neteye/local/elasticsearch/conf/certs"
elastic_curl="/usr/bin/curl -sS -E $ELASTICSEARCH_MONITORING_CERTS_PATH/admin.crt.pem --key $ELASTICSEARCH_MONITORING_CERTS_PATH/private/admin.key.pem https://elasticsearch.neteyelocal:9200"

# Retrieve the list of custom component templates of the kubernetes integration
kube_component_templates=$($elastic_curl/_component_template/*kubernetes*@custom | jq -r '.component_templates[].name')
for component_template in $kube_component_templates; do
  echo "Configuring component template: $component_template"

  # Set the ILP for the given custom component template
  $elastic_curl/_component_template/$component_template \
    -H 'Content-Type: application/json' \
    -d "{
      \"template\": {
        \"settings\": {
          \"index.lifecycle.name\": \"${index_lifecycle_policy_name}\"
        }
      }
    }"
done

Force the rollover of the indexes

Even if we assigned the Index Lifecycle Policy to the datastreams, it won’t be used by the indexes that already exist. You will have to force a rollover to bring them into use (close the index and create a new one).

# Retrieve the list of datastreams of the kubernetes integration
kube_data_streams=$($elastic_curl/_data_stream/*kubernetes* | jq -r '.data_streams[].name')

for kube_data_stream in $kube_data_streams; do
  echo "Data stream: $kube_data_stream"

  # Force the rollover of the index
  $elastic_curl/$kube_data_stream/_rollover -X POST
done

(Optional) Remove the legacy indexes

When forcing a rollover of indexes, be aware that the old indexes remain in the system and continue to follow their original lifecycle policies, preventing automatic deletion. After confirming that these legacy indexes are no longer needed for your operations, you can proceed with a bulk deletion to reclaim storage space.

# Remove legacy indexes
for kube_data_stream in $kube_data_streams; do
  # Retrieve the list of indexes associated with the data stream that have an ILM policy different from the one we want to set
  indexes=$($elastic_curl/_data_stream/${kube_data_stream} -H 'Content-Type: application/json' | jq -r ".data_streams[0].indices[] | select(.ilm_policy != \"$index_lifecycle_policy_name\") | .index_name")

  # Delete each index
  for index in $indexes; do
    echo "Deleting index: $index"
    $elastic_curl/$index -X DELETE -H 'Content-Type: application/json'
  done
done

Conclusion

Exercise caution when tweaking this code to fit your needs. It’s quite flexible, but these changes can lead to unexpected results if not handled carefully. You can find the integral version of the code here

Interest in more readings about monitoring Kubernetes? You might find this article useful!

These Solutions are Engineered by Humans

Did you find this article interesting? Does it match your skill set? Programming is at the heart of how we develop customized solutions. In fact, we’re currently hiring for roles just like this and others here at Würth Phoenix.