02. 10. 2024
Log Management, Log-SIEM, Machine Learning, NetEye
24. 10. 2024
Reinhold Trocker
Log Management, Log-SIEM
Categories of Documents – Create More Namespaces within an Agent’s Environment
In the ever-evolving landscape of IT monitoring and management, the ability to efficiently handle multi-dimensional namespaces is crucial. Within NetEye, Log-SIEM (Elastic), provides a comprehensive solution for managing the single namespace dimension with the namespace of a data_stream. This blog post delves into the intricacies of multi-dimensional namespaces and how NetEye’s Log-SIEM solution simplifies their management.
- Understanding Multidimensional Namespaces
- An Example Two-dimensional Namespace
- Two Simple Approaches
- Conclusion
Understanding Multidimensional Namespaces
A namespace in IT is a container that holds a set of identifiers, ensuring that all names within it are unique. When we talk about multi-dimensional namespaces, we refer to namespaces that span multiple dimensions, such as time, location, and context. These are particularly common in large-scale IT environments where resources and services are distributed across various regions and time zones.
Potential challenges in managing multidimensional namespaces
- Complexity: Managing namespaces across multiple dimensions can be complex
- Scalability: As the number of dimensions increases, the scalability of the namespace management system becomes a critical concern
- Consistency: Ensuring consistency across different dimensions and avoiding conflicts is a significant challenge
An Example Two-dimensional Namespace
Let’s assume that within the collected logs in your SIEM system, you classify the servers (document sources) into the following two dimensions:
- Customer/tenant (customerA, customerB, customerC)
- Environment (development, test, production)
So typically you would create Elastic Agent Policies using its namespace functionality either:
- For each customer/tenant
- For each environment
Two Simple Approaches
Approach 1: A naming convention approach to two-dimensional namespaces
Continuing with the example above, let’s say there are only 3 different tenants and 3 environments. You could use the namespace functionality of Elastic to map the two dimensions by just combining two strings as in the following table:
| Elastic namespace (data_set.namespace) | Development | Test | Production |
| customerA | customer_a.dev | customer_a.test | customer_a.prod |
| customerB | customer_b.dev | customer_b.test | customer_b.prod |
| customerC | customer_c.dev | customer_c.test | customer_c.prod |
Here you can see that there are 3×3=9 namespaces, and correspondingly 9 Elastic Agent Policies that you’ll have to create and maintain.
Pros:
- For access rights (authorization), you can continue to assign indices to different user groups (i.e., the tenant customerA would get access to all
logs-*-customer_a.*)
Cons:
- You can imagine that with growing numbers (let’s say you acquire 10 more customers), this will quickly grow and lead to a scalability problem doing manual policy management.
- Imagine that the dev and test environments in the above example produce only a very limited number of logs. In that case two thirds of indices in our Elastic infrastructure would be nearly empty, even though Elastic resources are bounded.
Approach 2: Multidimensional namespaces with environment variables
In the example above, the two categories/dimensions can be seen as two properties of a SIEM document. If you maintain one field per (additional) category in the SIEM document, it’s clear which category each document belongs to.
The question is: how can we manage the additional category information without using the Elastic namespace functionality.
In our example you’d probably use the elastic namespace functionality for the customer/tenant information like this:
| Elastic namespace (data_set.namespace) | Any environment |
| customerA | customerA |
| customerB | customerB |
| customerC | customerC |
The missing environment information then has to be “injected” in some other way.
Curious how this works? Please see the next paragraph 😉
Approach 3: Practical implementation
The missing environment information can be “injected” using environment variables on the operating system (server) which is running an Elastic Agent instance. Such servers usually only reside in one environment, in other words: a server is either a development server or a test server or a production server.
The following settings will set the environment variable EA_ENVIRONMENT and add this field to incoming documents, using the Elastic 8.15+ feature add-custom-fields to set document field service.environment to the content of EA_ENVIRONMENT variable.
Set Windows environment variable on Elastic Agent of log source
On each log source computer (running Elastic Agent) where you wish to set the specific environment variable, change it:
- In system environment variables
- Or in the specific Elastic Agent service like this:
In the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Elastic Agent , set a new multi string value named “Environment” with the variables you want to set
Set/add Linux environment variable on Elastic Agent of log source
Usually a tgz-deployed Elastic Agent in a Linux OS reads its environment variables from /etc/sysconfig/elastic-agent. You can use the following script to add an entry in that file:
if [ -d /etc/sysconfig ]; then echo EA_ENVIRONMENT=test >> /etc/sysconfig/elastic-agent; else mkdir /etc/sysconfig; echo EA_ENVIRONMENT=test >> /etc/sysconfig/elastic-agent;fi
Configure the Elastic Agent policy
For every (one-dimensional) Elastic Agent policy you want to add to the new additional field, use the feature add-custom-fields to set additional fields at the policy level:
- Choose your Elastic Agent policy
- Go to the Settings tab
- Click on the “Add another field” button
- Add the field name you want to find in your document
- Add the desired (previously set) environment variable
(in this example${env.EA_ENVIRONMENT|''})
Example Result
Notes
This solution relies on the fact that every Elastic Agent instance is only a part of one category of the wanted additional namespace (in this example “environment”).
If you want additional dimensions, just repeat the steps of:
- Adding environment variables
and - Adding “Add another field” in elastic agent policies
The part|''of the variable reference is only to make sure that a non-set environment variable will be interpreted as empty, see alternative variables and constants
If you have roles that should only have access to a subset of the new namespace, please refer to Document level security
Conclusion
Handling multi-dimensional namespaces can be easily done by setting environment variables where the Elastic Agent is running, and using its definition in the Elastic Agent policy that is used by this Elastic Agent.
Reinhold Trocker
IT professional, IT security, (ISC)2 CISSP, technical consultant
Author
Latest posts by Reinhold Trocker
19. 10. 2023
Log Management, Log-SIEM
Integration of Elasticsearch Clients without Authentication and without TLS
12. 10. 2023
Log Management, Log-SIEM
stunnel TCP keepalive Settings Preventing Firewall from Blocking Log Traffic