The purpose of this article is to show a real-life case study of the integration of the new Elastic ES|QL language within the detetion rules used by the SOC to detect cyber threats.
Overview
ES|QL (Elasticsearch Query Language) is a SQL-like query language developed by Elastic specifically for querying time series and event data stored in Elasticsearch. It is designed to make it easier to query structured data using a syntax that is familiar to anyone who has used SQL (https://www.elastic.co/guide/en/elasticsearch/reference/current/esql.html).
Here the query structure:
SELECT field1, field2
FROM index_name
WHERE condition
ORDER BY field1 DESC
LIMIT 10
The case proposed
In this blog I would like to show the remarkable improvement of one particular case using the ES|QL language.
Let us consider the assumption that we want to detect the creation of a considerable amount of files on a Windows system in a very short period of time. This could certainly indicate a very normal routine activity of the system (updates/upgrades), but there could also be the possibility (not so remote) of detecting a mass encryption action, typical behaviour of ransomware.
The idea is therefore to define a rule that triggers an alert when the total number of files created on the same machine by the same process exceeds a certain threshold.
The Threshold Rules
Before the introduction of the new ES|QL language, to create such a rule based on a threshold, it was necessary to use rules of type Threshold.
Below is a simplified basic example.
event.category:file
and event.action: ("file created (rule: filecreate)" or "creation" or "filecreate" or "overwrite")
Results aggregated by process.name >= 3000
The concept of the threshold rules is very simple and easy to understand .
It is necessary to define which fields to aggregate and an useful threshold.
There is, however, one strong limitation. Any exceptions to the rule are only possible by considering the fields on which they are to be aggregated, so whitelisting certain patterns rather than certain locations particularly prone to file creation by legitimate processes is not possible.
Fields such as file.path are not taken into account.
Let’s try using ES|QL
The great advantage of ES|QL in these cases in addition to the possibility of aggregating, enriching data and creating sub-queries, an so on, is the possibility of considering for exceptions fields that are not present within the query, but only need to be present in the analysed logs.
Below is the ES|QL rule used to detect the creation of large amounts of data, thus potentially the detection of encryption by ransomware.
FROM logs-endpoint.events.file-*, winlogbeat-*-sysmon-*, logs-windows.sysmon_operational-*
| EVAL file_entropy = COALESCE(file.Ext.entropy, 6.0)
| EVAL proc_trusted = COALESCE(process.code_signature.trusted, false)
| EVAL proc_signature = COALESCE(process.code_signature.subject_name, "notsigned")
| WHERE event.action IN ("file created (rule: filecreate)", "creation", "filecreate", "overwrite")
and event.category == "file"
// filter for high values of entropy
and file_entropy >= 6.0
// ignore some trusted and signed software
and not (proc_trusted==true and proc_signature in ("Microsoft Corporation" ,"Microsoft Windows" ,"Microsoft Windows Publisher", "Microsoft Dynamic Code Publisher", "Adobe Inc.", "Adobe Systems, Incorporated") and (process.name != "powershell.exe" or process.name != "cmd.exe"))
//count the number of created files
| STATS file_count = COUNT(file.name) by host.hostname, process.name, event.action, event.category
| WHERE file_count > 3000
| KEEP host.hostname, process.name, event.action, event.category, file_count
By analysing the query, we can see how the two discriminating factors on which we base ourselves to filter or consider the evidence are entropy and the signature of the process.
As regards entropy, values greater than or equal to 6 are considered (on a scale of 8 values), while as regards the signature the idea is to exclude processes signed by well-known and widespread software houses.
In this case we are still going to consider trusted processes like powershell and cmd as they could be used to run the actual malware.
But if the logs you are considering do not have the necessary fields?
It is fundamental take into account that the data received is collected by several integrations and not all of them provide fields such as entropy, hash or signature of a process.
The EVAL pattern used in combo with the COALESCE command allows us to define default values to be taken into account in case the Elastic logs do not provide them.
So, in this specific case, in order to not lose any evidence, for that logs that do not have the required fields, entropy is set to 6 and the process is considered not trusted.
Abount entropy
Entropy in the context of information theory refers to the measure of uncertainty or randomness in a set of data. It quantifies the unpredictability of information content; higher entropy means more randomness, and lower entropy indicates more predictability.
Essentially, higher entropy makes the encryption more robust by ensuring that the key or data is sufficiently random and unpredictable.
These are the values considered by Elastic in the EDR rule :
Surely taking entropy into account is not a foolproof way, but it is necessary to introduce some filters for such rules otherwise they would be obscured by continuous noise.
On the other hand, two ransomware simulators, the one provided by Elastic (https://github.com/elastic/protections-artifacts/tree/main/ransomware/testing) and QuickBuck (https://github.com/NextronSystems/ransomware-simulator) encrypt data with an entropy greater than 7 and very close to 8.
Final considerations
It is important to understand that rules like this are not substitutes for rules intrinsic to EDR/XDR that consider a series of factors, evidence and techniques and correlate them with each other .
Below are the techniques used for example by Elastic’s EDR to detect ransomware behaviour.
But where an EDR system cannot be installed either due to commercial issues (cost) or architectural/technological issues, rules of this type can help in the detection of potential threats such as ransomware.
The challenge is to try to reduce the noise, isolate false positives as much as possible and continuously improve the detection rules.
Daniel Degasperi
Cyber Security Team | Würth Phoenix
Author
