17. 01. 2022 Massimo Giaimo Blue Team, SEC4U

An Evaluation of Elastic EDR with APT Simulator

We decided to carry out an evaluation of Elastic’s EDR using the APT Simulator (https://github.com/NextronSystems/APTSimulator) tool.

This tool is widely used within the cyber security community and is highly reputed, as it was developed by Florian Roth, who also created the Sigma Rule project.

APT Simulator is a Windows Batch script that uses a set of tools and output files to make a system look as if it has been compromised. In contrast to other adversary simulation tools, APT Simulator is designed to make the application as simple as possible. You don’t need to run a web server, database or any agents on a set of virtual machines. Just download the prepared archive, then extract and run the Batch file it contains as Administrator.

The following table shows the various test cases carried out by the tool along with their category.

Test Case	Category
Collect Local Files	Collection
C2 Connects	Command and Control
DNS Cache 1 (Cache Injection)	Command and Control
Malicious User Agents (Malware, RATs)	Command and Control
Ncat Back Connect (Drop & Exec)	Command and Control
WMI Backdoor C2	Command and Control
LSASS Dump (with Procdump)	Credential Access
Mimikatz 1 (Drop & Exec)	Credential Access
WCE 1 (Eventlog entries)	Credential Access
Active Guest Account Admin	Defense Evasion
Fake System File (Drop & Exec)	Defense Evasion
Hosts File (AV/Win Update blocks)	Defense Evasion
Obfuscated JS Dropper	Defense Evasion
Obfuscation (RAR with JPG ext)	Defense Evasion
Nbtscan Discovery (Scan & Output)	Discovery
Recon Activity (Typical Commands)	Discovery
PsExec (Drop & Exec)	Execution
Remote Execution Tool (Drop)	Execution
At Job	Persistence
RUN Key Entry Creation	Persistence
Scheduled Task Creation	Persistence
StickyKey Backdoor	Persistence
UserInitMprLogonScript Persistence	Persistence
Web Shells	Persistence
WMI Backdoor	Persistence

The details of the various tests carried out by APT Simulator are shown below.

Collection

Collect Local Files

Drops pwdump output to the working dir
Drops directory listing to the working dir

Command and Control

C2 Connects

Uses Curl to access well-known C2 servers

DNS Cache 1

Looks up several well-known C2 addresses to initiate DNS requests and get the addresses into the local DNS cache

Malicious User Agents

Uses malicious user agents to access web sites

Ncat Back Connect

Drops a PowerShell Ncat alternative to the working directory and runs it to back connect to a well-known attacker domain

WMI Backdoor C2

Uses Matt Graeber’s WMIBackdoor to contact a C2 at certain intervals

Credential Access

LSASS DUMP

Dumps LSASS process memory to a suspicious folder

Mimikatz-1

Dumps mimikatz output to working directory (fallback if other executions fail)
Runs a special version of mimikatz and dumps the output to working directory
Run Invoke-Mimikatz in memory (github download, reflection)

WCE-1

Creates Windows Eventlog entries that look as if WCE has been executed

Defense Evasion

Active Guest Account Admin

Activates Guest user
Adds Guest user to the local administrators

Fake System File

Drops suspicious executable with system file name (svchost.exe) in %PUBLIC% folder
Runs that suspicious program in %PUBLIC% folder

Hosts

Adds entries to the local hosts file (update blocker, entries caused by malware)

JS Dropper

Runs obfuscated JavaScript code with wscript.exe and starts decoded bind shell on port 1234/tcp

Obfuscation

Drops a cloaked RAR file with a JPG extension

Discovery

Nbtscan Discovery

Scans 3 private IP address class-C subnets and dumps the output to the working directory

Recon

Executes commands used by attackers to get information about a target system

Execution

PsExec

Dumps a renamed version of PsExec to the working directory
Runs PsExec to start a command line in LOCAL_SYSTEM context

Remote Execution Tool

Drops a remote execution tool to the working directory

Lateral Movement

No test cases yet

Persistence

At Job

Creates an at job that runs mimikatz and dumps credentials to file

RUN Key

Create a suspicious new RUN key entry that dumps “net user” output to a file

Scheduled Task

Creates a scheduled task that runs mimikatz and dumps the output to a file

Scheduled Task XML

Creates a scheduled task via XML file using Invoke-SchtasksBackdoor.ps1

Sticky Key Backdoor

Tries to replace sethc.exe with cmd.exe (a backup file is created)
Tries to register cmd.exe as a debugger for sethc.exe

Web Shells

Creates a standard web root directory
Drops standard web shells into that directory
Drops a GIF obfuscated web shell into that directory

UserInitMprLogonScript Persistence

Uses the UserInitMprLogonScript key to get persistence

WMI Backdoor

Uses Matt Graeber’s WMIBackdoor to kill local procexp64.exe when it starts

For our tests we used a Windows 10 client, with the latest available updates applied. The installation of the EDR Endpoint Security was carried out through the Fleet Server present within our SOC and a default policy was applied, without any particular exceptions.

Below are the results of the various tests performed and the possible detection capability by Elastic’s EDR:

Test Case	Detection Elastic EDR
Collect Local Files
C2 Connects	OK
DNS Cache 1 (Cache Injection)	OK
Malicious User Agents (Malware, RATs)	OK
Ncat Back Connect (Drop & Exec)	OK
WMI Backdoor C2	OK
LSASS Dump (with Procdump)	OK
Mimikatz 1 (Drop & Exec)	OK
WCE 1 (Eventlog entries)	OK
Active Guest Account Admin	OK
Fake System File (Drop & Exec)	OK
Hosts File (AV/Win Update blocks)	OK
Obfuscated JS Dropper	OK
Obfuscation (RAR with JPG ext)	OK
Nbtscan Discovery (Scan & Output)	OK
Recon Activity (Typical Commands)	OK
PsExec (Drop & Exec)	Failed
Remote Execution Tool (Drop)	Failed
At Job	OK
RUN Key Entry Creation	OK
Scheduled Task Creation	OK
StickyKey Backdoor	OK
UserInitMprLogonScript Persistence	OK
Web Shells	OK
WMI Backdoor	OK

Once the tool has been run from a Powershell shell with administrative privileges, you can choose which tests to run:

During our tests we decided to start each single test individually, so as to be able to verify the events transmitted to Elastic from time to time.

As the tool suggests, if you want to replicate these tests within your environment, we recommend that you take a snapshot of the virtual machine used.

We now present the details of the detection capability of the various test categories.

Collection

This test is out of scope, as it is undetectable by EDR technology.

Command and Control

Result: Detected.

Evidence:

16:07:56.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:798   type:       diagnostic
16:07:56.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:799   name:       Delayed Execution via Ping
16:07:56.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:800   trusted:    false
16:07:56.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:807   alert sent: true
16:07:56.617
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] ResponseActions.cpp:86 Successfully killed process pid=13184, name=powershell.exe, result=0

Credential Access

Result: Detected

Evidence:

16:09:12.437
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:798   type:       diagnostic
16:09:12.437
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:799   name:       Delayed Execution via Ping
16:09:12.437
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:800   trusted:    false
16:09:12.437
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:807   alert sent: true
16:09:12.480
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] BulkQueueConsumer.cpp:224 Sent 1 documents to Elasticsearch
16:09:12.521
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] ResponseActions.cpp:86 Successfully killed process pid=8812, name=powershell.exe, result=0

Defense Evasion

Result: Detected

Evidence:

16:10:37.946
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:798   type:       diagnostic
16:10:37.946
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:799   name:       Execution from Unusual Directory
16:10:37.946
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:800   trusted:    false
16:10:37.946
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:807   alert sent: true
16:10:37.959
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] ResponseActions.cpp:86 Successfully killed process pid=13704, name=wscript.exe, result=0

Discovery

Result: Detected

Evidence:

16:11:50.130
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:798   type:       diagnostic
16:11:50.130
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:799   name:       Delayed Execution via Ping
16:11:50.130
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:800   trusted:    false
16:11:50.130
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:807   alert sent: true
16:11:50.178
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] BulkQueueConsumer.cpp:224 Sent 1 documents to Elasticsearch
16:11:50.341
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] ResponseActions.cpp:86 Successfully killed process pid=9352, name=WMIC.exe, result=0

Execution

Result: Not Detected

Evidence:

Persistence

Result: Detected

Evidence:

16:14:15.612
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:798   type:       diagnostic
16:14:15.612
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:799   name:       Delayed Execution via Ping
16:14:15.612
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:800   trusted:    false
16:14:15.612
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] RulesEngine.cpp:807   alert sent: true
16:14:15.649
elastic_agent.endpoint_security
[elastic_agent.endpoint_security][info] ResponseActions.cpp:86 Successfully killed process pid=12888, name=powershell.exe, result=0

From the point of view of the end user, the detections are displayed through the integration of the EDR in the notification system of the operating system.

While from the point of view of the SOC cyber security analysts it’s possible to obtain evidence of the various attack actions by activating the detection rule available by default within Elastic.