31. 12. 2021 Damiano Chini Development, Log Management, Log-SIEM, NetEye

Real Time Log Signing on Fleet-managed Elastic Agents – A Preliminary Investigation

The R&D Team is currently working on the integration of the new Elastic Fleet management tool in NetEye 4. Once Elastic Fleet is fully integrated in NetEye 4, all of the Log Management features currently supported will also need to work with the Elastic Fleet.

In particular, the integration of Elastic Fleet with the Log Management Real Time Log Signing currently performed by El Proxy is of particular interest. Currently El Proxy receives documents to be signed on a dedicated endpoint invoked by Logstash. The challenge here is that Elastic Agents managed via Fleet only support “Elasticsearch” outputs.

This means that Elastic Agents expect to talk directly with Elasticsearch when writing data or getting some data from Elasticsearch, thus we can’t tell Fleet-managed Elastic Agents to send data to Logstash, nor tell Elastic Agents to write data to our dedicated El Proxy endpoint.

Given these facts, how can we then support the Real Time Log Signing of documents coming from Fleet-managed Elastic Agents?

What we soon understood is that to achieve our goal, the Elastic Agents need to talk to Elasticsearch transparently when passing through El Proxy. In other words, El Proxy should act as a real proxy between Elastic Agents and Elasticsearch, and perform the Real Time Log Signing on the fly when Elastic Agents try to index documents in Elasticsearch.

During our investigations we found out that the Elastic Agents use the standard Elasticsearch REST APIs on the HTTP protocol, with, for example, some GET requests to retrieve information from Elasticsearch and bulk requests to write documents in Elasticsearch.

Given these facts, we understood that a possible solution to our challenge could be an extension to El Proxy, in which we directly pass to Elasticsearch all requests coming from Elastic Agents as they are, except for indexing requests (i.e., bulk requests). When El Proxy encounters an indexing request, it then modifies the documents contained in such requests as normal by performing the Log Signing of the documents.

The Log Signing process itself would not differ from the current implementation. What would change is just the way El Proxy receives the documents to be signed.

To see if this solution is viable and to discover potential problems at an early stage, we implemented the El Proxy extension prototype and configured the output of Elastic Agents to point to El Proxy instead of Elasticsearch and got promising preliminary results. The prototype manages to put documents sent by Elastic Agents into dedicated blockchains which can be successfully verified, and the Elastic Agent behaves correctly thanks to the fact that all requests except indexing requests are transparently passed to Elasticsearch by El Proxy.

All of these investigations allowed us understand that El Proxy can actually perform Real Time Log Signing on data coming from Elastic Agents. This is a great result since in the future this could allow us to remove Logstash from the data flow, hence simplifying the overall Log Management architecture.

Many more questions about the integration of Elastic Agents in the Real Time Log Signing are yet to be answered, mostly regarding the way the Log Management administrator can decide which documents coming from the Elastic Agents need to be signed. In the coming weeks we’ll work actively to design appropriate solutions to these challenges, and we’ll keep you updated on our findings!