01. 04. 2021 Damiano Chini Development, Log Management, Log-SIEM, NetEye

Log Management – Real Time Log Signing

Meeting the highest security standards is an absolute priority in NetEye. For this reason, in the continuous process of improving security in NetEye 4, we brought an important architectural improvement in the Log Manager module in the NetEye 4.17 release.

The new architecture takes the name of Real Time Log Signing and its main focus is to strengthen the guarantees on the integrity and the confidentiality of the logs handled by the Log Manager module.

The new Log Manager architecture is shown below, where the Logstash EBP Pipeline and the El Proxy are its main components, while the Logstash Main Pipeline and the “Other Sources” remain present in NetEye 4 for backwards compatibility with the architecture based on rsyslog.

One of the requisites to guarantee data integrity (log integrity in our case), is avoiding data loss. With the Real Time Log Signing architecture we guarantee that, from the moment that a log arrives to the Logstash, the log will not be lost, thanks to the preconfigured Logstash persistent EBP Pipeline.

The persistent Logstash pipeline ensures that what arrives to the pipeline will never be discarded (not even after a shutdown of the machine) until it is successfully output, which in our case means that logs will never be discarded until they are successfully saved in Elasticsearch.

To improve the guarantees on the integrity and the confidentiality of the logs, it is necessary prevent everyone from accessing and altering the logs before they are secured (i.e. signed). This is why one of the main goals of the Real Time Log Signing is to sign and store logs on the fly, in real time, without giving the possibility to alter a log before it is signed, even by having access to the filesystem of the NetEye 4 machine.

To sign the logs on the fly, the Real Time Log Signing relies on the completely new component El Proxy. El Proxy is a daemon written in Rust which performs a blockchain signature of each single log sent by Logstash, and immediately stores it in the Elasticsearch indices.

The blockchain of logs gives us a lot of guarantees on the integrity of the logs, because no log can be altered without having access to a secret key, and no log can be altered without altering all of the subsequent logs, which becomes very challenging for a possible attacker.

The integrity of the blockchain can be verify at any moment with a CLI tool provided by El Proxy itself.

We wanted to allow the users to use all of the Elasticsearch features that they want for managing the logs, including for example the Elasticsearch Ingest pipelines which modify the incoming documents before being indexed. As you understand, this may have been a problem since the signing of the logs is performed by El Proxy before being sent to Elasticsearch.

In order to overcome this, El Proxy stores all the information related to the signature of the log in a dedicated custom field of the log, named . This way, the logs can be managed in Elasticsearch without affecting the validity of the blockchain.

All of these security features would be useless if the communications between components could be intercepted and/or modified, so for this reason all communications are secured by TLS.