09. 01. 2020 Damiano Chini Log-SIEM, NetEye

Support for Elasticsearch-only Nodes

Until NetEye 4.8, customers who needed to expand the capacity of their Elasticsearch cluster running alongside their Red Hat cluster could add new standard nodes to NetEye clusters. This meant, however, that the new nodes would dedicate their resources not just to improving the capacity of the Elasticsearch cluster, but also to maintaining all services running in NetEye.

For this reason, starting with NetEye 4.9, we decided to design and develop a solution supporting an architecture where some NetEye cluster nodes run all NetEye resources, while other nodes (which we call Elasticsearch-only nodes) run only Elasticsearch, .

In order to let a cluster node determine whether it should run all NetEye resources, rather than only Elasticsearch, we decided to first add a new entry ElasticsearchOnlyNodes to the configuration file /etc/neteye-cluster. This can be specified in addition to the Nodes entry, which defines which nodes are those that belong to the standard Red Hat cluster. A cluster node can then decide on its own if it’s an Elasticsearch-only node or not, just by checking which entry it belongs to (i.e., to either Nodes or ElasticsearchOnlyNodes).

The following is an example of the configuration file for a cluster running an Elasticsearch-only node in addition to two standard NetEye nodes:

{
    "ClusterInterface" : "bond0",
    "Created" : 1555424390,
    "Hostname" : <hostname>,
    "Nodes" : [
       {
          "addr" : <ip_addr>,
          "hostname" : <hostname>,
          "hostname_ext" : <hostname_ext>,
          "id" : 1
       },
       {
          "addr" : <ip_addr>,
          "hostname" : <hostname>,
          "hostname_ext" : <hostname_ext>,
          "id" : 2
       }
    ],
    "ElasticOnlyNodes": [
       {
          "addr" : <ip_addr>,
          "hostname" : <hostname>,
          "hostname_ext" : <hostname_ext>
       }
    ]
}

In addition to this we created a new systemd target dedicated to Elasticsearch only nodes, in which we included only the elasticsearch systemd service. This target is of course enabled on Elasticsearch-only nodes.

We also needed to keep the installation process for Elasticsearch-only nodes as standard as possible. But this installation, of course, differs between standard NetEye nodes and Elasticsearch-only nodes, since a node running only Elasticsearch cannot run all the autosetup scripts normally executed by the neteye_secure_install command.

However, since cluster nodes can recognize whether or not they themselves are Elasticsearch-only nodes, we were also able to retain the neteye_secure_install command as the installation command for Elasticsearch-only nodes just by making the command execute the autosetup scripts contained in the new folder /usr/share/neteye/secure_install_elastic_only rather than the usual scripts in/usr/share/neteye/secure_install. One bit of good news for developers was that most of the autosetup scripts needed for Elasticsearch-only nodes could simply be set up as symbolic links to the standard ones.

A final change to accommodate the addition of Elasticsearch-only nodes is that passwords and certificates are no longer generated by one node and then synced to the other nodes of the cluster. Instead they are now pulled from the other cluster nodes first, and only if they cannot be found are they then generated. However, if the node which is pulling passwords and certificates sees that they are not consistent across cluster nodes, it halts the installation by warning the user about the dangerous cluster state.