05. 09. 2023 Rocco Pezzani Development, NetEye

Upgrading to the New nep-setup on NetEye 4.31

After a first semester full of great opportunities (many colleagues like to say so), the wheels have started turning again for NetEye Extension Packs. Now, a big evolution is beginning: in the near future, NEP will encompass the entire NetEye infrastructure, meaning it will also configure and manage NetEye Satellites. To do so, the current setup utility (that is nep-setup) has to extend its capabilities and its grasp of the infrastructure it’s working on.

This is going to be a long journey: a lot of new features must be packed into nep-setup, as well as a great deal of improvements in the current behavior. Our wish-list is quite long: improved logging, improved robustness, version management for NEPs, satellite configuration and management, and detailed dry runs. Also, future improvements in the structure of NEPs will include RPM management, an improved import procedure, error handling and so on. A hell of a lot of work, and we have to begin somewhere.

The most obvious thing to do is to rethink and rationalize all the steps performed by nep-setup. While we’re at it, why not use a language that will give us more robustness than Python? So, we rewrote the whole script using Rust: this will allow for better error handling, more detailed logging, and performance improvements. During the rewrite process, we took advantage of the semver crate, allowing us to introduce semantic versioning for NEPs.

All these things per-se are nothing short of a revolution, and even if they are few in number, we think they are still almost too much. Why? Because a NetEye installation that already uses NEP will require some additional steps for it to become understandable by the new nep-setup.

Now, we won’t pretend that all the features listed above are already implemented. For now, we’ve just achieved:

Improved logging
Improved robustness
Version management

In short, the essential behavior of the tool still remains the same, but we’ve placed the cornerstones for future activities.

Reasons behind the manual upgrade

The main point is: version management. Prior to NetEye 4.31, the version of a NEP was just a progressive number. It was done out of simplicity, yet still played its role perfectly: help finding if something changed, no matter how small it might be. Now, because the NEP Project is becoming more and more enterprise-class, we need some serious version management for all NEPs. Therefore, the new nep-setup is preparing the terrain for this feature by introducing version semantics; however, this change is not backward compatible.

Another reason: Python works well with files in INI format, while Rust prefers the TOML format. INI and TOML are very similar, but not exactly the same: an INI file is not a TOML file, and Rust wants a TOML file. All of this means: the prerequisite file of each NEP (file prerequisite.ini shipped with every NEP) must be converted to TOML format, and it must implement version semantics in its directives.

The new prerequisite files are shipped via RPM, but an existing system keeps a copy of all installed NEPs in a separate directory for management reasons. The contents of this directory must be updated to make them understandable to the new nep-setup; otherwise, each NEP will be installed as if it were new, causing a lot of damage to your NetEye installation. While we’ve automated this migration, we don’t feel comfortable automatically running it during the RPM installation, and since it’s a one-time activity we don’t want to integrate it into the nep-setup code. So you’ll have to launch it manually on your system.

The migration process

The migration process consists of just two steps that you need to perform on your NetEye Master, be it a single node or a cluster. If it’s a single node, just run the required steps on that node; if it’s a cluster, run each step on all PCS cluster nodes (excluding voting nodes and Elastic data only nodes, where it’s not required). Ah yes, the obvious requirement is: your NetEye Master must be at version 4.31.

Step 1: Upgrade NEP RPM

Ensure you have the latest available version of nep-setup installed on your NetEye Master. To do so, update the neteye-nep RPM by running this command:

dnf -y --enablerepo=neteye-contrib update neteye-nep

This will ensure the latest version of the NEP binaries are on your system.

As you may notice, alongside the updated version of neteye-nep RPM, DNF will also install Python 3.9: it’s a requirement for the migration process. Don’t worry, it won’t disturb your system.

Step 2: Perform the migration

Run the migration script provided with NEP, it will take care of everything. Just run the following command:

bash /usr/share/neteye/nep/setup/migration/migrate-nep-data.sh

Obviously, the output varies based on which NEP you have installed on your system.

Now you can enjoy the new nep-setup:

If the migration process didn’t complete correctly, nep-setup will not be able to run. If you try to start it, you will get a cryptic error like the one in this screenshot:

If this happens, don’t worry, the migration script is designed so it can be executed as many times as you want, it will simply continue from where the last run stopped.

These Solutions are Engineered by Humans

Did you find this article interesting? Does it match your skill set? Our customers often present us with problems that need customized solutions. In fact, we’re currently hiring for roles just like this and others here at Würth Phoenix.