14. 06. 2024 Alessandro Valentini DevOps, NetEye

Automating the Full NetEye Release Procedure

One of the first issues we added more than 2 years ago to our DevOps backlog was automating the infrastructure preparation and release of NetEye, but we postponed it for a long time because it was too big to do.

A bit of background

Every 2 months we release a new NetEye version: this basically means that repos for the new version will become available to our customers and they can then start the upgrade procedure and jump to the next NetEye version.

But… behind this simple concept there is much more: our developers need several tools to speed up their job, and while customers are upgrading those same developers are already working on the next NetEye version.

For internal use we have several tools and pipelines:

Docker images
RPM testing pipelines
ISO build pipeline
vSphere templates
Cluster testing pipeline
Mirror testing pipeline
Azure testing pipeline
User guide
All the repos used by the tools above containing the in-development RPMs

Probably I’ve forgotten something: the entire procedure was 20 pages long and requires a couple of days to be completed.

How to automate this mess?

In this context creating a pipeline was probably not the most complex part, first of all we had to do a lot of preliminary activities:

Review the entire procedure looking for out-of-date documentation and deprecated steps
Re-write the out-of-date documention
Identify useless or over-engineered steps
Create a single source of truth from which all the pipelines may pick the right information about the version to be tested (e.g. in some cases we test the in-development version, in others we will also test the stable version)
Automate some manual procedures (opening repos to customers, create vSphere templates, …)
Create a single job which will trigger all the required changes and monitor the results

Implementation

The first thing was the creation of a single source of truth: there were several jobs with hard-coded versions that had to be changed manually at every release. We created an API which picks data from Jira and our repos in order to define which version of NetEye is released or in development. An then we integrated all the CI jobs with the API in order to automatically switch to the right version.

Next we created some specific new jobs to automate the creation of vSphere templates including introducing a new tool (maybe I’ll say more on this in a future blog post) and we created a simple playbook to handle our security repo setup.

Finally we created a single pipeline to trigger all the various jobs (for example we have steps which trigger a pipeline still on Jenkins to rebuild the ISO, and steps which trigger pipelines on Tekton to rebuild the user guide) and we also added some tasks to automatically bump the package version containing NetEye repo definitions.

Everything has been implemented on Tekton, and now the entire process requires 2-3 hours and just a single click to start.

These Solutions are Engineered by Humans

Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth Phoenix.