Desired State Configuration (DSC) is a feature in Powershell 4.0 and above that helps administrators to automate the configuration of Windows. I’ll show you below how to use it in order to maintain a consistent Icinga agent configuration across your Windows servers.

Our use case

As an admin I would like to distribute and configure the Icinga agent systematically in order to monitor my servers via a NetEye Satellite. The Icinga agent configuration is the same for all the servers.

Requirements

• The target Windows server must be able to receive PowerShell remote commands
• The PowerShell version on both target and source must be at least Powershell 4
• .NET must be 4.6 or above (otherwise Icinga MSI will have to be 10.5.0 or below)
• Windows minimum version is 2008 R2 or 7
• The shared folder with all the configuration files must be reachable from all servers
• You must have local admin rights on all the target servers
• You’ll need NetEye API credentials in order to generate the agent tickets:
  On Satellite it’s: /neteye/local/icinga2/conf/icinga2/conf.d/deploy-api-user.conf
• The NetEye Satellite FQDN must be resolvable by each server
• A working NetEye Satellite
• curl.exe and libcurl.dll
• Icinga windows agents

Source information

• Powershell DSC Requirements
• Powershell DSC Getting Started

Content of the shared folder

1. Create a DSC folder in C:\
2. Inside DSC, copy the DSC_ICINGA.ps1 from this guide (Distributing Icinga paragraph)
3. Inside DSC, create a subfolder called SW
4. Inside SW create a subfolder called icinga2
5. Copy these files into the icinga2 directory
   • curl.exe
   • libcurl-x64.dll
   • the latest icinga2 agent msi from the requirements above

Creating the shared folder

We have to choose one server which will share the configuration files with all the other servers. This is needed as DSC will get the files from the shared folder and then copy them locally onto the target servers. We also usually share the files from the same server that we use to deploy the DSC configurations.

Please note that DSC will use the account you are logged in to on the server as an access token, but will use the SYSTEM account in order to perform the various actions (copy, install, etc). The NTFS folder permissions should be set like this:

• Domain computers, read
• Domain controllers, read

and the share permissions are simple as this:

Example: Creating the Powershell DSC configuration MOF files

The DSC configuration is divided into these steps:

1. Getting a server list from a text file
2. Configuring every server with the same configuration we are creating using the DSC resources

Let’s create a simple DSC configuration which will copy a folder onto the target servers

configuration ServerConfig {

    Import-DscResource -ModuleName PSDesiredStateConfiguration

    #Global configurations
    $list = get-content "C:\DSC\servers.txt"
    node @($list){

            File DirectoryCopyNEP
            {
                Ensure = "Present"
                Type = "Directory"
                Recurse = $true
                SourcePath = '\\FS01\SW\Icinga2NEP'
                DestinationPath = "c:\Scripts\icinga"
                MatchSource = $true
                Checksum = "modifiedDate"
            }

        }

}

ServerConfig -OutputPath "C:\DSC\MOF_ICINGA"

As you can see the DSC will read the server names in servers.txt and will apply the resource File DirectoryCopyNEP to each one. This resource will copy the \\FS01\SW\Icinga2NEP folder to the local C:\Scripts\icinga for each server.

ServerConfig -OutputPath “C:\DSC\MOF_ICINGA” will generate our MOF files. These are the DSC configuration files for each server we defined in servers.txt.

Example: applying the configuration to the servers

Now that we have successfully created the MOF files, we need to apply them. We need to execute the command with an elevated PowerShell shell:
Start-DSCConfiguration -Path C:\DSC\MOF_ICINGA -Wait -Verbose -Force

TIP: use the parameter ‘-ComputerName HOSTNAME‘ to apply the MOF to specific servers

Please note that the command output must be error-free, otherwise DSC will try to apply the configuration to the failed server every 15 minutes. Also if we change any files in the folder we have distributed (both target and source), these changes will be applied to the configured servers every 15 minutes. This means that if you manually change a file on the target server this will be overwritten, or if you changed a file on the FS01 this will be synchronized to the target.

In order to stop this behavior we can stop the DSC with the following commands (or we can keep applying the MOF until we have no errors):

#Remove all mof files (pending,current,backup,MetaConfig.mof,caches,etc) 
rm C:\windows\system32\Configuration\*.mof* 
#Kill the LCM/DSC processes 
gps wmi* | ? {$_.modules.ModuleName -like "*DSC*"} | stop-process -force

Distributing Icinga

The DSC configuration below will do the following:

1. Get the server list from servers.txt
2. Create the MOF files with the configured resources

Note that no duplicates are allowed in servers.txt

Then we can use Start-DSCConfiguration to apply those MOF files. They will:

1. Copy the files needed from the shared folder to the target servers
2. Execute the Icinga script which will install and configure Icinga as long as the TestScript conditions are met (more info)

Note that you must fill in your values where it says CUSTOMER VARIABLES in the script below

configuration ServerConfig {

    Import-DscResource -ModuleName PSDesiredStateConfiguration 

    $list = get-content "C:\DSC\servers.txt"
    node @($list){

            File IcingaMSI
            {
                Ensure = "Present"
                Type = "Directory"
                Recurse = $true
                SourcePath = '\\HOSTNAME\SW\Icinga2'
                DestinationPath = "c:\icinga"
                MatchSource = $true
                Checksum = "modifiedDate"
            }
	        Script Icinga
            {
            SetScript  = {
                ### CUSTOMER VARIABLES ###
                [string]$icinga2ver="2.11.9.123" #latest Icinga MSI version
                [string]$username = "deploy"
                [string]$password = "PWD"
                [string]$parent_zone = "ICINGA_ZONE"
                [string]$sat_server = "SAT_MASTER_FQDN"

                ### STATIC VARIABLES ###
                [string]$workpath="C:\icinga"
                [string]$icinga2="C:\Program Files\ICINGA2\sbin\icinga2.exe"
                [string]$icinga2data = 'C:\Programdata\icinga2'
                [string]$CertificatesPath = "C:\ProgramData\icinga2\var\lib\icinga2\certs"
                [string]$myFQDN=((Get-WmiObject win32_computersystem).DNSHostName+"."+(Get-WmiObject win32_computersystem).Domain).ToLower()

                # 1 step: Icinga2 install msi
                $r = Get-WmiObject Win32_Product | Where {($_.Name -match 'Icinga 2')}
				$test = $r.IdentifyingNumber
                if (($r -ne $null) -and (-not ($r.Version -match $icinga2ver))) {
                	Write-Output "Icinga must be uninstalled" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                	$MSIArguments = @(
                	    "/x"
                	    $r.IdentifyingNumber
                	    "/qn"
                	    "/norestart"
                	)
                	Start-Process "msiexec.exe" -ArgumentList $MSIArguments -Wait -NoNewWindow
                	$r = $null
                    Remove-Item $icinga2data -Force  -Recurse -ErrorAction SilentlyContinue
                }
                if ($r -eq $null) {
                	Write-Output "Icinga must be installed" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                	$MSIArguments = @(
                	    "/i"
                	    $workpath + "\Icinga2-v${icinga2ver}-x86_64.msi"
                	    "/qn"
                	    "/norestart"
                	)
                    Remove-Item $icinga2data -Force  -Recurse -ErrorAction SilentlyContinue
                	Start-Process "msiexec.exe" -ArgumentList $MSIArguments -Wait -NoNewWindow
					$r = Get-WmiObject Win32_Product | Where {($_.Name -match 'Icinga 2')}
                }

                # 2 step: generate ticket from satellite
                $parms = '-k', '-s', '-u', "${username}:${password}", '-H', '"Accept: application/json"', '-X', 'POST', "`"https://${sat_server}:5665/v1/actions/generate-ticket`"", '-d', "`"{ `\`"cn`\`":`\`"${myFQDN}`\`" }`""
                $cmdOutput = &"$workpath\curl.exe" @parms | ConvertFrom-Json

                if (-not ($cmdOutput.results.code -eq "200.0")) {
                    Write-Output "Cannot generate ticket. Abort now!" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
               	    Write-Output "Icinga must be uninstalled" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                    $MSIArguments = @(
                	"/x"
                	$r.IdentifyingNumber
                	"/qn"
                	"/norestart"
                    )
					Start-Process "msiexec.exe" -ArgumentList $MSIArguments -Wait -NoNewWindow
                    exit
                }

                Write-Output "Generated ticket: " $cmdOutput.results.ticket | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                $ticket = $cmdOutput.results.ticket

                # 3 step: generate local certificates
                $parms = 'pki', 'new-cert', '--cn', "${myFQDN}", '--key', "${CertificatesPath}\${myFQDN}.key", '--cert', "${CertificatesPath}\${myFQDN}.crt"
                $cmdOutput = &$icinga2 @parms

                # Write-Output $cmdOutput | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber

                if (-not ($cmdOutput -match "Writing X509 certificate")) {
                    Write-Output "Cannot generate certificate. Abort now!" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
               	    Write-Output "Icinga must be uninstalled" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                    $MSIArguments = @(
                	"/x"
                	$r.IdentifyingNumber
                	"/qn"
                	"/norestart"
                    )
                    Start-Process "msiexec.exe" -ArgumentList $MSIArguments -Wait -NoNewWindow
                    exit
                }

                # 4 step: get trusted certificates
                $parms = 'pki', 'save-cert', '--host', "${sat_server}", '--port', '5665', '--trustedcert', "${CertificatesPath}\trusted-parent.crt"
                $cmdOutput = &$icinga2 @parms

                # Write-Output $cmdOutput | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber

                if (-not ($cmdOutput -match "Retrieving X.509 certificate")) {
                    Write-Output "Cannot retrieve parent certificate. Abort now!" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
               	    Write-Output "Icinga must be uninstalled" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                    $MSIArguments = @(
                	"/x"
                	$r.IdentifyingNumber
                	"/qn"
                	"/norestart"
                    )
                    Start-Process "msiexec.exe" -ArgumentList $MSIArguments -Wait -NoNewWindow
                    exit
                }

                # 5 step: node setup
                $parms = 'node', 'setup', '--parent_host', "${sat_server},5665", '--listen', '::,5665', '--cn', "${myFQDN}", '--zone', "${myFQDN}", '--parent_zone', """${parent_zone}""", '--trustedcert', "${CertificatesPath}\trusted-parent.crt", '--endpoint', "${sat_server},${sat_server}", '--ticket', "${ticket}", '--accept-config', '--accept-commands', '--disable-confd'
                Write-Output "Starting node setup with parms: " $parms | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                $cmdOutput = &$icinga2 @parms

                Write-Output $cmdOutput | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber

                if ($cmdOutput -match "Make sure to restart Icinga 2") {
                    Restart-Service -Name icinga2
                    &"sc.exe" config icinga2 obj= Localsystem
                    Start-Sleep -s 15
                    Restart-Service -Name icinga2
                    Write-Output "Icinga2 service restarted" | Out-File -FilePath "$workpath\icinga.log" -Append -NoClobber
                }
            }

            TestScript = {
                $dotNET = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full'
                $dotNET = $dotNET.Version.Split(".")
                $dotNET = $dotNET[0] + "." + $dotNET[1]
                if([float]$dotNET -gt '4.5'){
                    [string]$icinga2ver="2.11.9.123"
                    $Products = @()
                    #$Products += Get-ItemProperty "HKLM:\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\*" # 32 Bit
                    $Products += Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*"             # 64 Bit
                    $result = $Products | Where {($_.DisplayName -match 'Icinga 2') -and ($_.DisplayVersion -match $icinga2ver)}
                    if ($result -eq $null) {return $false} else {return $true}
                    return $false
                }
                else{
                    write-host "dotNET Version not compatible, at least 4.6"
                    return $true
                }
            }

            GetScript  = {
                $fileContent = $null
                return @{
                    Result = $fileContent
                }
            }
        }

        }

}

ServerConfig -OutputPath "C:\DSC\MOF_ICINGA"

This DSC will create the MOFs in C:\DSC\MOF_ICINGA. To apply them, use this command in an elevated prompt:

Start-DSCConfiguration -Path C:\DSC\MOF_ICINGA -Wait -Verbose -Force

TIP: Before executing the configuration on all the servers, try it first on a single target server using the parameter -Computername CNAME

When it tries to install Icinga it will generate a log file in C:\icinga on the target server. If the installation fails please check the log, fix the problem, and use the Start-DSCConfiguration command again until Icinga is installed.

If Icinga installation fails while executing the MOF, the agent will be uninstalled.

These Solutions are Engineered by Humans

Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth Phoenix.