Programming, Technology

Dynamic Nagios Host Groups With Puppet

So this little problem caused me some headaches. Coming from a C/C++ systems programming background, Puppet takes some getting used to and this assignment was a learning experience and a half!

The setup we wanted was to have Puppet define a set of host groups on our icinga server, then have every host export a nagios host resource which selected host groups it was a member of based on which classes were assigned to it by the ENC. The little database experience I have suggested that this way was best as it avoided the sprawl of host groups and services being redefined on a per host basis. A lot of other blogs suggest this is the way to go, but allude to the fact that they have a node variable which controls membership, and doesn’t provide the dynamism that we wanted.

Plan Of Attack

With puppet being declarative, using global or class variables to flag which host groups to include wont work as you’re at the mercy of compilation order, which by its very nature is non-deterministic. It gets worse with global variables when using fat storeconfigs, as a definition on one host seems to get propagated to all the others, and thin configs don’t export variables. Placing the exported nagios host definitions in a post-main stage suffered from the scope being reset.

The next idea was to use the ENC, foreman in our case, to generate the host groups. Problem here was our foreman host groups refer to hardware platforms, whereas our nagios host groups refer to software groups. And defining per host host-group membership isn’t going to cut it!

And then there was the eureka moment. Facter 1.7 supports arbitrary facts being generated from files in /etc/facter/facts.d. Facts are available at compilation time, so regardless of ordering they are always available. Better still we can generate them dynamically per host based on the selected profiles and collect them using an ERB template. And here’s how…

Dynamic Nagios Host Groups In Puppet

First piece of the puzzle is to define a module to allow easy generation of custom facts, first by creating the directory structure

class facter::config {
  File {
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0755',
  }

  file { '/etc/facter': } ->
  file { '/etc/facter/facts.d':
    recurse => true,
    purge   => true,
  }
}

Then by creating the reusable definition

class facter {
  define fact ( $value = true ) {
    file { "/etc/facter/facts.d/${title}.txt":
      ensure  => file,
      owner   => 'root',
      group   => 'root',
      mode    => '0644',
      content => "${title}=${value}",
      require => Class['facter::config'],
  }
}

Next up we define a set of virtual host groups. The idea being we can realise them in multiple locations based on profile and not worry about collisions. Think multiple profiles having an apache vhost, rather than fork the community apache sub-module we can just attach at the profile level.

class icinga::hostgroups {
  include facter
  @facter::fact { 'hg_http': }
  @facter::fact { 'hg_ntp': }
}

Then to finally create the facts on the host system something like the following will suffice

class profile::http_server {
  include icinga::hostgroups
  realize Facter::Fact['hg_http']
}

The final piece of the jigsaw is the host definition itself

class icinga::client {
  @@nagios_host { $::hostname:
    ensure     => present,
    alias      => $::fqdn,
    address    => $::ipaddress,
    hostgroups => template('icinga/hostgroups.erb'),
  }
}

And the ERB template to gather the facts we’ve exported

hg_generic<% -%>
<% scope.to_hash.keys.each do |k| -%>
<% if k =~ /(hg_[\w\d_]+)/ -%>
<%= ',' + $1 -%>
<% end -%>
<% end -%>

And there you have it. I’m by no means an expert at either puppet or ruby so feel free to suggest better ways of achieving the same end result. Note this isn’t production code, just off the top of my head, so there may be some mistakes, but you get the gist. Happy monitoring!