Programming, Technology

Puppet Custom Types

Usually when creating custom types in puppet you will use templates to define a set of resources to manage. If you find yourself littering your template classes with exec statements then the likelihood is that you should probably consider creating a native custom type and directly extending the puppet language. This post is dedicated to just that as online documentation surrounding this topic is hazy at best, so I shall attempt to describe in layman’s terms what I discovered. Not only is this my first attempt at creating a custom type but also my first dalliance with Ruby, so I’m sure noob style errors will abound. Take it easy on me guys!

Background

So for this example I’m going to walk you through the development of my type to handle DNS resource records in BIND. Existing solutions out there typically just create zone files then don’t touch the contents again, assuming that the administrator will have made local changes which shouldn’t be randomly deleted. Some improvements are out there which use the concat library and templates to build the zone files completely under the control of puppet, but that doesn’t fit our requirements.

In our cloud, hardware provisioning is conducted with foreman, which manages detection of new machines via PXE, then provisioning which includes setting up PXE, DHCP and DNS. DNS in this case is handled via a foreman proxy running locally on the name server and performing dynamic zone updates with nsupdate. As you can no doubt appreciate using puppet resource records exclusively will destroy any dynamic updates performed by foreman. What we need is a custom type which creates resource records, which cannot be done in foreman (think CNAME and MX records), but in the same manner so they are not lost in the ether. So without further ado, creating a custom type to manage resource records via a canonical provider, in this case nsupdate.

Concepts

The first consideration when designing a custom type is how is it going to be used. I’m just going to dive straight in and show you a couple of examples of my interface.

dns_resource { 'melody.angel.net/A':
  ensure   => present,
  rdata    => '192.168.2.1',
  ttl      => '86400',
  provider => 'nsupdate',
}

dns_resource { '1.2.168.192.in-addr.arpa/PTR':
  nameserver => 'a.ns.angel.net',
  rdata      => 'melody.angel.net',
}
Title
This is an identifier that uniquely identifies the resource on the system. In this case it is an aggregate of the DNS record name and record type. My first attempt ignored this field entirely and used class parameters to identify the resource, but as we will see later on this is not the correct way of doing it
Properties
These are things like the TTL and relevant data class parameters. Upon identifying a resource or set of resources with the class title, properties are tangible things about the resource that can be observed and modified. In fact these are the fields that puppet will interrogate to determine whether a modification is necessary, which is an important distinction to make
Parameters
These are the remaining class parameters which have nothing to do with the resource being created, but inform puppet how to manage the resource. Parameters such as resource presence and the name server to operate on have no bearing on the resource itself as reported by DNS. These parameters will not be interrogated to determine whether a modification is necessary

Fairly straight forward, but easy to turn down a blind alley if not spelled out explicitly.

Module Layout

Before we delve into code let us first consider the architecture puppet uses to organise custom types. There are two layers we need to consider. At the top level is the actual type definition, which is responsible for defining how the type will manifest itself in your puppet code. Here you define the various properties and parameters which will be exposed, validation routines to sanitise the inputs, munging to translate inputs into canonical forms, default values and finally you can add in automatic requirements. To expand upon this last point a little here you can define a set of puppet resources that are a prerequisite for your type. Puppet will if these resources actually exist, add in dependency edges to the graph and ensure that the prerequisites are executed before your type. Admittedly I don’t like having to rummage through code to identify whether any implicit behaviour is forthcoming, however on this one occasion I will let it slide as it does remove a load of messy meta parameters from the puppet modules themselves.

The second layer is the provider which actually performs the actions to inspect and manage the resource. And here is the flexibility of puppet, you needn’t be limited to a single provider, in this example I’m creating an nsupdate provider but there is no reason why you cannot have a plain text zone file provider, or one for tinydns. These are runtime selectable with the provider class parameter, or are implicitly chosen by way of being the only provider, or based on facts. As an example the package builtin type will check which distro you are running and based on that use apt or yum etc.

Delving a little deeper into providers the general functionality is as follows. When puppet executes an instance of your type it will first ask the provider if the resource exists, if it doesn’t and is requested to be present then the provider will be asked to create it. Likewise if it exists and is requested absent then the provider will be asked to delete the resource. The final case is where the resource exists and is requested to be present. Puppet will inspect each property of the real resource defined by the type and compare with the requested values from the catalogue. If they differ then puppet will ask the provider to perform the necessary updates. Simple

Type code

Hopefully those concepts were straight forward and made clear sense. So lets look at how this all fits together. First lets look at the type definition which lives in the path <module>/puppet/type/<type>.rb

# lib/puppet/type/dns_resource.rb
#
# Typical Usage:
#
# dns_resource { 'melody.angel.net/A':
#   rdata => '192.168.2.1',
#   ttl   => '86400',
# }
#
# dns_resource { '1.2.168.192.in-addr.arpa/PTR':
#   nameserver => 'a.ns.angel.net',
#   rdata      => 'melody.angel.net',
# }
#
Puppet::Type.newtype(:dns_resource) do
  @doc = 'Type to manage DNS resource records'

  ensurable

  newparam(:name) do
    desc 'Unique identifier in the form "/"'
    validate do |value|
      unless value =~ /^[a-z0-9\-\.]+\/(A|PTR|CNAME)$/
        raise ArgumentError, 'dns_resource::name invalid'
      end
    end
  end

  newparam(:nameserver) do
    desc 'The DNS nameserver to alter, defaults to 127.0.0.1'
    defaultto '127.0.0.1'
    validate do |value|
      unless value =~ /^[a-z0-9\-\.]+$/
        raise ArgumentError, 'dns_resource::nameserver invalid'
      end
    end
  end

  newproperty(:rdata) do
    desc 'Relevant data e.g. IP address for an A record etc'
  end

  newproperty(:ttl) do
    desc 'The DNS record time to live, defaults to 1 day'
    defaultto '86400'
    validate do |value|
      unless value =~ /^\d+$/
        raise ArgumentError, "dns_resource::ttl invalid"
      end
    end
  end

  # nsupdate provider requires bind to be listening for
  # zone updates
  autorequire(:service) do
    'bind9'
  end

  # nsupdate provider requires nsupdate to be installed
  autorequire(:package) do
    'dnsutils'
  end

end

The first few lines are just boilerplate code, here you define the name of your type as it will appear in puppet code, and documentation, because everyone loves documentation right?

The ensurable method adds in support for the ensure parameter which shouldn’t come as too much of a surprise. What it also does is forces the creation of create, destroy and exists? methods by the provider.

The name parameter must be defined. desc allows documentation of your class parameters. Following that is our first encounter with parameter validation, which is basically checking for a hostname followed by a slash and one of our supported resource record types. Probably not the most RFC compliant regular expression but it works for now!

The nameserver parameter introduces default values so you don’t have to specify them in your puppet code, and the final thing I wish to draw attention to are the autorequires which add implicit dependencies to the graph as discussed previously and may reference any puppet resource.

Provider Code

Now for the guts of the operation, without further ado here are the contents of <module>/puppet/provider/<type>/<provider>.rb

# lib/puppet/provider/dns_resource/nsupdate

require 'resolv'

# Make sure all resource classes default to an execption
class Resolv::DNS::Resource
  def to_rdata
    raise ArgumentError, 'Resolv::DNS::Resource.to_rdata invoked'
  end
end

# A records need to convert from a binary string to dot decimal
class Resolv::DNS::Resource::IN::A
  def to_rdata
    ary = @address.address.unpack('CCCC')
    ary.map! { |x| x.to_s }
    ary.join('.')
  end
end

# PTR records merely return the fqdn
class Resolv::DNS::Resource::IN::PTR
  def to_rdata
    @name.to_s
  end
end

# CNAME records merely return the fqdn
class Resolv::DNS::Resource::IN::CNAME
  def to_rdata
    @name.to_s
  end
end

Puppet::Type.type(:dns_resource).provide(:nsupdate) do

  private

  # Run a command script through nsupdate
  def nsupdate(cmd)
    Open3.popen3('nsupdate -k /etc/bind/rndc-key') do |i, o, e, t|
      i.write(cmd)
      i.close_write
      raise RuntimeError, e.read unless t.value.success?
    end
  end

  public

  # Create a new DNS resource
  def create
    name, type = resource[:name].split('/')
    nameserver = resource[:nameserver]
    rdata = resource[:rdata]
    ttl = resource[:ttl]
    nsupdate("server #{nameserver}
              update add #{name}. #{ttl} #{type} #{rdata}
              send")
  end

  # Destroy an existing DNS resource
  def destroy
    name, type = resource[:name].split('/')
    nameserver = resource[:nameserver]
    nsupdate("server #{nameserver}
              update delete #{name}. #{type}
              send")
  end

  # Determine whether a DNS resource exists
  def exists?
    name, type = resource[:name].split('/')
    # Work out which type class we are fetching
    typeclass = nil
    case type
    when 'A'
      typeclass = Resolv::DNS::Resource::IN::A
    when 'PTR'
      typeclass = Resolv::DNS::Resource::IN::PTR
    when 'CNAME'
      typeclass = Resolv::DNS::Resource::IN::CNAME
    else
      raise ArgumentError, 'dns_resource::nsupdate.exists? invalid type'
    end
    # Create the resolver, pointing to the nameserver
    r = Resolv::DNS.new(:nameserver => resource[:nameserver])
    # Attempt the lookup via DNS
    begin
      @dnsres = r.getresource(name, typeclass)
    rescue Resolv::ResolvError
      return false
    end
    # The record exists!
    return true
  end

  def rdata
    @dnsres.to_rdata
  end

  def rdata=(val)
    create
  end

  def ttl
    @dnsres.ttl.to_s
  end

  def ttl=(val)
    create
  end

end

I’m using ruby’s builtin resover library to check the presence of a resource on the DNS server. The first 4 classes highlight one of the cool things about ruby, classes aren’t static. What we’re doing here is attaching new methods to the DNS resource types to marshal the relevant data into our canonical form i.e. a string, and also providing an exception case in a super class to catch when we add in support for a new resource type. It would have been easy to omit the override and just let things raise exceptions, but I like giving my peers useful debug.

Onto the main body of the provider. nsupdate unsurprisingly calls the same binary with an arbitrary set of commands. Usually you’d use puppet’s commands method to define external commands which enables a load of debug details but in this situation I needed access to standard in. create, destroy & exists? basically do just that, create, destroy and probe for the existence of a resource as defined by the name. The final four calls are accessors for the properties we defined earlier. You have to be careful with these as regards types as puppet will mismatch 86400 and “86400” and try to update the resource on each execution.

Conclusions

All in all, going from zero to hero in the space of 2 days wasn’t as daunting as I’d expected, new language, new framework. Hopefully I’ve summarised my experiences in a way which my readers will be able to easily digest. On reflection the whole expansion of puppet has been breathtakingly easy, and I’m hoping it will provide some inspiration to better improve our orchestration and provisioning efforts. And hopefully yours too! Until next time.