Have you ever been in the scenario where you need puppet to manage something dynamic, such as a list of pool members in HAProxy or a list of backend servers in Varnish?

Puppet created exported resources via storeconfigs to handle exactly that. On one node you publish a resource and from another node you would subscribe to it. This could be as simple as publishing the FQDN along with an IP address of an application server, to then be picked up by HAProxy as a pool member.

However an issue arises when you are provisioning new servers so quickly that the default Puppet runinterval of 30 minutes won’t suffice. You need this increase of capacity now.

So in this post we are going to go over at a high level a couple of different ways to create callbacks that will trigger Puppet run’s using MCollective as the orchestration piece.

Simple

I have to say it, the easiest way would be to decrease the runinterval of the Puppet Agent. But that’s neither fun nor an efficient use of your servers resources. There are tool’s such as MCollective that allow you to kick puppet when you need it to run, rather rely on it to run at some interval. Also using MCollective allow you to ensure the Puppet Agent hasn’t been disabled manually.

AWS - SNS/SQS

The first approach is rather AWS specific and it’s with instances in an auto scaling group.

There are two types of notifications you can take advantage of:

  • autoscaling:EC2_INSTANCE_LAUNCH
  • autoscaling:EC2_INSTANCE_TERMINATE

These notifications will get sent to an SNS Topic. SQS should be setup to subscribe to this topic. You can then create a simple application that will poll SQS and take action.

The work flow in this example would be: ASG Notification -> SNS -> SQS -> Poller

The code snippet below requires the ruby gem “aws-sdk” 

sqs = AWS::SQS.new(
    :access_key_id     => 'access_key',
    :secret_access_key => 'access_secret'
)
sqs.queues[url].poll do |receive|
	if receive
		# 1. Parse the message
		# 2. What's the event
		# 3. Get to work
	end
end

You can read more about auto scaling notifications here

This works great if you are only in Amazon but breaks down if are hosting with multiple vendors and who wants to manage more than one process when the end goal is the same.

Puppet Report Processor

I colleague of mine shared this article with me, written by R.I. Pienaar (awesome guy). It discusses using the Puppet report processor as a callback to trigger Puppet run’s via MCollective of course.

You can read more about the puppet report processor here.

I like that this approach is vendor agnostic, extends something that’s built right into Puppet, and is rather simple to use.

PuppetDB

PuppetDB is an awesome tool that will collect data generated by Puppet. With every new feature, it’s becoming a necessity to deploy in any environment that is using Puppet. This is evident by the fact that it’s now included with Puppet Enterprise as of version 3.0.

PuppetDB receives a complete catalog and set of facts for each distinct node that’s associated with the Puppet Master. It’s also being queried for exported resources during catalog compilation and it even has an experimental report/event functionality.

I’m toying with the idea of having PuppetDB send out simple notifications to various upstream systems, similar to AWS notifications. Upon a Puppet Agent’s first check in you could notify MCollective to kick a subset of servers. If you are using the node-ttl you could perform another kick or do some clean up of the Puppet Agent’s cert or call a rake command to remove the server from the Puppet Dashboard.

PuppetDB also has an extensive API that could be harnessed to gather this exact kind of information.

MCollective

If you are already using MCollective in you infrastructure, you could also use it to detect when machines come online and offline (take into account network partitions). I plan on exploring this case in another post.