Reification

When working with puppet, eventually you’ll arrive at any one of several conclusions; you have a mass of code and want to refactor it, you have a cool idea that you want to prototype, or you want to add some code but don’t want to push it to production until it’s done. Puppet’s answer to this is environments. The documentation says that you can have arbitrary environments and redefine your configuration parameters in puppet.conf based on which environment you wish your clients to use. The general idea is to have several environments like “Production,” “Development,” and “Testing.”

Now, for me this falls down when I want to break away from more standard development cycles that no longer fit in the dev->test->prod workflow. Or if I have groups of developers working on disjointed projects but don’t want to support completely separate puppetmasters with their own clients. Or when I want to follow git’s mentality of “branching is cheap; branch often.”

I’d like to be able to say `git checkout -b newfeature`, add some resources and templates, `git push`, and instantly use it on my clients. No reconfiguration of the puppetmaster needed. When I finish, `git checkout production ; git merge newfeature ; git branch -d newfeature` will put all aright again.

In puppet.conf I can use $environment to reference the current environment for setting modulepath and manifest, but I don’t actually need the [newfeature] environment declaration section in the puppet.conf. This allows me to create a config thus:

[master]
  environment = production
  manifest    = $confdir/environments/$environment/manifests/site.pp
  modulepath  = $confdir/environments/$environment/modules

[agent]
  environment = production

This says that the master will base the manifest and module path on the environment, which is passed by the agent to the master. If I changed the [agent] section on a puppet agent to have environment = newfeature instead, then the catalog compiled by the master would be from /etc/puppet/environments/newfeature. If an agent does not pass the environment variable then the default production will be used. (Big Note is that your files and templates really only work cross-environment if you’re using modules. You can either clone all of your modules per environment, or you can have some modules be per environment and some be global. Check out Volcane’s blogpost on using your modules in this manner.)

Now that I have puppet set up to take arbitrary environments, lets make this useful. To implement it so that git branches are recognized as separate environments, I have to keep your puppet manifests and modules in git. (You’re already doing that, right?) Starting without any environments, I’ll rearrage my puppet configs to look like this:

/etc/puppet/
     | puppet.conf
     | fileserver.conf
     - environments/
       - production/
         + manifests/
         + modules/

In the production directory I `git init ; git add .` (then considering that I have a gitosis instance set up and an ssh key ready to be used by the puppet user,) `git remote add origin git@gitosis.host:puppet-environments.git ; git push origin master`. Now for the git hook I go open puppet-environments.git/hooks/post-receive and put this bourne code in, configured with my git repo url and the key by which my puppet master can pull from gitosis:

#!/bin/sh
read oldrev newrev refname

REPO="git@gitosis.host:puppet-environments.git"
BRANCH=`echo $refname | sed -n 's/^refs\/heads\///p'`
BRANCH_DIR="/etc/puppet/environments"
SSH_ARGS="-i /var/lib/puppet/.ssh/id_rsa"
SSH_DEST="puppet@puppetmaster.host"

if [ "$newrev" -eq 0 ] 2> /dev/null ; then
  # branch is being deleted
  echo "Deleting remote branch $BRANCH_DIR/$BRANCH"
  ssh $SSH_ARGS $SSH_DEST /bin/sh <<-EOF
    cd $BRANCH_DIR && rm -rf $BRANCH
EOF
else
  # branch is being updated
  echo "Updating remote branch $BRANCH_DIR/$BRANCH"
  ssh  $SSH_ARGS $SSH_DEST /bin/sh <<-EOF
    { cd $BRANCH_DIR/$BRANCH && git pull origin $BRANCH ; } \
    || { mkdir -p $BRANCH_DIR && cd $BRANCH_DIR \
         && git clone $REPO $BRANCH && cd $BRANCH \
         && git checkout -b $BRANCH origin/$BRANCH ; }
EOF
fi

Now after I’ve cloned puppet-environment.git to my laptop for hacking, I can `git checkout -b newfeature` and end up with an exact representation of my production code, but free to be hacked on until it’s ready to be pushed and tested. Pushing will instantiate it in a new environment. After I’ve pushed, I can run `puppet agent --test --environment newfeature` on a puppet client and get a one-time run inside your newfeature environment, and it will revert back to production after that run.

Finally, what if I want my hosts to remember which --environment I last ran on it? Or how about being able to configure the nodes’ environment right from puppet? For that I can make a small module with a parameterised class called “environment.” Here’s my init.pp:

class environment($env = $environment) {
  file { '/etc/puppet/puppet.conf':
    owner   => 'puppet',
    group   => 'puppet',
    mode    => '644',
    content => template("environment/puppet.conf.erb"),
  }
}

As the final step, I copy my clients’ puppet.conf to modules/environment/files/puppet.conf.erb and replace the environment = production section in [agent] with environment = <%= env %>.

Now I can add include environment in my base node or any node that you want to “remember” it’s environment given via --environment. If I’d like to configure a node to always use a specific environment then I can instead use class { environment: env => "production" } and it will always revert to the production environment.

Thanks to Marut and the guys at PSU for the inception!