CNK's Blog

Test-Driven Infrastructure With Chef

If you are using code to set up your infrastructure, then that code demands as good or better software development practices as your application code. One of the reasons I wanted to try Chef is because I knew that people were doing automated testing of chef configurations. (The puppet community may be doing similar kinds of testing; I don’t know.) There is even a book about TDD and Chef: Test-Driven Infrastructure With Chef (and it’s even in its second edition). So I am going to try working through it. The first several chapters are background or things I have sort of done already. But they start to use Opscode’s Hosted Chef and Vagrant in chapter 4.

Setting Up a New Playground

When I did the Quick Start exercises, I downloaded the starter kit which included a .chef directory with a knife configuration file and two .pem files: ckiser.pem and ckiser-validator.pem. I explored downloading the starter kit again - but it wanted to reset my keys which would have made my example stuff not work any more. It took me a little while to figure out the instrutions on mean that one or the other file is read - not that one is read and then the second file is read, overwriting configuration variables. I was hoping to put shared information in ~/.chef/knife.rb and then project specific information in a knife.rb in the project directory. From my ‘puts’ statement, only one of the files is read - either the one in the current directory or the one in ~/.chef/.

I can still get part of what I want - reuse of my keys - but moving them to ~/.chef/ and then editing my knife.rb file to look for them there. And then I created a new directory for working the TDI Chef exercises, ~/chef-tdd/. I copied the knife.rb from the starter kit into ~/.chef/knife.rb. I also created a cookbooks directory and copied the chefignore file from the starter kit in there. And added the extensive .gitignore from the starter kit. There are some other things I think I will probably need but am going to wait until something I am doing uses them before mving them here.


The last part of chapter 4 is about installing Vagrant and using it to create virtual machines. Mostly this is just like what I have been doing for a while now - but it linked me to a very promising source for base boxes, Opscode’s Bento Boxes One thing that makes them especially attractive is that the don’t come with Chef preinstalled. Instead they recommend installing the most current Chef using the vagrant-omnibus plugin. From the TDI book:

The plug-in we installed into Vagrant (vagrant plugin install vagrant-omnibus) works with Vagrant boxes that do not have Chef installed, and adds a hook to vagrant up to install Chef using the omnibus package, just as we did in “Exercise 1: Install Chef ” on page 47. This helps keep the Vagrant box slim and as close to upstream as possible, and does not require a fleet of Vagrant boxes to be created with every Chef patch release.

I want to test with a box that matches my Linode VPS as closely as possible so despite the fact I already have an Opscode Ubuntu 12.04 in 32 bit, I downloaded and added the 64 bit version:

    vagrant box add opscode-ubuntu-12.04 \

    [03:31 PM] (brazen:~/chef-tdd) $ vagrant plugin install vagrant-omnibus
    Installing the 'vagrant-omnibus' plugin. This can take a few minutes...
    Installed the plugin 'vagrant-omnibus (1.2.1)'!

Then in ‘~/chef-tdd/ I did vagrant init opscode-ubuntu-12.04` and then edited the Vagrant file to have the following:

    Vagrant.configure("2") do |config| = "opscode-ubuntu-12.04"
      config.vm.box_url = ""
      config.omnibus.chef_version = :latest
      config.vm.hostname = "vagrant-ubuntu-12-04" :forwarded_port, guest: 80, host: 8080

Then I did vagrant up. But I didn’t see the lines I expected regarding installing chef. I tried destroying the VM and then tring vagrant up again - same result. I do have warnings about the Guest Additions not matching the version of VirtualBox that the Opscode base boxes were built against:

    [default] Waiting for VM to boot. This can take a few minutes.
    [default] VM booted and ready for use!
    [default] The guest additions on this VM do not match the installed
    version of VirtualBox! In most cases this is fine, but in rare cases
    it can cause things such as shared folders to not work properly. If
    you see shared folder errors, please update the guest additions within
    the virtual machine and reload your VM.

    Guest Additions Version: 4.3.2
    VirtualBox Version: 4.2
    [default] Setting hostname...

So I downloaded the latest version of VirtualBox, 4.3.6, and tried again. That wasn’t so great. I got the following error message:

    $ vagrant up
    Vagrant has detected that you have a version of VirtualBox installed
    that is not supported. Please install one of the supported versions
    listed below to use Vagrant:
    4.0, 4.1, 4.2

I currently have Vagrant 1.2.2 installed. Looking at, it looks like the latest is 1.4.3. Let’s see if upgrading Vagrant fixes this. I downloaded the latest .dmg file and ran the installer. Then my next attempt at vagrant up gave me:

    The following plugins were installed with a version of Vagrant that
    had different versions of underlying components. Because these
    component versions were changed (which rarely happens), the plugins
    must be uninstalled and reinstalled.

    To ensure that all the dependencies are properly updated as well it is
    _highly recommended_ to do a `vagrant plugin uninstall` prior to

    This message will not go away until all the plugins below are either
    uninstalled or uninstalled then reinstalled.

    The plugins below will not be loaded until they're uninstalled and

    vagrant-berkshelf, vagrant-omnibus

I unstalled both plugins. I don’t remember installing vagrant-berkshelf so, for now, I am not going to reinstall it. I only reinstalled vagrant-omnibus (1.2.1). But still no joy: no chef-client or knife when I log into the box. ARRRGGGGGG because I was editing the wrong Vagrantfile! I had put the line about vagrant-omnibus into the file in chef-repo that I had opened for reference. OK let’s destroy and recreate the VM. And now, we get chef:

    04:20 PM] (brazen:~/chef-tdd) $ vagrant up
    Bringing machine 'default' up with 'virtualbox' provider...
    [default] Importing base box 'opscode-ubuntu-12.04'...
    [default] Matching MAC address for NAT networking...
    [default] Setting the name of the VM...
    [default] Clearing any previously set forwarded ports...
    [default] Clearing any previously set network interfaces...
    [default] Preparing network interfaces based on configuration...
    [default] Forwarding ports...
    [default] -- 22 => 2222 (adapter 1)
    [default] -- 80 => 8080 (adapter 1)
    [default] Booting VM...
    [default] Waiting for machine to boot. This may take a few minutes...
    [default] Machine booted and ready!
    [default] Setting hostname...
    [default] Mounting shared folders...
    [default] -- /vagrant
    [default] Installing Chef 11.8.2 Omnibus package...
    [default] Downloading Chef 11.8.2 for ubuntu...
    [default] downloading
    [default]   to file /tmp/
    [default] trying wget...
    [default] url
    md5     3d3b3662830a44eeec71aadc098a4018
    sha256  a5b00a24e68e29a01c7ab9de5cdaf0cc9fd1c889599ad9af70293e5b4de8615c
    [default] downloaded metadata file looks valid...
    [default] downloading
    [default]   to file /tmp/
    [default] trying wget...
    [default] Checksum compare with sha256sum succeeded.
    [default] Installing Chef 11.8.2
    [default] installing with dpkg...
    [default] Selecting previously unselected package chef.
    [default] (Reading database ...
    [default] 54659 files and directories currently installed.)
    [default] Unpacking chef (from .../chef_11.8.2_amd64.deb) ...
    [default] Setting up chef (11.8.2-1.ubuntu.12.04) ...
    [default] Thank you for installing Chef!
    [04:24 PM] (brazen:~/chef-tdd) $ vagrant ssh
    Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.8.0-29-generic x86_64)

     * Documentation:
    Last login: Tue Nov 26 11:27:55 2013 from
    vagrant@vagrant-ubuntu-12-04:~$ chef-client --version
    Chef: 11.8.2

Ch 7.2: Berkshelf

Much of configuration management (and really any software development) is dependency management. Throughout the TDI book we were managing that manually - downloading cookbooks, checking dependencies, downloading more cookbooks. Finally in chapter 7 the author introduces Berkshelf to do the managing for us. So now I am going to need the vagrant-berkshelf plugin that I delayed reinstalling in the section above. vagrant plugin install vagrant-berkshelf. And then the book told me to run berks configure. This creates a default configuration file in ~/.berkshelf/config.json. Not really sure what the implications of some of the items in this are. I’ll have to see as they arise.

Since I didn’t make the stand alone IRC cookbook from chapter 3, I can’t do the berks init stuff. And the discussion of Berkshelf kind of trailed off promising more information about preferred workflows later in the chapter.

Ch 7.3: Application Cookbooks and Test Kitchen

What are application cookbooks and why should we use them?

The application cookbook pattern is characterized by having decided the top-level service that we provide and creating a cookbook for that service. That cookbook wraps all the dependent services that are needed to deliver the top-level service. … This looks a lot like the kind of thing that might be accomplished using a Chef role, but has some significant advantages. First of all, cookbooks can be explicitly versioned and tracked in a way that roles can’t.

If there is a need to alter the behavior of an upstream cookbook, attributes can be set in a recipe, and if functionality needs to be added, tested, or tweaked, this can be achieved by wrapping upstream cookbooks in a manner that looks much like object inheritance. This has the twin advantages again of being testable, but also of avoiding constant fork‐ ing of upstream cookbooks.

The test harness tool of choice - at least of this author - is Test Kitchen. I installed it with gem install test-kitchen. The tests themselves are created using a combination of cucumber, rspec, and leibniz (a library written by the author).

Things were going OK until about page 200 where there were some missing steps - such as where are we putting our specs and how did we get our spec_helper - and for that matter, where did we get rspec since it wasn’t on the list of gems we were told to add to our Gemfile. The errata page has a complaint from someone else who noticed this - but not resolution. I spent some time wandering around, read, asked some questions on the #chef IRC channel. And then with some perspective, went back to the book to see if I could reverse engineer stuff using the code listings.

First, if we are going to run rspec -fd, we need rspec installed. So I added it to the Gemfile and did bundle install. Now I can run rspec --init and that created a spec directory and put a spec_helper.rb file in there. I am kind of unclear about where to put the integration test at the bottom of p 202 so I skipped them and started with the unit test at the top of p 204. I typed in the ‘let’ and the first ‘it’ and then ran the spec. It complained uninitialized constant ChefSpec so I added ‘chefspec’ to my Gemfile and a “require ‘chefspec’” to the top of my default_spec.rb. Then it complained it couldn’t find chef. I know I have chef - but it wasn’t in my Gemfile. So I added it to my Gemfile and reran ‘bundle’. Now I am getting an odd error about about Chef - and a really old version of chef. How did that get in here? And how did I end up with chefspec 0.0.1?

    (brazen:~/chef-tdd/cookbooks/cnk-blog) $ rspec
    /Users/cnk/.rvm/gems/ruby-2.0.0-p353@chef/gems/chef-0.8.10/lib/chef/provider/package/dpkg.rb:26:in `<class:Package>': uninitialized constant Chef::Provider::Package::Apt (NameError)
        from /Users/cnk/.rvm/gems/ruby-2.0.0-p353@chef/gems/chef-0.8.10/lib/chef/provider/package/dpkg.rb:25:in `<class:Provider>'

I created a completely new rvm gemset, removed my Gemfile.lock and did a completely new bundle install. This time I only have chef 11.8.2 along with chef-spec 3.2.0 But even after fiddling with how I required chefspec, I am still getting complaints abotu uninitialized constant ChefSpec::ChefRunner. So I am going to give up on the example in Chapter 7 “Acceptance Testing: Cucumber and Liebniz”.

Ch 7.5: Integration Testing: Test Kitchen with Serverspec

The next section on integration testing does a better job of showing us the steps. But introduces one more layer - using Test Kitchen to manage your Vagrant boxes. It is sort of nice because it gives you one file that declares which OSs you are going to test your cookbook on - and takes care of setting up what one needs to do that. But it is one additional layer AND it means I am swimming in Vagrantfiles - most of which are not used. I had thought I would test my full configuration on a local Vagrant instance, so I created a Vagrantfile at the top of my chef-tdd directory. When I did berks cookbook cnk-blog, that automatically created a Vagrant file. And now when I edited the .kitchen.ym file (which berks also created for me) and ran kitchen create all, that created a Vagrantfile in .kitchen/kitchen-vagrant/default-ubuntu-1204/ I think it is this last Vagrantfile that I am actually seeing when I do kitchen list - since I can’t ssh in when I am in ~/chef-tdd/cookbooks/cnk-blog/ but I can if I am in ~/chef-tdd/cookbooks/cnk-blog/.kitchen/kitchen-vagrant/default-ubuntu-1204

My cookbook doesn’t yet have a run list, but if I run kitchen converge anyway, that installs ruby and chef from the Opscode Omnibus installer. So now, where do I put the tests. I don’t think I want to use BATS unless I have to - I am better at Ruby than bash. So I read that part but then started working through the Serverspec exercises starting at p 241. I created a test file in the magic place (test kitchen figures out it needs busser and then what kinds of tests to run based on the directory names).

    $ mkdir -p test/integration/default/serverspec/localhost
    # create file cnk-blog_spec.rb
    $ cat  test/integration/default/serverspec/localhost/cnk-blog_spec.rb
    require 'spec_helper'

    describe "Cynthia's static html blog site" do
      it 'should have installed apache' do
        expect(package apache2).to be_installed

    $ kitchen verify
    -----> Starting Kitchen (v1.1.1)
    -----> Setting up <default-ubuntu-1204>...
    Fetching: thor-0.18.1.gem (100%)
    Fetching: busser-0.6.0.gem (100%)
    Successfully installed thor-0.18.1
    Successfully installed busser-0.6.0
    2 gems installed
    -----> Setting up Busser
           Creating BUSSER_ROOT in /tmp/busser
           Creating busser binstub
           Plugin serverspec installed (version 0.2.6)
    -----> Running postinstall for serverspec plugin
           Finished setting up <default-ubuntu-1204> (0m22.57s).
    -----> Verifying <default-ubuntu-1204>...
           Suite path directory /tmp/busser/suites does not exist, skipping.
    Uploading /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb (mode=0664)
    -----> Running serverspec test suite
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serversp
    ec/localhost/cnk-blog_spec.rb --color --format documentation
    /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- spec_helper (LoadError)
            from /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
            from /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:1:in `<top (required)>'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `block in load_spec_files'

            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `each'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load_spec_files'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/command_line.rb:22:in `run'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:80:in `run'
            from /tmp/busser/gems/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:17:in `block in autorun'
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation failed
    Ruby Script[/tmp/busser/gems/gems/busser-serverspec-0.2.6/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/busser/suites/serverspec] exit code was 1
    >>>>>> Verify failed on instance <default-ubuntu-1204>.
    >>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
    >>>>>> ------Exception-------
    >>>>>> Class: Kitchen::ActionFailed
    >>>>>> Message: SSH exited (1) for command: [sh -c 'BUSSER_ROOT="/tmp/busser" GEM_HOME="/tmp/busser/gems" GEM_PATH="/tmp/busser/gems" GEM_CACHE="/tmp/busser/gems/cache" ; export BUSSER_ROOT GEM_HOME GEM_PATH GEM_CACHE; sudo -E /tmp/busser/bin/busser test']
    >>>>>> ----------------------

Oops. I didn’t read carefully and missed creating the helper file in cnk-blog/test/integration/default/serverspec/spec_helper.rb

    require 'serverspec'
    require 'pathname'
    include Serverspec::Helper::Exec
    include Serverspec::Helper::DetectOS

    RSpec.configure do |c|
      c.before :all do
        c.os = backend(Serverspec::Commands::Base).check_os

Now I get properly failing tests - that is failing because I have not written the cookbook code to make them work.

    $ kitchen verify
    -----> Starting Kitchen (v1.1.1)
    -----> Verifying <default-ubuntu-1204>...
           Removing /tmp/busser/suites/serverspec
    Uploading /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb (mode=0664)
    Uploading /tmp/busser/suites/serverspec/spec_helper.rb (mode=0664)
    -----> Running serverspec test suite
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation

    Cynthia's static html blog site
    No packages found matching apache2.
      should have installed apache (FAILED - 1)
      have enabled the apache service (FAILED - 2)
    httpd: unrecognized service
      be running the apache service (FAILED - 3)
      should listen on port 80 (FAILED - 4)
      should have a virtual host for cnk-blog


      1) Cynthia's static html blog site should have installed apache
         Failure/Error: expect(package 'apache2').to be_installed
           dpkg-query -f '${Status}' -W apache2 | grep '^install ok installed$'
           expected Package "apache2" to be installed
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:5:in `block (2 levels) in <top (required)>'

      2) Cynthia's static html blog site have enabled the apache service
         Failure/Error: expect(service 'httpd').to be_enabled
           ls /etc/rc3.d/ | grep -- '^S..httpd' || grep 'start on' /etc/init/httpd.conf
           grep: /etc/init/httpd.conf: No such file or directory

           expected Service "httpd" to be enabled
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:9:in `block (2 levels) in <top (required)>'

      3) Cynthia's static html blog site be running the apache service
         Failure/Error: expect(service 'httpd').to be_running
           ps aux | grep -w -- httpd | grep -qv grep
           expected Service "httpd" to be running
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:13:in `block (2 levels) in <top (required)>'

      4) Cynthia's static html blog site should listen on port 80
         Failure/Error: expect(port 80).to be_listening
           netstat -tunl | grep -- :80\
           expected Port "80" to be listening
         # /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:17:in `block (2 levels) in <top (required)>'

    Finished in 0.06817 seconds
    5 examples, 4 failures

    Failed examples:

    rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:4 # Cynthia's static html blog site should have installed apache
    rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:8 # Cynthia's static html blog site have enabled the apache service
           rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:12 # Cynthia's static html blog site be running the apache service
           rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb:16 # Cynthia's static html blog site should listen on port 80
    /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -S /opt/chef/embedded/bin/rspec /tmp/busser/suites/serverspec/localhost/cnk-blog_spec.rb --color --format documentation failed
    Ruby Script[/tmp/busser/gems/gems/busser-serverspec-0.2.6/lib/busser/runner_plugin/../serverspec/runner.rb /tmp/busser/suites/serverspec] exit code was 1
    >>>>>> Verify failed on instance <default-ubuntu-1204>.
    >>>>>> Please see .kitchen/logs/default-ubuntu-1204.log for more details
    >>>>>> ------Exception-------
    >>>>>> Class: Kitchen::ActionFailed
    >>>>>> Message: SSH exited (1) for command: [sh -c 'BUSSER_ROOT="/tmp/busser" GEM_HOME="/tmp/busser/gems" GEM_PATH="/tmp/busser/gems" GEM_CACHE="/tmp/busser/gems/cache" ; export BUSSER_ROOT GEM_HOME GEM_PATH GEM_CACHE; sudo -E /tmp/busser/bin/busser test']
    >>>>>> ----------------------

For Future Reference

  1. A tutorial in the form of a git repo: RallySoftware-cookbooks/chef-tutorials

  2. An excellent blog post