Testing Opscode’s Apache2 Cookbook

Opscode’s apache2
cookbook is commonly used as an example for reference material because
Apache HTTPD server is fairly ubiquitous. Many, if not most, web
operations teams currently use it, or have used it in their
application stack.

While our apache2 cookbook is released to
the Chef Community site,
the “test” directory is not uploaded to the site when using
the knife cookbook site install command, due to the way
the cookbook upload API works. However, the cookbook is available as
a Git
repository in the “opscode-cookbooks” organization.

[sourcecode]
git clone git://github.com/opscode-cookbooks/apache2.git
[/sourcecode]

In this post, we’re going to give a brief overview of Opscode’s
apache2 cookbook. Then, we’re going to describe some of the tests that
have been added and updated in the cookbook and how they’re used in
the context of testing
with Test Kitchen.
Finally, we’ll wrap up with information on how contributors can add
tests to the cookbook for new contributions, and what we’re working on
for other cookbooks.

Cookbook Overview

The apache2 cookbook supports several platforms. We do this by
using the style of configuration that Debian uses for the apache2
package. We feel that the Debian way lends itself well to automation
through configuration management. The cookbook
has definitions
to facilitate automating this. The default recipe of the cookbook
handles the basic setup and configuration of the package, service and
common configuration. It also ads the scripts used by the definitions
for managing virtual hosts and modules. Several built in Apache HTTPD
modules are configurable in the cookbook through module-specific
recipes. Many additional modules are also supported by installing
their package or installing them from source, and then dropping off
the configuration. These work on a wide variety of platforms (nine per
the metadata “supports” list). However, not all platforms are created
equally – different package and service names, different compile-time
flags and available modules, etc. However, because “families” of
platforms have enough similarities, such as Debian and Ubuntu, or Red
Hat and its derivatives like CentOS, we use Ohai’s “platform_family”
attribute to reduce the complexity in writing and testing the
cookbook.

Testing the Cookbook

Back in July this year, we featured a guest post
from Andrew
Crump on testing infrastructure. Andrew did a lot of work
on Test Kitchen,
and
added initial
tests to our apache2 cookbook. He did a great job with this, and
this post is going to explore those in more detail. First of all, the
tests described here are run through Test Kitchen, automatically,
across four of the cookbook’s supported platforms:

• Ubuntu 10.04, 12.04
• CentOS 5.8, 6.3

If you have cloned
the git
repository, you can install test-kitchen and dependencies by
cd’ing into the directory, running bundle install, and then running
test-kitchen.

[sourcecode language="bash"]
cd apache2
bundle install
kitchen test
[/sourcecode]

Note that test-kitchen is not a subsitute for running the cookbook’s
recipes and its related components in your own infrastructure on
actual development/testing/staging systems. It also does not use
why-run
mode, which you may wish to use prior to deploying changes on your
systems as an additional level of testing for confidence.

Configurations

Through the “Kitchenfile,” Test Kitchen allows you to specify multiple
configurations. Each configuration in a cookbook project represents a
set of tests that should be run. A configuration may have a recipe in
the “test” cookbook, and it may have a minitest-chef test.

In the apache2 cookbook, there are several configurations.

• default: for testing the default recipe including its minitests
• basic_web_app: for setting up the “basic web app” feature tests
• modules: runs the “mod_” recipes in the apache2 cookbook to enable
  simple modules
• mod_*: all the “mod_” configurations are for testing the various
  Apache module recipes that are more complex than simply enabling a module.

apache2_test Cookbook

When using configurations, you can create a “test” cookbook. It should
be named after the project being tested with “_test” appended. In this
case of the “apache2” cookbook, the test cookbook is named
“apache2_test.” The cookbook was created in the
test/kitchen/cookbooks directory using this command:

[sourcecode language="bash"]
knife cookbook create -o test/kitchen/cookbooks apache2_test
[/sourcecode]

The test cookbook was edited like any other cookbook. Its primary
purpose is to ensure that the node is set up to run the minitest-chef
tests, and/or cucumber features. Test cookbooks can be used for other
purposes too, such as recipes that use definitions and LWRPs to verify
that they work w/o exceptions. Some of the recipes do this, too. Each
recipe in the cookbook corresponds to a configuration in the
Kitchenfile, and should take care of anything so the node is
configured for the tests to run.

For example, the “apache2_test::default” recipe includes the
“apache2::default” recipe. Then, when the “default_test” minitest-chef
tests run, we know for that configuration that the node is setup.

Another example is the “apache2_test::modules” recipe, which is used
for the “modules” configuration so that the “modules” minitest-chef
tests are ready to run. This particular recipe has an array of module
names that are simply enabling the module.

Most of the modules in the apache2 cookbook merely enable the module.
Others set up the node to provide some kind of application framework,
such as “apache2::mod_python” or “apache2::mod_php5“. The
configurations for these modules need some additional setup done. This
is completed by the corresponding apache2_test recipe.

Most of the minitest-chef tests for the apache2 cookbook are in its
files/default/tests/minitest directory. However, two tests are in
the apache2_test directory. This is because of the way that
test-kitchen builds the run list, the apache2 cookbook
doesn’t have corresponding recipes. These are:

• basic_web_app
• modules

They are located in the
test/kitchen/cookbooks/apache2_test/files/default/tests/minitest
directory instead. The helpers.rb is symlinked to this directory
under support.

Minitest Chef

As mentioned previously, with two noted exceptions, the minitest-chef
tests are located in the files/default/tests/minitest directory of
the apache2 cookbook. We use minitest-chef-handler
RubyGem with
minitest-handler
cookbook to
automatically run tests on the node after the specified configuration
has been converged with test-kitchen.

Additionally, some helper methods are in the support directory for
minitest-chef to use. This should be required in any tests used for
the cookbook.

[sourcecode language="ruby"]
require File.expand_path('../support/helpers', __FILE__)

describe "apache2::RECIPENAME" do
  include Helpers::Apache
  # other test code
end
[/sourcecode]

The following helper methods are available.

• apache_config_parses?: predicate method that checks if the
  configuration passes Apache’s syntax check.
• apache_configured_ports: returns the ports configured in
  ports.conf
• apache_enabled_modules: uses the apache binary command to check
  loaded modules, used in module tests
• apache_service: returns the correct apache service object for
  minitest-chef with the name based on the node’s platform
• config: returns the correct main apache config file object for
  minitest-chef with the location based on apache dir attribute and
  node’s platform
• ran_recipe?: predicate method that checks if a particular recipe has
  been seen by Chef for the node.

Most of the apache2 cookbook’s recipes are Apache modules. The
apache_enabled_modules method is used in the tests to verify that
the module was in fact loaded. We can rely on this with confidence
that it is correct because:

• If minitest tests are being run, the recipe(s) converged.
• We trust that Chef does what we tell it to do because Chef recipes
  contain declarative resources to configure the system, and this part
  of Chef’s codebase is itself is well-tested.

Several of the recent changes to the minitest-chef tests for this
cookbook are related to ensuring that platform specifics are handled
so we can accurately test what is actually going to happen on
different supported platforms under test-kitchen. For example,
mod_apreq2 had a separate test
added
instead of using the general modules test because it has a different
Apache module name than the recipe name, meaning the automatic
inclusion wasn’t working. Other changes account for platform specific
differences.

Cucumber Features

The most recent changes to the cookbook didn’t affect the
Cucumber features that were added by Andrew
Crump. However, a recent change in
test-kitchen means that
the features aren’t automatically run, because we’re not using
“runtimes”, which couples the cookbook use to RVM.
Note that RVM isn’t bad, but it isn’t part of the base boxes we
use, so additional consideration
is required that complicates the basic use case.

If you would like to test this cookbook with RVM, please get ahold
of us on irc channel
#chef-hacking or
the Chef
Developer’s mailing list and we can help you get up and
running.

Contributions

The apache2 cookbook
CONTRIBUTING.md
file has been updated with new content, including contributions with
tests. In this section of the post we’ll talk about what we’re looking
for in new tests for the apache2 cookbook. We’ll also talk about what
we’re doing in other cookbooks for testing.

First, you do need to have the Git repository of the cookbook cloned.
The released version on the community site does not (yet) have the
test directory.

Second, you need to have your system set up to run test kitchen. This
means you should have the latest test-kitchen RubyGem installed, and
you need VirtualBox and
Vagrant. Vagrant is a dependency of
test-kitchen, but VirtualBox is a separate install. We use
bundler so bundle install will ensure that
your system has the right gems to run test-kitchen.
Next, add the minitest-chef test(s) for the configuration. In TDD/BDD
style, write the test before you write the new code in the cookbook.
Ideally, run Test Kitchen to prove the test fails first.

Do note that because kitchen test launches a Vagrant VM for each
platform and does a separate Chef run for each configuration, it can
take a long time to run the full tests. If you’re adding a new recipe
and configuration, or if you’re only modifying an existing one, you
can test only that configuration on its own. For example, if you
updated the “apache2::default” recipe:

[sourcecode language="bash"]
bundle exec kitchen test --configuration default
[/sourcecode]

Or, if you added a new recipe and configuration:

bundle exec kitchen test --configuration new_configuration

After verifying that the test fails, write the code for your
contribution, and then rerun kitchen test to make sure it passes.

Adding a New Apache Module

The scope of our apache2 cookbook isn’t to support every module
ever created. Our recommendation is to use
the apache_module definition in your own recipe where you
need it. However, commonly used modules may be useful for the
community, and are good candidates for contribution. If this is the
case, create a new recipe mod_MODNAME, and do use
the apache_module definition to configure the module as
necessary. See the README.md for documentation on how to use the
definition. If the module itself requires any particular packages or
other configuration or setup, include that in the recipe. The various
modules in the cookbook provide lots of examples for that, such
as apache2::mod_apreq2 or
apache2::mod_auth_cas.

If the recipe is simply this:

[sourcecode language="ruby"]
apache_module "MODNAME"
[/sourcecode]

Then add it to the modules configuration. Two files need to be
updated with MODNAME:

• test/kitchen/cookbooks/apache2_test/recipes/modules.rb
• test/kitchen/cookbooks/apache2_test/files/default/tests/minitest/modules_test.rb

If the module is more complicated, such as platform specific setup or
checks, then a separate minitest-chef test and apache2_test recipe may
be required for the configuration. Do the following:

• Add the configuration in the test/kitchen/Kitchenfile named after
  the apache2 cookbook recipe, e.g. “mod_MODNAME“
• Create a recipe “mod_MODNAME.rb” in the apache2_test cookbook.
• Add a minitest-chef test for the configuration,
  files/default/tests/minitest/mod_MODNAME_test.rb

When creating the test file, be sure to include the helper, as
described above:

[sourcecode language="ruby"]
require File.expand_path('../support/helpers', __FILE__)

describe "apache2::RECIPENAME" do
  include Helpers::Apache
  # other test code
end
[/sourcecode]

Here, “#other test code” is your test. For example, to check that the module is
enabled:

[sourcecode language="ruby"]
it 'enables MODNAME_module' do
  apache_enabled_modules.must_include "MODNAME_module"
end
[/sourcecode]

Note: “_module” is significant in the must_include method. The
apache_enabled_modules helper method returns an array and this will
will compare the string passed as a parameter using that. This can be
seen in the other mod_ tests in the minitest directory.

Once your configuration is added with a test recipe and a
minitest-chef test, run it with test kitchen:

[sourcecode language="bash"]
bundle exec kitchen test --configuration mod_MODNAME
[/sourcecode]

Other Cookbooks

This post was largely about our apache2 cookbook. We’re working to add
test kitchen support across all cookbooks that we publish. For many of
the cookbooks, this simply means initializing test kitchen in the
cookbook with kitchen init. For others, this means adding a variety
of tests as we’ve done and discussed for apache2. And for a few such
as Windows, we need to extend the toolset – more baseboxes in bento
or improvements to test kitchen itself.

Coupling configurations to a specific recipe and minitest-chef test
is not required. A configuration could be there to describe a
regression test or a new feature.

For example, say we’re working
on COOK-1684,
which aims to unify behavior of Opscode’s rabbitmq cookbook across
Debian and Red Hat distributions. We can add a configuration
“cook-1684” in the Kitchenfile, and then in the
“rabbitmq_test” cookbook, add a “cook-1684” recipe that
looks like this:

[sourcecode language="ruby"]
node.set['rabbitmq']['use_apt'] = false
node.set['rabbitmq']['use_yum'] = true

log "#{cookbook_name}::#{recipe_name} tests that COOK-1684 is implemented."

include_recipe "rabbitmq::default"
[/sourcecode]

When the configuration is run, the attributes are set and then the
recipe is included, and we can see whether the outcome is as desired
by writing a minitest-chef test. In this particular example, it is
still a work in progress, so we don’t have the minitest written yet.
The idea would be to check that the apt repository is not present,
but the package will be installed from the distribution.

Conclusion

The work we’re doing with Test Kitchen in our cookbooks is evolving.
We’re adding basic support so it can run, and we’ll expand the
coverage of tests to ensure that the cookbooks do what they’re
supposed to do.

Part of that evolution includes expanding our baseboxes that we
build using the bento
project. Another part is resolving bugs and adding improvements and
features to Test
Kitchen itself. If you’d like to help out, please let us know by
posting on the chef-dev mailing list, the ##chef-hacking IRC channel
or simply by opening tickets and resolving them (or resolving existing
tickets!).

We’re working to create a screencast illustrating how to add
test-kitchen, minitest-chef and a test cookbook to an existing
cookbook. A link to the video will be posted on this blog and through
the Opscode twitter account.