SysAdvent Day 21: Baking Delicious Resources with Chef

Growing up, every Christmas time included the sweet smells of fresh baked cookies. The kitchen would get incredibly messy as we prepped a wide assortment from carefully frosted sugar cookies to peanut butter cookies. Holiday tins would be packed to the brim to share with neighbors and visiting friends.

My earliest memories of this tradition are of my grandmother showing me how to carefully imprint each peanut butter cookie with a crosshatch. We’d dip the fork into sugar to prevent the dough from sticking and then carefully press into the cookie dough. Carrying on the cookie tradition, I am introducing the concepts necessary to extend your Chef knowledge and bake up cookies using LWRPs.

To follow the walkthrough example as written you will need to have the Chef Development Kit (Chef DK), Vagrant, and Virtual Box installed (or use the Chef DK with a modified .kitchen.yml configuration to use a cloud compute provider such as Amazon).

Resource and Provider Review

Resources are the fundamental building blocks of Chef. There are many available resources included with Chef. Resources are declarative interfaces, meaning that we describe the state we want the resource to be in, rather than the steps required to reach that state. Resources have a type, name, one or more parameters, actions, and notifications.

Let’s take a look at one sample resource, Route.

route “NAME” do
  gateway “10.0.0.20”
  action :delete
end

The route resource describes the system routing table. The type of resource is route. The name of the resource is the string that follows the type. The route resource includes optional parameters of device, gateway, netmask, provider, and target. In this specific example, we are only declaring the gateway parameter. In the above example we are using the delete action and there are no notifications.

Each Chef resource includes one or more providers responsible for actually bringing the resource to the desired state. It is usually not necessary to select a provider when using the Chef-provided resources, Chef will select the best provider for the job at hand. We can look at the underlying Chef code to examine the provider. For example here is the Route provider code and rubydoc for the class.

While there are ready-made resources and providers, they may not be sufficient to meet our needs to programmatically describe our infrastructure with small clear recipes. We reach that point where we want to reduce repetition, reduce complexity, or improve readability. Chef gives us the ability to extend functionality with Definitions, Heavy Weight Resources and Providers (HWRP), and Light Weight Resources and Providers (LWRP).

Definitions are essentially recipe macros. They are stored within a definitions directory within a specific cookbook. They cannot receive notifications.

HWRPs are pure ruby stored in the libraries directory within a specific cookbook. They cannot use core resources from the Chef DSL by default.

LWRPs, the main subject of this article, are a combination of Chef DSL and ruby. They are useful to abstract repeated patterns. They are parsed at runtime and compile into ruby classes.

LWRPs

Extending resources requires us to revisit the elements of a resource: type, name, parameters, actions, and notifications.

Idempotence and convergence must also be considered.

Idempotence means that the provider ensures that the state of a resource is only changed if a change is required to bring that resource into compliance with our desired state or policy.

Convergence means that the provider brings the current resource state closer to the desired resource state.

Resources have a type. The LWRP’s resource type is defined by the name of the file within the cookbook. This implicit name follows the formula of: cookbook_resource. If the default.rb file is used the new resource will be named cookbook.

File names should match for the LWRP’s resource and provider within the resources and providers directories. The chef generators will ensure that the files are created appropriately.

The resource and it’s available actions are described in the LWRP’s resource file.

The steps required to bring the piece of the system to the desired state are described in the LWRP’s provider file. Both idempontence and convergence must also be considered when writing the provider.

Resource DSL

The LWRP resource file defines the characteristics of the new resource we want to provide using the Chef Resource DSL. The Resource DSL has multiple methods: actions, attribute, and default_action.

Resources have a name. The Resource DSL allows us to tag a specific parameter as the name of the resource with :name_attribute.

Resources have actions. The Resource DSL uses the actions method to define a set of supported actions with a comma separated list of symbols. The Resource DSL uses the default_action method to define the action used when no action is specified in the recipe.

Note: It is recommended to always define a default_action.

Resources have parameters. The Resource DSL uses the attribute method to define a new parameter for the resource. We can provide a set of validation parameters associated with each parameter.

Let’s take a look at an example of a LWRP resource from existing cookbooks.

djbdns includes the djbdns_rr resource.

actions :add
default_action :add

attribute :fqdn,     :kind_of => String, :name_attribute => true
attribute :ip,       :kind_of => String, :required => true
attribute :type,     :kind_of => String, :default => "host"
attribute :cwd,      :kind_of => String

The rr resource as defined here will have one action: add, and 4 attributes: fqdn, ip, type, and cwd. The validation parameters for the attribute show that all of these attributes are expected to be of the String class. Additionally ip is the only required attribute when using this resource in our recipes.

Provider DSL

The LWRP provider file defines the “how” of our new resource using the Chef Provider DSL.

In order to ensure that our new resource functionality is idempotent and convergent we need the:

• desired state of the resource
• current state of the resource
• end state of the resource after the run

Let’s take a look at an example of a LWRP provider from an existing cookbook to illustrate the Chef DSL provider methods.

djbdns includes the djbdns_rr provider.

action :add do
  type = new_resource.type
  fqdn = new_resource.fqdn
  ip = new_resource.ip
  cwd = new_resource.cwd ? new_resource.cwd : "#{node['djbdns']['tinydns_internal_dir']}/root"

  unless IO.readlines("#{cwd}/data").grep(/^[\.\+=]#{fqdn}:#{ip}/).length >= 1
    execute "./add-#{type} #{fqdn} #{ip}" do
      cwd cwd
      ignore_failure true
    end
    new_resource.updated_by_last_action(true)
  end
end

new_resource

new_resource returns an object that represents the desired state of the resource. We can access all attributes as methods of that object. This allows us to know programmatically our desired end state of the resource.

type = new_resource.type assigns the value of the type attribute of the new_resource object that is created when we use the rr resource in a recipe with a type parameter.

load_current_resource

load_current_resource is an empty method by default. We need to define this method such that it returns an object that represents the current state of the resource. This method is responsible for loading the current state of the resource into @current_resource.

In our example above we are not using load_current_resource.

updated_by_last_action

updated_by_last_action notifies Chef that a change happened to converge our resource to its desired state.

As part of the unless block executing new_resource.updated_by_last_action(true) will notify Chef that a change happened to converge our resource.

Actions

We need to define a method for each supported action within the LWRP resource file. This method should handle doing whatever is needed to configure the resource to be in the desired state.

We see that the one action defined is :add which matches our LWRP resource defined actions.

Cooking up a cookies_cookie resource

Preparing our kitchen

First, we need to set up our kitchen for some holiday baking! Test Kitchen is part of the suite of tools that come with the Chef DK. This omnibus package includes a lot of tools that can be used to personalize and optimize your workflow. For now, it’s back to the kitchen.

Note: On Windows you need to verify your PATH is set correctly to include the installed packages. See this article for guidance.

Download and install both Vagrant, and Virtual Box if you don’t already have them. You can also modify your .kitchen.yml to use AWS instead.

We’re going to create a “cookies” cookbook that will hold all of our cookie recipes. First we will use the chef cli to generate a cookbook that will use the default generator for our cookbooks. You can customize default cookbook creation for your own environments.

chef generate cookbook cookies
Compiling Cookbooks...
Recipe: code_generator::cookbook

followed by more output.

We’ll be working within our cookies cookbook so go ahead and switch into the cookbook’s directory.

$ cd cookies

By running chef generate cookbook we get a number of preconfigured items. One of these is a default Test Kitchen configuration file. We can examine our kitchen configuration by looking at the .kitchen.yml file:

$ cat .kitchen.yml

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-12.04
  - name: centos-6.5

suites:
  - name: default
    run_list:
      - recipe[cookies::default]
    attributes:

The driver section is the component that configures the behavior of Test Kitchen. In this case we will be using the kitchen-vagrant driver that comes with Chef DK. We could easily configure this to use AWS or any other cloud compute provisioner.

The provisioner is chef_zero which allows us to use most of the functionality of integrating with a Chef Server without any of the overhead of having to install and manage one.

The platforms define the operating systems that we want to test against. Today we will only work with the CentOS platform as defined in this file. You can delete or comment out the Ubuntu line.

The suites is the area to define what we want to test. This includes a run_list with the cookbook::default recipe.

Next, we will spin up the CentOS instance.

Note: Test Kitchen will automatically download the vagrant box file if it’s not already available on your workstation. Make sure you’re connect to a sufficiently speedy network!

$ kitchen create

Let’s verify that our instance has been created.

$ kitchen list

➜  cookies git:(master) ✗ kitchen list
Instance             Driver   Provisioner  Last Action
default-centos-65    Vagrant  ChefZero     Created

This confirms that a local virtualized node has been created.

Let’s go ahead and converge our node which will install chef on the virtual node.

$ kitchen converge

Cookie LWRP prep

We need to create a LWRP resource and provider file and update our default recipe.

We create the LWRP base files using the chef cli included in the Chef DK.
This will create the two files resources/cookie.rb and providers/cookie.rb

chef generate lwrp cookie

Let’s edit our cookie LWRP resource file and add a single supported action of create.

Edit the resources/cookie.rb file with the following content:

actions :create

Next edit our cookie LWRP provider file and define the supported create action. Our create method will log a message that includes the name of our new_resource to STDOUT.

Edit the providers/cookie.rb file with the following content:

use_inline_resources

action :create do
 log " My name is #{new_resource.name}"
end

Note: use_inline_resources was introduced in Chef version 11. This modifies how LWRP resources are handled to enable the inline evaluation of resources. This changes how notifications work, so read carefully before modifying LWRPs in use!

Note: The Chef Resource DSL method is actions because we are defining multiple actions that will be defined individually within the providers file.

We will now test out our new resource functionality by writing a recipe that uses it.
Edit the cookies cookbook default recipe. The new resource follows the naming format of #{cookbookname}_#{resource}.

cookies_cookie "peanutbutter" do
   action :create
end

Converge the image again.

$ kitchen converge

Within the output:

Converging 1 resources
Recipe: cookies::default
  * cookies_cookie[peanutbutter] action create[2014-12-19T02:17:39+00:00] INFO: Processing cookies_cookie[peanutbutter] action create (cookies::default line 1)
 (up to date)
  * log[ My name is peanutbutter] action write[2014-12-19T02:17:39+00:00] INFO: Processing log[ My name is peanutbutter] action write (/tmp/kitchen/cache/cookbooks/cookies/providers/cookie.rb line 2)
[2014-12-19T02:17:39+00:00] INFO:  My name is peanutbutter

Our cookies_cookie resource is successfully logging a message!

Improving the Cookie LWRP

We want to improve our cookies_cookie resource. We are going to add some parameters. To determine the appropriate parameters of a LWRP resource we need to think about the components of the resource we want to modify.

There are some basic common components of cookies. The essential components are fat, binder, sweetener, leavening agent, flour, and additions like chocolate chips or peanut butter. The fat provides flavor, texture, and spread of a cookie. The binder will help “glue” the ingredients together. The sweetener affects the color, flavor, texture, and tenderness of a cookie. The leavening agent adds air to our cookie changing the texture and height of the cookie. The flour provides texture as well as the bulk of the cookie structure. All of the additional ingredients differentiate our cookies flavoring.

A generic recipe would involve combining all the wet ingredients and dry ingredients separately and then blending them together adding the additional ingredients last. For now, we’ll lump all of our ingredients into a single parameter.

Other than ingredients, we need to know the temperature at which we are going to bake our cookies, and for how long.

When we add parameters to our LWRP resource, it will start with the keyword attribute, followed by an attribute name with zero or more validation parameters.

Edit the resources/cookie.rb file:

actions :create  

attribute :name, :name_attribute => true
attribute :bake_time
attribute :temperature
attribute :ingredients

We’ll update our recipe to incorporate these attributes.

cookies_cookie "peanutbutter" do
   bake_time 10
   temperature 350
   action :create
end

Using a Data Bag

While we could add the ingredients in a string or array, in this case we will separate them away from our code. One way to do this is with data bags.

We’ll use a data_bag to hold our cookie ingredients. Production data_bags normally exist outside of our cookbook within our organization policy_repo. We are developing and using chef_zero so we’ll include our data bag within our cookbook in the test/integration/data_bags directory.

To do this in our development environment we update our .kitchen.yml so that chef_zero finds our data_bags.

For testing our new resource functionality, add the following to the default suite section of your .kitchen.yml:

data_bags_path: "test/integration/data_bags"

At this point your .kitchen.yml should look like this.

$ mkdir -p test/integration/data_bags/cookies_ingredients

Create peanutbutter item in our cookies_ingredients data_bag by creating a file named peanutbutter.json in the directory we just created:

{
  "id" : "peanutbutter",
  "ingredients" :
    [
      "1 cup peanut butter",
      "1 cup sugar",
      "1 egg"
    ]
}

We’ll update our recipe to actually use the cookies_ingredients data_bag:

search('cookies_ingredients', '*:*').each do |cookie_type|
  cookies_cookie cookie_type['id'] do
    ingredients cookie_type['ingredients']
    bake_time 10
    temperature 350
    action :create
  end
end

Now, we’ll update our LWRP resource to actually validate input parameters, and update our provider to create a file on our node, and use the attributes. We’ll also create an ‘eat’ action for our resource.

Edit the resources/cookie.rb file with the following content:

actions :create, :eat

attribute :name, :name_attribute => true
# bake time in minutes
attribute :bake_time, :kind_of => Integer
# temperature in F
attribute :temperature, :kind_of => Integer
attribute :ingredients, :kind_of => Array

We’ll update our provider so that we create a file on our node rather than just logging to STDOUT. We’ll use a template resource in our provider, so we will create the required template.

Create a template file:

$ chef generate template basic_recipe

Edit the templates/default/basic_recipe.erb to have the following content:

Recipe: <%= @name %> cookies

<% @ingredients.each do |ingredient| %>
<%= ingredient %>
<% end %>

Combine wet ingredients.
Combine dry ingredients.

Bake at <%= @temperature %>F for <%= @bake_time %> minutes.

Now we will update our cookie provider to use the template, and pass the attributes over to our template.
We will also define our new eat action, that will delete the file we create with create.

Edit the providers/cookie.rb file with the following content:

use_inline_resources

action :create do

  template "/tmp/#{new_resource.name}" do
    source "basic_recipe.erb"
    mode "0644"
    variables(
      :ingredients => new_resource.ingredients,
      :bake_time   => new_resource.bake_time,
      :temperature => new_resource.temperature,
      :name        => new_resource.name,
    )
  end
end

action :eat do

  file "/tmp/#{new_resource.name}" do
    action :delete
  end

end

Try out our updated LWRP by converging your Test Kitchen.

kitchen converge

Let’s confirm the creation of our peanutbutter resource by logging into our node.

kitchen login

Our new file was created at /tmp/peanutbutter. Check it out:

[vagrant@default-centos-65 ~]$ cat /tmp/peanutbutter
Recipe: peanutbutter cookies

1 cup peanut butter
1 cup sugar
1 egg

Combine wet ingredients.
Combine dry ingredients.

Bake at 350F for 10 minutes.

Let’s try out our eat action. Update our recipe with

search("cookies_ingredients", "*:*").each do |cookie_type|
  cookies_cookie cookie_type['id'] do
    action :eat
  end
end

Converge our node, login and verify that the file doesn’t exist anymore.

$ kitchen converge
$ kitchen login
Last login: Fri Dec 19 05:45:23 2014 from 10.0.2.2
[vagrant@default-centos-65 ~]$ cat /tmp/peanutbutter
cat: /tmp/peanutbutter: No such file or directory

To add additional cookie types we can just create new data_bag items.

Cleaning up the kitchen

Finally once we are done testing in our kitchen today, we can go ahead and clean
up our virtualized instance with kitchen destroy.

kitchen destroy

Next Steps

We have successfully made up a batch of peanut butter cookies yet barely touched the surface of extending Chef with LWRPs. Check out Chatper 8 in Jon Cowie’s book Customizing Chef and Doug Ireton’s helpful 3-part article on creating LWRP. You should examine and extend this example to use load_current_resource and updated_by_last_action. Try to figure out how to add why_run functionality. I look forward to seeing you share your LWRPs with the Chef community!

Feedback and suggestions are welcome iennae@gmail.com.

Additional Resources

• Three part article on creating LWRP from Doug Ireton
• Creating your own chef Cookbook Generator
• Supermarket – Community cookbook site
• Science behind the Best Chocolate Chip Cookies
• Christmas Cookies from around the world
• Nathen Harvey’s Cookie Cookbook Repo

Thank you

Thank you to my awesome editors who helped me ensure that these cookies were tasty!

• Katherine Daniels
• H. “Waldo” Grunenwald
• Matt Straton
• Nathen Harvey