In this blog post, you’ll take a look at some practices that can help you get a running start with your Chef automation and the principle of Policy as Code. It’s intended for any IT professional or Chef newcomer.
Quick Links
- Prologue: What do I need to know?
- Part 1: Chef Infra Server
- Part 2: Chef Workstation
- Part 3: The Chef Cookbook
- Part 4: The Chef InSpec Profile
- Part 5: The Policyfile
- Part 6: Test Kitchen
- Part 7: Bootstrapping Nodes
- Part 8: Extending what you've learned
- Epilogue: Where do we go from here?
- Additional Resources
Prologue: What do I need to know?
This post is designed to be beginner-friendly. You don’t need any coding experience to follow along — just an interest in automation, and curiosity about how best to implement it with Chef. That said, let’s define some terms before we dive in:
Chef Cookbooks – Cookbooks are how Chef Code is organized. A cookbook contains many files and subdirectories, including the all-important recipes and profiles, which define the configurations and compliance controls you want to apply to the systems you manage.
Chef Server – A Chef Infra Server stores cookbooks, profiles, policyfiles and other Chef code, and shares it with the target nodes you want to manage. It provides information about each system, including its assigned cookbooks and profiles. It also maintains a search index of information about your environment so you can easily identify and manage discrete subsets of systems as needed.
Chef Automate – Chef Automate is Chef's enterprise platform that allows developers, operations and security engineers to collaborate on delivering application and infrastructure changes, and compliance statuses. Chef Automate provides insights across multiple data centers and cloud providers, and can be installed alongside Chef Infra Server and Habitat Builder to give you the complete modern Chef experience. This post won’t go into specific detail about Automate, but we have other resources to help you get up and running.
Chef Client – The Chef Client is an agent that runs on each managed system, including Linux, Windows or macOS. It communicates with the Chef Server and pulls down any cookbooks, compliance profiles and configurations required by its Policyfile or run_list, executes those recipes and compliance scans, and reports its results back to the Chef Server.
Chef InSpec – Chef InSpec provides the auditing and compliance profiles that test your cookbook logic and ensure your systems are in the states you desire. Like Chef cookbooks, profiles are triggered and applied by the Chef Client, and return output results via the CLI and in Chef Automate dashboards. Successfully applied profiles show if your systems are, or are not, compliant.
Chef Workstation – Chef Workstation contains everything you need to get started writing and working with Chef cookbooks, including command-line utilities for communicating with the Chef Server, and testing tools so you can validate cookbooks locally before updating your environments. It can be freely installed on Linux, macOS and Windows systems, and includes embedded Ruby and other dependencies out of the box. Think of it as a toolbox used to create and build all your Chef things. It replaces the now-deprecated ChefDK.
Chef Repo – In this context, a Chef Repo is the directory structure on your workstation or laptop where you store your Chef content. It’s generally located at the root of your home directory, such as /home/jsmith/chef-repo on Linux, /Users/jsmith/chef-repo on macOS or C:\Users\jsmith\chef-repo on Windows. It can be created anywhere on your system, but placing it in your home directory is easy to remember and reference. It’s also good practice to initialize the directory with Git or other version-control tool to version, remotely save and share your code.
Test Kitchen – Test Kitchen uses a driver plugin architecture to enable you to test your Chef cookbooks and compliance profiles locally using Docker (with special containers designed to perform like full-up systems), VMware, Hyper-V, VirtualBox and other hypervisors. It also enables you to test on cloud instances from providers such as Amazon EC2, Google Compute Engine, and Microsoft Azure. Using Test Kitchen allows you to develop, test and troubleshoot your code quickly.
Chef Supermarket – The Chef Supermarket is a publicly available repository of nearly 4,000 user-created cookbooks that are ready to use. Some are written by Chef engineers and our partners, and some are written by members of the Chef community. In either case, cookbooks on the Supermarket are open-source, and can be used, inspected or modified to suit an individual or organization's specific needs.
Before we begin
Feel free to follow along on your own systems. All the examples provided are functional. If you already have a Chef server and Chef Workstation installed on your laptop, you can jump straight to Part 3: The Chef Cookbook.
Part 1: Chef Infra Server
It's possible to use Chef without running a full Chef Infra Server (by using Chef Zero on a workstation), but to get the full modern experience, it's best to run your own on-prem or cloud-based server. Having a full-blown Chef Infra Server doesn’t require much in the way of resources (a single VM or modest cloud instance will do) and it gives you the full experience of storing and distributing your code among target nodes.
Today, installing Chef Infra Server is more straightforward than ever, and it can be installed side by side with Chef Automate and Chef Habitat Builder, used for application continuous integration. Each Chef product can be installed with a --product flag using the standalone chef-automate binary. This enables you to install and configure your Chef server quickly, in about 15 minutes in most cases.
You can also deploy a Chef server in the cloud from either the Amazon or Azure marketplaces. These systems come pre-configured with Chef Automate and Chef Infra Server. Just create an admin user and an organization to get started.
What you'll need:
- A virtual machine with at least 2 CPU, 8 GB of RAM and a 32GB disk running a modern version of Linux, such as Ubuntu 20.04 or CentOS 8. (Note: This is well below the minimum system requirements for a production environment)
- A static IP address for the VM
- A non-root user with sudo privileges
- A subnet that allows your Chef server to communicate with your workstation and Linux, Windows or macOS target nodes. Target nodes can be other VMs.
- An optional DNS entry on your subnet that allows your server, nodes and workstation to communicate via FQDN. Without DNS, you'll need to edit the hosts file on your Chef server and target nodes so they can reach each other by name.
- OR an AWS or Azure Marketplacedeployment
With a raw Linux VM running, log in via the shell and run the following commands or simply copy the contents of this deploy-chef-server.sh script. Update the Variables before running it:
$ chmod +x deploy-chef-server.sh
$ ./deploy-chef-server.sh
To deploy your Chef server without the script, follow these steps:
Set the fully qualified domain name with hostnamectl set-hostname hostname. It’s critical that the fqdn value in the file matches the hostname –f value of the system because the installation process sets up certificates with the FQDN.
$ hostnamectl set-hostname automate.chef.lab
Install the chef-automate tool by downloading the standalone package and making it executable.
$ curl https://packages.chef.io/files/current/latest/chef- automate-cli/chef-automate_linux_amd64.zip | gunzip - > chef- automate && chmod +x chef-automate
Adjust environment settings:
$ sudo sysctl -w vm.max_map_count=262144 $ sudo sysctl -w vm.dirty_expire_centisecs=20000
Be sure these are written to /etc/sysctl.conf to persist across reboots.
Execute chef-automate to deploy Chef Infra Server:
$ sudo ./chef-automate deploy --product infra-server
Create an admin user and organization
To use your Chef server, you'll need to create an administrative user and at least one organization. These provide secure access between your server and your workstation (among other things), and establish a location for your Chef content. Notice the chef-server org-create command uses the admin user name you create as the --association-user:
$ sudo chef-server-ctl user-create jsmith Jack Smith jsmith@gmail.com "password" --filename jsmith.pem
$ sudo chef-server-ctl org-create lab "My Lab" --association_user jsmith --filename lab-validator.pem
The commands in this example generate two certificate files, jsmith.pem and lab-validator.pem. In Part 2, you'll copy the jsmith.pem file to your workstation to enable secure communication between the Chef Server and your laptop.
Part 2: Chef Workstation
What you'll need:
- A Linux, Windows or macOS workstation
- Internet access
- Network access to the same subnet as your Chef server
Install Chef Workstation
Chef Workstation is a collection of tools that enable your laptop to securely interact with your Chef Server. It include chef, knife, inspec, cookstyle, habitat and kitchen. It also contains embedded Ruby and other dependencies so you don’t have to install anything else to immediately start using Chef Workstation tools.
You can install Chef Workstation by downloading an OS-specific installer or use the Chef install.sh script.
If using the OS-specific installer, set up chef shell-init for your environment and set a path that adds the embedded Chef binaries directory. This ensures the Workstation tools are using the embedded version of Ruby and not some other version on your workstation machine. On a Linux system, this would look something like this:
$ echo 'eval "$(chef shell-init bash)"' >> /home/$USER/.bash_profile
$ echo 'export PATH="/opt/chef-workstation/embedded/bin:$PATH"' >> /home/$USER/.configuration_file
Confirm the installation by running the following from a terminal:
$ chef -v
Chef Workstation version: 21.10.640
Chef CLI version: 5.4.2
Chef Habitat version: 1.6.351
Test Kitchen version: 3.1.0
Cookstyle version: 7.25.6
Chef Infra Client version: 17.6.18
Chef InSpec version: 4.46.13
Create a Chef repo
Creating the Chef repo where all your Chef cookbooks and other files will live is simplified using one of the Chef generators:
$ cd ~ $ chef generate repo chef-repo
The bold bit can be anything, but chef-repo is descriptive and easy to remember. Running this generator creates a Chef repository and several subdirectories, including ~/chef-repo/cookbooks, where your Chef cookbooks and recipes will live.
Initialize your system
In order to enable your workstation to communicate and interact with your Chef server, you need to create a credentials file. This is located in a directory named .chef in your home directory, such as /home/jsmith/.chef. This folder and credentials file can be created with another generator:
$ knife configure init-config
Running this command in a terminal will present you with several questions, including your Chef server URL and the username you created when setting up your server. This will output a ~/.chef/credentials file with contents that look something like this:
[default] client_name = jsmith client_key = '~/.chef/jsmith.pem' chef_server_url = 'https://automate.chef.lab/organizations/lab'
Edit the credentials file to add your default cookbook location, in this case the cookbooks folder inside the ~/chef-repo directory you created in the previous step. You can add more than one path, ['/path/to/one', '/path/to/two']:
cookbook_path = ['~/chef-repo/cookbooks']
Copy the user.pem file from your Chef server to your workstation
The client_key entry in the above .credentials file points to the user.pem file. Copy this file from your Chef server to your workstation and place it in the ~/.chef/ directory using a tool like scp. Alternatively, you can copy the contents of the user.pem file on your Chef server and paste them into a new file on your workstation:
$ cd ~/.chef $ scp jsmith@automate.chef.lab:/home/jsmith/jsmith.pem .
Fetch and verify your SSL certs
You’re now ready to fetch the certs, which Chef Workstation will automatically place in the ~/.chef/trusted_certs folder. Certs stored here will be used for secure communication and will be trusted – even if they’re self-signed.
$ knife ssl fetch $ knife ssl check
With your certificates successfully verified, test the connection between your workstation and your Chef server with another simple knife command. This will return the name of the client organization you created when setting up your Chef server:
$ knife client list
lab-validator
With Chef Infra Server and Chef Workstation set up you're now ready to start using Chef to configure and manage nodes. It bears noting that these configuration steps are ideal for both lab and enterprise environments. In a production environment, be sure to follow the Chef server System Requirements.
To make your life even easier, install VS Code on your workstation and add the Chef Extension. Its auto-completion and built-in Chef Language references make it easy to get started writing Chef code.
Part 3: The Chef Cookbook
What you’ll need:
- Chef Workstation installed and configured on your workstation or laptop
- A running and accessible Chef server
- A text editor or IDE, such as VSCode
In this section, you'll create a simple cookbook to manipulate when the chef-client agent runs on your systems.
In order for Chef to manipulate systems for your automation and system compliance, the Chef Infra Client is installed on each node during the bootstrap process (described later) along with a client.rb file that enables proper communication with your Chef server. The chef-client installation and related configurations are done automatically when you bootstrap a node, and each time chef-client runs on a node, it communicates with the Chef server, downloads any updates or changes, performs the tasks you've defined, and reports back the results.
You can run chef-client manually from the terminal on any system at any time, but that's not practical or the "Chef" way. Instead, you can automate the process with the following simple cookbook that will do the work for you. The result will be that your systems will check in with your Chef server every 30 minutes.
In the working sample below, you’ll use the windows_task and the cron resources. These are just two of nearly 250 built-in Chef Resources you can use to manipulate your systems. In this example, you'll create a single recipe to run chef-client every 30 minutes on Linux and Windows systems. On Linux, this is done with a cron job. On Windows, this is done with a Windows Task.
Create a cookbook
On your workstation, navigate to your ~/chef-repo/cookbooks folder and use the Chef cookbook generator to create a cookbook called run-chef-client:
$ cd ~/chef-repo/cookbooks $ chef generate cookbook run-chef-client -k dokken
This will create a new directory called run-chef-client and populate it with several other directories and files. The -k dokken flag generates a Test Kitchen kitchen.yml file pre-configured to work with Docker. You'll use that in a later step.
Change into the new ./run-chef-client directory and list its contents to get an idea of all the files the generator command created:
$ cd ~/chef-repo/cookbooks/run-chef-client $ tree ├── CHANGELOG.md ├── chefignore ├── kitchen.yml ├── LICENSE ├── metadata.rb ├── Policyfile.rb ├── README.md ├── recipes │ └── default.rb └── test └── integration └── default └── default_test.rb
In addition to these files and folders, create a directory to hold attributes (using another Chef generator) and a directory to hold your compliance profiles. You'll use these a little later:
$ cd ~/chef-repo/cookbooks/run-chef-client $ chef generate attribute default $ mkdir -p ./compliance/profiles
Edit the metadata.rb file to include your personal information. This file not only sets the version of your cookbook, but contains maintainer information, which will be automatically passed to any files you create using a Chef generator, such as recipes, files and templates:
name 'run-chef-client' maintainer 'John Tonello' maintainer_email 'jtonello@chef.lab' license ' Apache-2.0' description 'Set chef-client to run every 30 minutes' version '0.1.0' chef_version '>= 17.5'
The maintainer, maintainer_email and license values can all be set with flags when you generate the cookbook. For the above example, that would look like this:
$ chef generate cookbook run-chef-client -k dokken -C 'John Tonello' -m 'jtonello@chef.lab' -I 'apachev2'
Create the recipe
Edit the ./cookbooks/run-chef-client/recipes/default.rb file, update its header if necessary, and enter the following content. Be sure to set the user and password entries to match your environment. This recipe includes logic to work on both Windows and non-Windows systems. In more complicated cookbooks, you might have multiple recipes with different names. If you do that, just be sure to use the include_recipe resource with the paths to your other recipes in your main default.rb recipe:
# Cookbook:: client-run
# Recipe:: default
#
# Copyright:: 2021, John Tonello, All Rights Reserved.
include_profile 'run-chef-client::client-run'
if platform?('windows')
windows_task 'run-chef-client' do
user 'WIN10\jtonello'
password 'Sup3r!Us3r!'
command 'chef-client'
run_level :highest
frequency :minute
frequency_modifier 30
end
else
cron 'Run chef-client every 30 minutes' do
minute '0,30'
user 'root'
command '/usr/bin/chef-client'
action :create
end
end
The include_profile line is a new feature in Chef Infra Client 17.5.22 that allows you to define the compliance profile you want to associate with your cookbook right in the recipe. In this way, your profiles will be linked and uploaded when you use the Policyfile workflow in Step 5. The Chef notation 'run-chef-client::client-run' is pointing to ~/chef-repo/cookbooks/run-chef-client/compliance/profiles/client-run/.
The if…else statement in this recipe tells Chef to run either the windows_task (if the platform is Windows) or the cron resource (if the platform is anything else, such as Linux and macOS). The Windows Task requires a user and password, and the specific command to run as that user (chef-client). Both resources set the task to run every 30 minutes.
After you save the recipe, check that it's syntax is correct by running Chef Cookstyle, which detects errors and automatically corrects them with the -a flag.
$ cookstyle -a
When the results reports “no offenses,” the cookbook is syntactically correct.
Note: The knife cookbook upload wayIf you just want to apply this cookbook to a node – and you don’t want to do anything else, such as add compliance – you can upload this cookbook to your Chef server, bootstrap a node, and apply the cookbook. However, this is considered an older workflow: $ knife cookbook upload run-chef-client
$ knife bootstrap ubuntuserver01.fakedomain.tld -U ubuntu -i ~/.ssh/id_rsa.pub --sudo -N ubuntuserver01 -r 'recipe[run-chef-client]'
|
Part 4: The Chef InSpec Profile
Chef InSpec Profiles provide insights into the status of your nodes to see if they’re compliant. That means, it tells you if the cookbook you've applied and the state you want are correct.
Chef InSpec does the heavy lifting for you and reports compliance data via the CLI when you run the chef-client command from a terminal, and in the Chef Automate Compliance dashboard. To make this work, you need to create a cookbook attributes file and a profile containing controls.
Create an attributes file
If you haven't already, in the root of your run-chef-client cookbook folder, run the following command to generate an attributes directory inside your run-chef-client cookbook folder. This will create the folder and a file named default.rb:
$ chef generate attribute default
Edit ./attributes/default.rb and add the following:
default['audit']['reporter'] = %w(chef-server-automate cli)
The default['audit']['reporter'] entry sends output to both Chef Automate and the CLI.
Create a profile and edit your inspec.yml file
Generate a new profile by using the Chef generator:
$ inspec init profile [path-to-cookbook-root] [profile-name]
For example:
$ cd ~/chef-repo/cookbooks/run-chef-client/compliance/profiles/ $ inspec init profile client-run
This will create several folders and files in your new ./compliance/profiles/client-run directory, including a controls/example.rb file with example content, and an inspec.yml file. Edit inspec.yml, changing the bold items to values that make sense for your profile:
name: client-run title: Run the chef-client every 30 minutes maintainer: John Tonello copyright: John Tonello copyright_email: jtonello@chef.lab license: Apache-2.0 summary: Run the chef-client every 30 minutes version: 0.1.0 supports: platform: os
Create the InSpec profile
InSpec profiles contain controls, which describe each test you want to perform. These draw on system information available to the chef-client. In the following example, the control is called run-chef-client and it has an impact of 1.0, the highest priority. The control then adds logic to run different tests based on whether the target node is Windows or not. The describe entries tell Chef to check those resources on each system.
Rename the ./compliance/profiles/client-run/controls/example.rb to default.rb and edit it as follows:
control 'run-chef-client' do
impact 0.7
title 'Run the chef-client every 30 minutes'
if os.windows?
describe windows_task('run-chef-client') do
it { should exist }
it { should be_enabled }
end
else
describe crontab do
its('commands') { should include '/usr/bin/chef-client' }
its('minutes') { should include '0,30' }
end
end
end
Part 5: The Policyfile
A key consideration when creating any Chef cookbook is knowing that it contains code you know and trust. By using Policyfiles, you're creating immutable objects – bundles of cookbooks, recipes, profiles – and ensuring the code hasn't been altered in some way, either accidentally or maliciously.
When you created your run-chef-client cookbook with the Chef generator command earlier, a Policyfile.rb file was automatically created inside that cookbook folder. Open the file in your editor and change it to look something like the following:
# Policyfile.rb - Describe how you want Chef Infra Client to build your system. # A name describes what the system you're building with Chef does. name 'run-chef-client' # Where to find external cookbooks: default_source :chef_repo, "~/chef-repo/cookbooks" do |s| s.preferred_for "run-chef-client" end default_source :supermarket default_source :chef_server, 'https://automate.chef.lab/organizations/lab' # run_list: chef-client will run these recipes in the order specified. run_list 'run-chef-client::default'
The name entry gives your Policyfile a name, ideally something that matches your cookbook. The default_source can be your Chef server, the Chef Supermarket, or a local Chef repo, or all three as used here. Using :chef_repo with s.preferred_for means Chef will look in your workstation's cookbooks folder for content, not from a remote source, such as the Chef Supermarket or your Chef server itself.
The run_list entry (or entries) describe the cookbook and path in Chef notation. Here, run-chef-client::default is pointing to ~/chef-repo/cookbooks/run-chef-client/recipes/default.rb. You can add more than one run_list item and actually create profiles that tie together several different cookbooks into a single, immutable Policyfile.
With the Policyfile complete, go ahead and run cookstyle -a in your run-chef-client cookbook directory to double-check your syntax before the next step:
$ cd ~/chef-repo/cookbooks/run-chef-client $ cookstyle -a
Upload your cookbook and profiles all at once
With Policyfiles, you can bundle your cookbooks and compliance profiles and upload them to your Chef server. First, install the Policyfile.rb to create a Policyfile.lock.json file, then push that file to your Chef server.
cd ~/chef-repo/cookbooks/run-chef-client $ chef install Policyfile.rb $ chef push prod Policyfile.lock.json
The push command includes the name of the Policy Group this Policyfile will belong to. With Chef, you can create different policy groups for different purposes, such as build, test and prod. In this example, the run-chef-client cookbook and client-run profile are added to the prod (production) policy group. Don't worry if you haven't already created the policy group on your Chef server; chef push does it for you.
With your ./compliance/profiles/default/default.rb and inspec.yml in place and your Policyfile.rb set, you can now test your cookbook and compliance controls with Test Kitchen, or skip ahead to Part 7 to apply your run-chef-client cookbook and profile on a target node.
Uploading cookbooks with knife
It's possible to upload your cookbooks to your Chef server using a simple knife command, and then assign one or more cookbooks to each of your nodes. This would look like this:
$ knife upload cookbook run-chef-client $ knife node run_list add ubuntu01 'recipe[run-chef-client]'
The first command uploads your cookbook to the Chef server. The second assigns your run-chef-client Cookbook to run on that node.
Note: Cookbooks vs. Recipes Observant readers may note that even though "run-chef-client" is a cookbook, the run list refers to a recipe. What's the deal with that? This is a common stumbling block for new users, and the short answer is that the chef-client executes recipes, and cookbooks are simply where those recipes are stored. Typically in a run_list item, the notation is recipe[COOKBOOK_NAME::RECIPE_NAME]. A notable exception is our example above, which simply contained recipe[run-chef-client]. This is shorthand notation for recipe[run-chef-client::default]. Whenever a cookbook is specified without a recipe, that just means that the default recipe (default.rb) will be executed. This approach is valid, but it's more difficult and less immutable than using Policyfiles. It requires additional steps if you want to upload and use InSpec profiles or you have multiple dependent cookbooks. Therefore, it's best to take advantage of Policyfiles to do the work for you. |
Part 6: Test Kitchen
Test Kitchen provides a variety of ways to test your Chef cookbooks and compliance code locally, using Docker, VirtualBox, Hyper-V and the Amazon and Azure clouds. For this example, you'll use Docker, so you should have docker.io and docker-compose installed on your workstation with permissions to run it as a non-root user.
When you initially created the run-chef-client cookbook with the -k dokken flag, you automatically created a kitchen.yml file in the root of your cookbook directory that's already configured to use the Dokken driver (Docker). The file looks something like the example below.
--- driver: name: dokken privileged: true # allows systemd services to start provisioner: name: dokken transport: name: dokken verifier: name: inspec platforms: - name: ubuntu-20.04 driver: image: dokken/ubuntu-20.04 pid_one_command: /bin/systemd intermediate_instructions: - RUN /usr/bin/apt-get update - name: centos-8 driver: image: dokken/centos-8 pid_one_command: /usr/lib/systemd/systemd suites: - name: default run_list: - recipe[chef-client::default] verifier: inspec_tests: - compliance/profiles/client-run/ attributes: audit: compliance_phase: false reporter: 'cli'
The platform entries in this kitchen.yml file pull special Docker images that behave like virtual machines. The Chef team has taken base containers for those Linux OSes and added important bits back in to give you real-world test environments without all the overhead.
If you want to add other platforms in addition to, or instead of, Ubuntu 20.04 and CentOS 8 shown in this example, you can add entries from the dokken image repo. You can also edit the above entries – to, say, ubuntu-18.04 and centos-7 – and see Chef results on platforms that are relevant to your needs. At last count, there were 41 Dokken images available for testing.
Edit the verifier section under suitesto point to the relative path of your InSpec profile – compliance/profiles/client-run/ in this example – and add the following stanza to suppress the Compliance Phase in Test Kitchen. Verification will still work, but this will help avoid runtime errors:
attributes: audit: compliance_phase: false reporter: 'cli'
Run Test Kitchen
Test Kitchen works by first spinning up containers (or VMs or cloud instances) based on images described in your kitchen.yml platform entries, and deploying the chef-client and Chef configuration files. It then deploys your Chef code (known as converging), verifies your code, and finally shuts down and destroys your test containers or instances. You can run all these steps at once by navigating to the root of your run-chef-client cookbook directory and running kitchen test:
$ cd ~/chef-repo/cookbooks/run-chef-client $ kitchen test
This simple two-word command does a lot. It performs the create, converge, verify and destroy steps, and will show you in-terminal results of your cookbook being applied and the Compliance Phase scan results. You can also run reach step separately:
$ kitchen create $ kitchen converge $ kitchen verify $ kitchen destroy
By using the steps separately, you can re-run kitchen converge and kitchen verify to rapidly iterate changes. The results appear in your terminal. Green output indicates your systems are compliant.
Part 7: Bootstrapping Nodes
Now that the cookbook and profiles are pushed to your Chef server and tested, you can move on bootstrapping your first target node and applying your cookbook!
Note: Bootstrapping WindowsThe example below bootstraps an Ubuntu server over SSH. The run-chef-client cookbook also supports Windows, but SSH is often not an option in Windows environments. For Windows systems, knife can be used to bootstrap nodes over WinRM. In addition to the knife ssh command, it has a knife winrm counterpart for Windows systems. It requires passing a -P (password) value. |
In Chef, bootstrapping performs the following tasks on a node:
- Installs the chef-client
- Creates a configuration file (config.rb), which defines Chef Server and authentication details, among other things
- Installs any self-signed certs that need to be trusted, if applicable
Knife has a built-in bootstrap command that can take care of all these steps in one shot. Your first target-node bootstrap will look something like the command below, run on your Chef Workstation system and targeting a system that supports ssh, the default:
$ knife bootstrap ubuntu01.fakedomain.tld -U ubuntu -i ~/.ssh/id_rsa --sudo -N ubuntu01 --policy-group prod --policy-name run-chef-client
The bootstrap command allows you to use a few flags to define some details:
ubuntu01.fakedomain.tld – no flag, just the hostname or IP address of the server being bootstrapped.
-U (or --connection-user)ubuntu – The username to use when connecting to the remote node.
-i ~/.ssh/id_rsa – An authentication key for ssh. ( -P can be used for password authentication instead).
–-sudo – Execute commands with superuser privilege. This is typically required for new software installation and bootstrapping.
-N (or --node-name) ubuntu01 – the “node name” to use when registering with the Chef server.
--policy-group prod --policy-name run-chef-client –This assigns your run-chef-clientpolicy to the node, and since the Policyfile.rb you created earlier includes your cookbook run_lists, you don't need to define a recipe. Chef knows about them already! Alternatively, you can use the -r 'recipe[run-chef-client]' to apply a cookbook to a node during the bootstrap. In this example, run the default.rb recipe from your run-chef-client cookbook is applied.
$ knife bootstrap ssh://10.128.1.25 -U ubuntu -i ~/.ssh/id_rsa -N ubuntu01 -r 'recipe[run-chef-client]'
Windows:
$ knife bootstrap winrm://10.128.1.26 -U Administrator -P mypassword -N windows10 -r 'recipe[run-chef-client]'
You can also bootstrap a node and not assign any recipes or policies:
$ knife bootstrap ssh://10.128.1.25 -U ubuntu -i ~/.ssh/id_rsa -N ubuntu01
$ knife bootstrap winrm://10.128.1.26 -U Administrator -P mypassword -N windows10
If you bootstrap one or more nodes without any flags, you can set policies and assign run_lists or policies later:
$ knife node policy set ubuntu01 prod run-chef-client
or:
$ knife node run_list add ubuntu01 'recipe[run-chef-client]'
These are just a few of the options available. For a full list of flags for any knife command, simply run it with the –h flag for descriptions of all options and usage examples.
See system state and compliance by running chef-client
After bootstrapping the example node ubuntu01, you’re able to see it by querying your Chef server with knife:
$ knife node list ubuntu01
If you bootstrapped the node with policy or recipe flags, you'll have seen output in the CLI showing the progress and results. However, you can see your Chef code in action by running chef-client on that node. Of course, your new run-chef-client cookbook will run it every 30 minutes, but you can do it manually in a couple ways:
Run chef-client using knife
The knife tool enables you to run ssh and winrm options to trigger commands on any remote node. On a Linux or macOS node, you would run something like the following to run the command sudo chef-client:
$ knife ssh 'name:ubuntu01' 'sudo chef-client'
If you're targeting a Windows machine, use the winrm option and provide the username (-x) and password (-P) of an administrative user:
$ knife winrm 'name:win10' 'chef-client' -x Administrator -P mypassword
For nodes that support ssh, you can shell into them and run sudo chef-client from the command line. On Windows, log in with RDP (or the desktop), open PowerShell as an administrative user, and run chef-client.
More than just policy as code
With your nodes up and running with the Chef Infra Client and your Chef code, you can now take a look at the data you have available to you. Indeed, now that the node called ubuntu01 has been bootstrapped, you can quickly learn a ton of information about it. To get a brief summary of that information, run the following:
knife node show ubuntu01 Node Name: ubuntu01 Policy Name: run-chef-client Policy Group: prod FQDN: ubuntu01.fakedomain.tld IP: 10.128.1.25 Run List: recipe[run-chef-client::default] Recipes: run-chef-client::default Platform: ubuntu 20.04 Tags:
You can also use Chef to query various system information pertaining to that node. For example:
$ knife search node 'name:ubuntu01' -a shells 1 items found ubuntu01: shells: /bin/sh /bin/bash /bin/rbash /bin/dash
Here, you’ve asked the Chef Server to display all the shells available on your ubuntu01 node.
In these example, the searches returned one node, but you could use a wildcard to return information about all nodes, or those with a particular prefix or suffix, such as 'name:*' (all nodes) or'name:web*' (just nodes with names that begin with "web") or 'name:*web' (just nodes with names that end with "web").
In addition to name, you could use platform (and other knife options):
$ knife search node 'platform:ubuntu' -a shells
But where did this information come from?
Even when you’re not applying any configuration changes to a node using Chef, every chef-client run uses a tool called ohai to collect system profiling information and send it back to the Chef server along with the results of your run. This means that right out of the box you have a way to learn all sorts of information about the systems you manage, and as you bootstrap more and more servers, you can easily query Chef for information across your environments. You can also use this information in your recipes to manipulate your nodes.
Here are a few more examples of the sort of data that can be pulled out of Chef:
$ knife search node 'name:ubuntu01' -a memory.total -a filesystem.by_mountpoint./.percent_used -a ipaddress 1 items found ubuntu01: filesystem.by_mountpoint./.percent_used: 13% ipaddress: 10.128.1.133 memory.total: 524288kB
In this search, you can see data about disk usage and memory. Ohai allows you to query all sorts of node information. It can even auto-detect if a node is in Amazon and query AWS internals, like the node's public IP address. To see that, instead of -a ipaddress in the above example, you would add -a ec2.public_ipv4. It’s worth noting that this data would be collected even if your node was bootstrapped without a run_list or policy applied.
As your use of Chef evolves, you'll learn that this data can be used a number of ways when writing automation code. The data collected by Ohai allows Chef recipes to take conditional action based on the results returned.
For example, rather than hard-coding the memory tuning for a database or webserver, you could define the resource allocation as a percentage of available resources, allowing the same code to be used in different systems, even if their hardware profiles are different. It also can be used to query Chef for information about other nodes, allowing for components like load balancers or monitoring servers to dynamically pull a list of its peers from the Chef server, and automatically add new nodes as they’re created.
A few more thoughts on attributes
Just as cookbooks contain recipes that will be executed by our nodes, they also have attributes that can modify that execution without altering any of the underlying code.
You can see all available attributes by running a knife search node on your node followed by the -l flag:
$ knife search node 'name:ubuntu01' -l 1 items found Node Name: ubuntu01 Policy Name: run-chef-client Policy Group: prod FQDN: ip-172-31-15-141.ec2.internal IP: 3.85.154.118 Run List: recipe[run-chef-client::default] Recipes: run-chef-client::default Platform: ubuntu 20.04 Tags: Attributes: tags: … chef_environment: prod chef_guid: 4d10ca10-7658-4693-91b6-6ed25d881a1c chef_packages: chef: chef_effortless: chef_root: /opt/chef/embedded/lib/ruby/gems/3.0.0/gems/chef-17.5.22/li b version: 17.5.22 ohai: ohai_root: /opt/chef/embedded/lib/ruby/gems/3.0.0/gems/ohai-17.5.2/lib/ohai version: 17.5.2 cloud: local_hostname: ip-172-31-15-141.ec2.internal local_ipv4: 172.31.15.141 local_ipv4_addrs: 172.31.15.141 provider: ec2 public_hostname: ec2-3-85-154-118.compute-1.amazonaws.com public_ipv4: 3.85.154.118 public_ipv4_addrs: 3.85.154.118 command: ps: ps -ef cookbooks: run-chef-client: version: 0.1.0 counters: network: interfaces: eth0: rx: bytes: 396541197 drop: 0 errors: 0 …
Part 8: Extending what you've learned
So far, you've seen how Chef and a single cookbook can do a lot of work for you. In the real world, though, you likely have multiple configurations you want applied, perhaps opening port 443 on your webservers or installing specific applications on other servers.
You can bring together multiple cookbooks in a couple ways, giving you baseline cookbooks or profiles you can apply to groups of nodes you define. As discussed earlier, Policyfiles make bundling many cookbooks, recipes and profiles into a single object simple. Another way is to add cookbook dependencies to a cookbook's metadata.rb file and using the include_recipe resource in a default.rb recipe.
Define a dependency in metadata.rb
Every cookbook has a metadata.rb file that defines its name, version, and any dependencies it might have. To add a dependency on another cookbook – telling it to use your run-chef-client cookbook – you can add a depends entry to metadata.rb:
name 'base_cookbook' maintainer 'Jack Smith' maintainer_email 'jsmith@example.com' license 'Apache-2.0' description 'Installs/Configures base_cookbooks' long_description 'Installs/Configures base_cookbooks' version '0.1.0' chef_version '>= 16.0' depends 'run-chef-client'
Define cookbook dependencies in recipes/default.rb
In Part 3, you saw that cookbooks contain a default.rb recipe that defines which actions should be performed if no other specific recipes are specified. In this base_cookbook example, the cookbook defines recipes to be included from its dependent or related cookbooks. Say you have a second cookbook called closed-ports with its own default.rb in its own cookbook and recipes directory. To invoke the closed-ports cookbook when you apply the run-chef-client cookbook, you can use the include_recipe resource in a modified cookbooks/run-chef-client/recipes/default.rb to read:
# # Cookbook:: base_cookbook # Recipe:: default # # Copyright:: 2021, Jack Smith, All Rights Reserved. include_recipe 'closed-ports::default'
In this way, you can create base cookbooks to include multiple recipes and cookbooks. This is made immutable when you use Policyfiles, which bundle everything together from one or more cookbooks, including any InSpec profiles you've defined. The Policyfile approach is preferred because it doesn't require older Chef techniques to upload cookbook dependencies.
Epilogue: Where do we go from here?
If you’ve read this far, you’re probably anxious to start getting some hands-on experience of your own.
Learn Chef is a great place to get that process rolling, and the various tracks will guide you through basic to advanced Chef techniques.