An Introduction to Chef

specifically v12.18

What Is Chef ?

Chef is a platform designed to facilitate the automated configuration and management of infrastructure via code; a capability commonly referred to or defined as infrastructure as code (IaC).

There are many components and aspects to Chef as a production IaC platform. To become familiar with the basics, the following might be considered the lowest common denominators that other Chef tooling and innovation is derived from:

• Chef Recipe DSL
• Chef Cookbooks
• Chef Client
• Chef Server

Chef Recipe DSL

The Chef Recipe DSL (Domain Specific Language) is based on the Ruby programming language. Ruby is a popular general purpose, object oriented language with a large ecosystem. As the Recipe DSL is based on Ruby, Ruby language functionality is available to use by default. The fundemental purpose of the Recipe DSL is to provide a means to apply policy against an instance of infrastructure.

Attributes

The Recipe DSL includes an Attributes concept, where attributes are specific details about a node (or instance of infrastructure) the Recipe DSL is being run against. Attributes are variables that are defined by configuration mechanisms such as Roles and Environments, declared in Cookbooks and are prepopulated by the Chef Client executing the DSL on a node. A tool called Ohai is used to provide the Chef Client properties of the node on which it is executing. Information derived by Ohai is made available in the node attribute by the Chef Client. Ohai extracts information using plugins, with an example being the hostname plugin.

A few examples of the information collected by Ohai include:

• Host name
• CPU data
• Network configuration
• Platform details

The data Ohai collects can be extended by adding additional plugins to those installed by default.

The node attributes available via Ohai may be utilized in a recipe, for example:

if node['platform'] == 'debian'
  # do something
end

Recipe DSL Methods

Recipe DSL Methods are provided to facilitate Chef platform functionality such as storing (encrypted) data and searching for things such as other instances of infrastructure currently being managed. Recipe DSL Methods are also provided to interact with and modify resources in addition to interacting with attributes. Four helper type methods are provided.

As an example of a helper method, one can set the package name for Apache based on the node[‘platform’] attribute, using the value_for_platform helper method:

#example from chef docs
package_name = value_for_platform(
  ['centos', 'redhat', 'suse', 'fedora' ] => {
    'default' => 'httpd'
  },
  ['ubuntu', 'debian'] => {
    'default' => 'apache2'
  }
)

If the platform is centos, package_name will be set to httpd.

Resources

A resource block defines a policy requirement. Consider a System V service configuration for an application such as Nginx (a multi purpose Webserver), managed as a Recipe DSL resource of the type Service. Configuration options are available to define the behaviour of the Nginx service. For example, to configure the service to start on boot we can declare the following Service Resource block in a recipe:

service 'nginx' do
  action [:enable]
end

The Chef Recipe DSL currently utilises a method_missing Ruby feature to provide a mechanism to declare Resource configurations in an easy to read way, as above. The method_missing implementation is scheduled to be removed in Chef 13.

The following is equivalent to the previous Nginx Service resource example, using the declare_resource Recipe DSL method:

declare_resource(:service, 'nginx', caller[0]) do
  action :enabled
end

If you are wondering about :string and ‘string’ syntax in the Recipe DSL (Ruby), I recommend reading about symbols and strings in Ruby.

Other Recipe DSL functionality

The Recipe DSL provides an event mechanism that allows for actions to be executed when an event is triggered. For example, an email alert could be sent if the run_failed event is triggered.

include_recipe is provided allowing a recipe from either the current cookbook or an external cookbook to be executed.

To include the recipe named “recipe” from the cookbook named “cookbook”:

include_recipe 'cookbook::recipe'

Chef Cookbooks

A cookbook is a structured collection of configuration and policy. Cookbooks may consist of several types of configuration and policy.

The available types of configuration and policy are as follows:

metadata.rb

Cookbooks are required to provide metadata which is supplied via a file named metadata.rb, required to be at the top level of a cookbooks directory structure. The absolute minimum requirement for the metadata supplied is name as can be seen in Chef::Cookbook:Metadata::run_validation. metadata.rb is (perhaps unsurprisingly) expected to be valid Ruby. Chef::Cookbook:Metadata hydrates state from metadata.rb via Chef::Mixin::FromFile. The hydration works using instance_eval, a Ruby metaprogramming facility which will evaluate the Ruby passed in the context of the object specified, in this case an instance of Metadata.
It is likely that a Cookbook will depend on other Cookbook Recipes for functionality. Cookbook dependencies can be configured via the depends config setting. Additionally, in practice, versioning of Cookbooks may end up being essential to control which iteration of a cookbook is required and evaluated when a client run occurs. The metadata.rb documentation provides an explanation of all settings.

Custom Resources

Custom resources may be specified in a cookbooks /resources directory. Chef includes many resources by default, such as the Service Resource mentioned previously. Custom resources allow the encapsulation of policy requirements not provided by default via the specification of attributes and the utilisation of the actions required. Custom resources can be used in recipes as per the resources provided by default, where the custom resource is named after the cookbook providing it and the name of the custom resource file seperated by an underscore. For example mycookbook_customresource. Resource documentation.

Custom Providers

Custom providers may be specified in a cookbooks /providers directory. Custom providers can be provided to define how actions are interpreted and applied during a client run, both via interaction with core resources and/or via Ruby directly. Custom Light Weight Resources and Providers documentation.

Recipes

Recipes provide the instructions and logic required to implement an instance of policy, such as the installation of software and provide this ability via utilising the Recipe DSL. Recipe documentation

Libraries

Libraries are a facility that enable the inclusion of Ruby programming in a cookbook. This ability provides the opportunity to write extended functionality that may be reusable. Libraries documentation

Attributes

Attributes are variables that may be defined in a cookbook by multiple sources including a cookbook attribute file, a recipe, an environment, a role. Attribute precedence rules define the order by which attributes are applied by the Chef Client. Attributes documentation

Files

The files directory in a cookbook provides a place to keep any files that will or could be copied to the node during a Chef Client run. Files can be copied using the cookbook_file resource provided by default. Files documentation

Templates

The templates directory in a cookbook provides a place to keep any templates that might be used. Chef Client uses an Embedded Ruby templating system to allow the usage of Ruby in plain text files to dynamically determine content. Template documentation

Chef Client

Typically a Chef Client is responsible for pulling policy configuration from a Chef Server and applying it to the instance of infrastructure on which it is running. Ordinarily a Chef Client will either run on a node as a daemon or be scheduled to run via crond on the node.

In a default configuration, the chef-client runs through 3 main phases, with multiple steps per phase, documented in the codebase. For clarity the precise implementation referred to here is the 2.18.19 tag of lib/chef/client.rb and excludes usage mechanisms such as Chef Solo or audit mode.

Setup Phase

• Run Ohai to collect system information

  • Executes Ohai plugins reporting node information

• Register / connect to the Chef server

  • If the client_key configuration option in the client.rb config is unspecified or the specified client key file does not exist, registration with the Chef Server is skipped, otherwise the node is registered with the Chef server. node_name is required to register with the Chef server and is determined via the expression “Chef::Config[:node_name] || ohai[:fqdn] || ohai[:machinename] || ohai[:hostname]”.
    To register the client with the server, an instance of Chef::ApiClient::Registration is created and run executed. This method “Runs the client registration process, including creating the client on the chef-server and writing its private key to disk.” On consequent runs, the client can use the private key issued to attempt to authenticate against the Chef server.

• Retrieve the node state from the Chef server

  • Chef::PoloicyBuilder::Dynamic::load_node is executed. “Loads the node state from the server, then picks the correct implementation class based on the node and json_attribs”. If the node is not found on the Chef server a node is created.

• Merge in json_attribs, Chef::Config.environment, and override_run_list

  • In both PolicyBuilder plugin implementations (PolicyFile and ExpandNodeObject), default and override attributes are cleared, “so that any deleted attributes are still gone”.
    External attributes “(e.g., from JSON file, environment, policyfile, etc.)” are applied and the correct expanded run list for the run is determined by Chef::RunList::RunListExpansion.

Compile Phase

• Sync required cookbooks to the local cache

  • The expanded run list previously determined is used to fetch and validate Cookbooks from the Chef server. An instance of Cookbook Compiler is created.

• Load libraries from all cookbooks

  • compile_libraries “Loads library files from cookbooks according to #cookbook_order”.

• Load attributes from all cookbooks

  • compile_attributes “Loads attributes files from cookbooks. Attributes files are loaded according to #cookbook_order; within a cookbook, +default.rb+ is loaded, first, then the remaining attributes files in lexical sort order.”

• Load Lightweight Resources and Providers (LWRPs) from all cookbooks

  • compile_lwrps “Loads LWRPs according to #cookbook_order. Providers are loaded before resources on a cookbook-wise basis.”

• Load resource definitions from all cookbooks

  • compile_resource_definitions “Loads resource definitions according to #cookbook_order”.

• Load recipes in the run list and the command line

  • compile_recipes “Iterates over the expanded run_list, loading each recipe in turn”

Converge (or Execute) Phase

• Converge the resources built from recipes in Compile Phase, bringing the node up to date.

  • A Runner instance is created with the RunContext. It then “Iterates over the ResourceCollection in the RunContext calling +run_action+ for each resource in turn.

• Save the node

  • The Chef server is updated via a Post request with a serialisation of the node object.

• Reboot if we were asked to

  • Chef Platform Rebooter checks if the Node RunContext includes a request to reboot the node, and if so, executes a shutdown with configuration such as delay and reason.

Chef Server

The Chef Server provides Chef Clients configuration details when requested, where the processing of configuration is handled by Chef Clients, allowing the distribution of the processing workload over Chef Clients, reducing the Chef Server workload and aiding availability.