414 Tools for loT 11.1 Introduction Managing JoT infrastructure and configuring and integrating various components can be complex and challenging task, ‘The complexity of infrastructure grows with increasing ‘number of components (such as load balancers, application server, database servers, etc.) 10 minimize the manual effort required, a new paradigm of infrastructure-as-a-code has been popularized by infrastructure automation and configuration management tools such &s Chef and Puppet. Tn the infrastructure-as-a-code paradigm, the computing, storage and nctwork infrastructure is modeling using declarative modeling languages. A modular approach is adopted for ‘modeling the infrastructure to improve code re-usability. The infrastructure models are ‘compiled and run by infrastructure automation tools to generate the desired infrastructure. Infrastructure-as-2-code improves the repeatability of the infrastructure as the same code ‘always produces the same infrastructure, Modular code design along with the automation capabilities improve the scalability of systems. Moreover, in the event of system failures or catastrophic events, the entire infrastructure can be restored from the infrastructure code, In this chapter you will lear about two popular infrastructure automation tools - Chef and Puppet. Case studies on using these tools for generating infrastructures such a8 @ ttree-tier deployment, a Hadoop cluster and Storm cluster are described. This chapters also provides case studies on [oT device management with NETCONF and YANG. 11.2 Chef Chef is an infrastructure automation and configuration management framework. Chef adopts the infrastructure-as-a-code paradigm and allows deploying, configuring and integrating various infrastructure components. Figure 1.1 shows the components of the Chef framework. “The Chef server stores the information about the infrastructure. The infrastructure code is organized into cockbooks. Cookbooks include recipes (which are written in Ruby language), templates, auributes and resources. The fundamental unit of configuration in a recipe is « resource (such as file, package, user, ei.). Recipes specify which resources to manage, how to manage the resources and also the order in which the resources should be managed. Chef recipes and cookbooks are authored on a Chef workstation and uploaded to the Chef server. The nodes to be managed run the chef-client. The chef-client connects to the server and obtains information on the desired state for the node. The chef-client performs ‘various tasks such as building the node object, synchronizing cookbooks and applying the cookbooks to bring the node to the desired state. ‘Chef server comes in two flavors - Enterprise Server and Open Source Server. You can setup either version on your own infrastructure (in-house or in the cloud) or use a Hosted Banga & Madisetl, © 201511.2 Chet a6 Enlerprise Chef Server. Chef provides « free trial of the Hosted Enterprise Chef Server ‘which is a convenient way to get started with Chef. Figure 11.2 shows a screenshot of the Hosted Chef Server. Nodes Chet Server Cookbooks Figure 11.1: Chef components Before we proceed with examples of using Chef, let us briefly look at the key concepts of the Chef framework: 4 Server: Server stores all the configuration data for the infrastructure including cookbooks, recipes, roles, attributes, ran lists and node objects. ‘¢ Node: Node can either be a cloud-based virtual machine instance, a physical node, a virtual node ora network node (such as a switch or router). ‘* Cookbooks: Cockbook is a collection of recipes, attributes, templates and resources. ‘« Recipe: Recipe is a configuration element written in Ruby language that specifies ‘various resources to be managed and how to manage the resources. pea ies ee Intornet of Things -A Hands-On Approach| thank you for choosing Enterprise Chet eal) nummy | Fotis renga on ore tg Ener St a ee mete Sar Setupyox worn Som ate a ee Figure 11.2: Screenshot of dashboard of hosted Chef server ‘e Resource: Resource is a fundamental unit of configuration ( such as a package, fit, ‘user, etc). Recipes include information on resources to manage and the desired sate of the resources. «« Provider: While resource specification tells which resource to manage and the deste state of the resource, the specification is abstract in nature and does not describe the steps to manage the resource. Provider describes the steps to bring the resource tothe desired state. ‘e Attributes: Attributes are used to provide specific details. Attributes are included in cookbooks, roles, environments and node objects. «¢ Templates: Templates are included in cookbooks and are used for complex configurations. Templates are written in ERB template language which isa feature of Ruby. «« Policy: Policy includes roles, environments and data-bags. Roles define the types of servers or patterns that should be applied to all the nodes in a role, For example, all nodes in a role - “Application Server” have the same configuration details (of fan application server). Data-bags are used to store sensitive information such a8 | ‘usemames and passwords. Information stored in dala-bags can be acoessed by nodes authenticated to the Chef server. Environments denote the processes and worfiows (e.g. development, staging, production, ete.) «¢ Run Lists: Run list is an ordered list of recipes and/or roles. The chef-client applies the recipes and roles in the run list in the order in which they appear, Ru lists are stored on the Chef server as a part of the node object. « Knife: Knife is a command line utility that provides interface between workstation and server, Using knife you perform tasks such as creating and uploading cookbooks, Bahga & Madisett, © 201511.3 Chef Case Studies a7 creating roles and environments, bootstrapping nodes, etc, 11.2.1. Setting up Chet ‘To set up a Chef environment, you will need to set up Chef Server, Workstation and CChef-client on nodes. For Chef Server, you can either setup the Open Source version on your ‘own node or use Hosted Enterprise Chef. For the examples in this Chapter, we recommend signing up for a free trial of Hosted Enterprise Chef, Whea you create an account on Hosted Enterprise Chef, you will be able to download a starter kit that includes the PEM certificates that allow the workstation to authenticate with the Chef server. Setting up Chef on the ‘workstation is as simple as running a single command as follows (for Linux workstation): ‘When you run the above command, Chefs omnibus installer installs all you need to get started with Chef. ‘With the Server and Workstation set up, the next step is to set up a node wo manage with Chef, You can use cloud-based node, or a physical or virtual node, For the examples in this "chapter We use Amazon EC2 instances as nodes. a & “After launching a new Amazon EC2 instance, note the IP address of the instance. The " next step is to bootstrap the node. The bootstrapping process installs the Chef client and ‘checks in with the Chef server. To bootstrap an Amazon EC2 node, run the following knife ommand on the Workstation: "You will require your EC2 keypair (PEM) file on the Workstation for bootstrapping the ‘node. The node name that you specify at the time of bootstrapping will be used in further ‘interactions, You can also view the node with the same name in the Hosted Enterprise Chef ‘dashboard. 11.3 Chef Case Studies 11.3.1 Multttier Application Deployment ~ Inthis section you will leam how to create a multi-tier deployment comprising of HAProxy Joad balancer, Django application server and MongoDB database server. Figure 11.3 shows the steps in creating @ three-tier deployment with Chef. In the first step the nodes for the load balancer, application server and database server are provisioned. Next, the software ener tee Internet of Things - A Hands-On Approach418 Tools for loT ‘packages for HAProxy, Django and MongoDB are setup on the respective nodes. Finally th nodes are integrated to setup the three-tier ‘The first step is to create a cookbook named threetierde ployment with the following. ‘command: HAProxy, Django and MongoDB. Box 11.1 shows the recipe generating se Siabesnalsatl te csaeag es paauaaertinea aati suthorized key fle on each node. This ensures that ll nodes are able to securely connect each other.11.8 Chef Case Studies 419 44.3 ChefCase Studies 000 Box 11.3 shows the recipe for seiting up HAProxy, The recipe defines a package resource ‘name ‘haproxy’ with the action for the resource set to ‘install’. The recipe also defines template resource for the haproxy.cfg file. The ERB template file for haproxy.efg is shown in Box 114. pr a le ae Internet of Things - A Hands-On ApproachInternet of Things - A Hands-On ApproachBox 11.5 shows the recipe for seuing up Django. Box 11.6 shows the recipe for setting up MongoDB.11.9 Chef Case Studies 423 Aer reatng te recipes, the aa sip ist plond the cookbook. To upload the “The next step is to create run lists forall the nodes as follows: lly, the chef-client is run on all nodes from the workstation as follows:424 11.3.2. Hadoop Cluster ‘This section describes a case study on setting up a Hadoop cluster using Chef. You al Jeamed about the steps in setting up a Hadoop cluster in Chapter 10. Let us build on: knowledge. Figure 11.4 shows the steps involved in setting up a Hadoop cluster with Chef In the first step the instances for Hadoop master and slave nodes are provisioned. Hadoop package is setup on the master and slave nodes. Finally, the Hadoop files are updated and the nodes are integrated to setup the cluster. ‘The first step is to create a cookbook named hadoop with the following command: Next, launch two Amazon EC2 nodes (one master and one slave node) and bo ‘them using the following commands from the workstation: ‘fier bootstrapping the nodes, create recipes for generating SSH keys, setiing up hos ‘installing Java and setting up Hadpop. Box 11.1 shows the recipe for generating 8 and Box 11.2 shows the recipe for collecting the public keys of all nodes and. authorized key file on each node. This ensures that all nodes are able to securely co cach other. Box 11.7 shows the Chef recipe for setting up hosts. In this recipe the ‘ete is created using the template in Box 11.8. All IP acdresses and host names of n Chef role (*hadoop_cluster_role”) are added to the hosts file,Figure 11.4; Setting up Hadoop cluster with Chef‘The recipe for installing Java-7 is shown in Box 11.9.: i : 5 8 Box 11.10 shows the Chef recipe for setting up Hadoop. In this recipe. the Hadoop package is downloaded and Hadoop configuration files (hafs-site.xml, core-site.xml and Internet of Things - A Hands-On Approach428 ‘mapred-site.xml) are created using the templates shown in Boxes 11.11, 11.12 and. 11. Bahga & Madisett, © 2015. 411.3 Chef Case Studies _ntomot of Things - A Hands-On ApproachBox 11.14 shows Chef recipe for setting up Hadoop master node. In this recipe | ‘master and slave configuration files are created and the hostnames of the master und are added to these files. Boxes 11.15 and 11.16 show the master and slave templates used for configuring the Hadoop master node. ;11.3 Chef Case Studies 431 Box 11.17 shows the attributes for the Hadoop cookbook. These attributes are used in ‘the recipes in the cookbook. Boxes 11.18 and 11.19 show the Chef roles for soting up Hadoop cluster and Hadoop master node. The roles contain definitions ofthe run lists. Internet of Things -A Hands-On Approach‘WSChetCaseStudies 1133 ‘Storm Cluster Tn this section you will leam how to setup an Apache Storm cluster with Chef. "The first step is tocreate a cookbook named hadoop with the following command: Dare ae eee et bootstrap them using the following commands from the workstation: After bootstrapping the nodes, create recipes for generating SSH keys, setting up hosts, aa oa pose wetter Sonne ae eatng Internet of Things-AHands-On Approach434 Tools for oT Box 11.21 shows the recipe for setting up Storm. In this recipe, the dependencies are first installed, Next, the packages for ZeroMQ and JZMQ are installed. The Storm package is then downloaded and installed.11.3 Chef Case Studies 435 Internet of Things - A Hands-On Approach“With the all the recipes for the Storm cookbook now created the next step is to define roles, Boxes 11.22, 11.23 and 11.24 show the Chef roles for setting up a cluster for Storm, setting up Zookeeper and setting up Storm respectively. :11.3 Chef Case Studi ee eg Hibeaaieiieiina i ame Ieee tenes ‘The roles are then added to the run lis of the Nimbus, Zookeeper and Supervisor nodes as follows: Finally, the chef-client is run on the nimbus, zookeeper and supervisor nodes (from the workstation) as follows: en {Internet ot Things - A Hands-On Approach438 Tools for loT 11.4 Puppet Puppet (like Chef) is also a configuration management tool that can be used to manage configurations on a variety of platforms. Figure 11.5 shows the deployment model for Puppet. Puppet is usually deployed in a client-server model. The server runs the Puppet Master and the client runs the Puppet Agents. Like the Chef-server, the Puppet Master maintains the configuration information for the clients. The puppet agents connect to the master to obtain ‘information on the desired state. Puppet agents on the clients make changes if the current state is different from the desired state, Puppet agents can be configured to automatically check for new or updated configuration from the master at regular intervals. Like Chef, Puppet also uses a declarative modeling language for defining the configurations. Let us now look at the key concepts of Puppet: # Resource: Resource is a fundamental unit of configuration, For example, file, user, package, service, etc. Similar resources are grouped together into resource types. ‘* Resource Abstraction Layer (RAL): Resource descriptions in a configuration are abstract in nature and not tied to a specific OS. RAL allows separation of resource descriptions from their implementations. RAL consists of high-level modules (types) and platform-specific implementations (providers). + Class: Classes define a collection of resources which are managed together as a single unit. ‘© Manifest: Manifests are Puppet programs (with .pp extension). Manifests include ‘various types of logic such as resource descriptions, classes, conditional statements, etc. Manifests can be applied using the ‘puppet apply’ command, which enforces the desired state as defined in the manifest file. ‘© Module: Instead of defining the entire logic in a single manifest fle, Puppet allows you to split the logic in multiple files which are organized as a module, A module Consists of multiple files containing the class definitions. Classes group the resource definitions. The classes defined in modules can be included in the manifest file which Bahga & Madisatt, © 2015