Test Driven Infrastructure

You can download me here:https://github.com/akranga/devops-hackathon-3

Activity 0Environment Prep- Test Kitchen installation

Gemfile

Activity 0Now we need to install Test Kitchen plugins. Busser can help us

source 'https://rubygems.org'gem 'test-kitchen', '~> 1.0'gem 'berkshelf' , '< 3.0.0'gem 'busser'

group :integration do gem 'bats' gem 'serverspec' gem 'kitchen-vagrant', '~> 0.11'end

$ cd C:\<your-vagrant-dir>\embedded\bin$ gem install bundle --no-ri --no-rdoc

1. Install bundle ruby gem

2. Add C:\<your-vagrant-dir\embedded\bin to your environment variable PATH.

3. Go to activity0 directory and create file Gemfile

Activity 0For this activity we will reuse ruby that comes embedded to vagrant

$ busser plugin listPlugin Versiondummy 0.6.2cucumber 0.1.0serverspec 0.2.6

$ busser plugin install busser-serverspec$ busser plugin install busser-cucumber

5. Run following command

6. Validate plugins setup

4. Run command

$ bundle install$ kitchen --version

Chef Resources

Resource AnatomyChef recipes is the set of declared resources

package "nginx" do version: "1.2.3“ source: "https://..."

action :install

not_if {something}end

Resource is the Server state that you declaring to achieve and Chef will transform it into OS/Arch specific instructions.

Resource type “package” instructs Chef to install package from apt or yum (or similar) manager

Resource Namemeaningful name of

the resource. Always implies value of

“required” resource attribute

AttributesResource key-value pair

Actions (optional)Specifies what to do

with resource. Each resource has

default action Conditional attributesnot_if, only_if,

creates

NotificationsResources can synchronize their actions

remote_file "website.zip" do source "http://....." notifies :run, "execute[uzip website.zip]", :immediatelyend

execute "uzip website.zip" do action :nothingend

After remote ZIP file will be downloaded, resource must notify “execute” to start action “run”. Untill that “execute” will do nothing

template "/etc/netatalk/netatalk.conf" do notifies :restart, "service[afpd]", :delayed notifies :restart, "service[cnid]", :delayedend

Render config file and then restart related services

NotificationsNotification works in two directions

notifies One resource will send notification message to another resource to execute an action

subscribes One resource can “wait listen” another resource until specific action will happen

Notification Timers

:immediately Send notification message immediately after action has been converded

:delayed Send delayed notification.Example: if multiple resources wants to restart service. Then chef will wait for some time and then restart service just once

Basic ResourcesNotification works in two directions

package Actions: [:install, :upgrade, :reconfig, :remove, :purge]Attributes: package_name (default attribute) version (specify version constraints) source (alternative source for package) arch (x86_64, i686)

template Actions: [:create, :create_if_missing, :touch, :delete]Attributes: source (erb file name in /template dir) backup (number of backups) owner (user that will rendered file {chown}) group (user group that ill own the file) righs (0644, 0755 etc. access rights)

Basic ResourcesNotification works in two directions

remote_file Actions: [:create, :create_if_missing, :touch, :delete]Attributes: path (default attribute) source (erb file name in /template dir) backup (number of backups) owner (user that will rendered file {chown}) group (user group that ill own the file) rights (0644, 0755 etc. access rights)

execute Actions: [:run, :nothing]Attributes: command (default attribute) cwd (current working directory for command) user (user that will execute this command) environment (Map of additional ENV variables)

Other resources can be found herehttp://docs.opscode.com/chef/resources.html

Activity 1Create cookbook that manages DB

TODO1 : Berksfile

Activity 1Go to directory activity1

cookbook "apt"cookbook "mysql"cookbook "database"

1. Customize Berksfile.

TODO2 : Vagrantfile chef.add_recipe "apt" chef.add_recipe "mysql::server" chef.add_recipe "myapp_db"

2. Specify Vagrant runlist

Cookbook Database provides DSL to do DB management operations

TODO3 : Vagrantfile

Activity 1Go to directory activity1

"mysql" => { "server_root_password" => "secret"}

3. Specify root password for mysql as Chef attribute

TODO4 : myapp_db/attributes/default.rbdefault.db.name = "world"default.db.user = "chuck"default.db.pass = "topsecret"

4. Now let’s start to write our DB cookbook. And let’s start with Attributes

Cookbook Database provides DSL to do DB management operations

We will create custom database with name “word”. Create DB user Chuck with password

TODO5 : myapp_db/recipe/default.rb

Activity 1Now let’s write recipe that will create DB and populate with data

include_recipe "database::mysql"include_recipe "myapp_db::create"include_recipe "myapp_db::load_data"

5. Our default recipe will be just a wrapper that will coordinate DB operations

TODO6 : myapp_db/recipes/create.rb:password => node.mysql.server_root_password

This recipe will just trigger 3 other recipes. All logic will be encapsulated inside. One will be from community cookbook “database” other two are just about to write

6. Operations with MySQL database will be performed as DB user “root” We need to take it’s password from node attribute

TODO7 : myapp_db/recipe/default.rb

Activity 1

mysql_database node.db.name do connection mysql_connection_info action :createend

7. Add script that will create DB

TODO8 : myapp_db/recipes/create.rbmysql_database_user node.db.user do connection mysql_connection_info password node.db.pass action :createend

Database name we specify in cookbook attributes file. (this name can be overridden at Chef provisioning level (Vagrantfile)

Resource mysql_database have been described in “database” cookbook. Our cookbook have dependency on that (see metadata.rb)

8. Specify User that must be created

TODO9 : myapp_db/recipe/load_data.rb

Activity 1

package "unzip"

Now lets add some logic that will put our DB with data. We will put it into different recipe (load_data.rb).

This recipe will download and unzip SQL from internet and upload to the DB9. We don’t know if unzip is actually installed. So we will put declaration that we expecting it is.

10. Now let’s download file from internet

TODO10 : myapp_db/recipe/load_data.rb"#{Chef::Config[:file_cache_path]}/world_innodb.sql.zip"

/tmp is not good directory. See Food Critic FC013http://acrmp.github.io/foodcritic/#FC013

TODO11 : myapp_db/recipe/load_data.rb

Activity 1

execute "unzip -o #{Chef::Config[:file_cache_path]}/world_innodb.sql.zip" do creates "#{Chef::Config[:file_cache_path]}/world_innodb.sql"end

Now let’s unzip the file. We need to be sure that we will unzip it just once. Creates attribute will help us

11. We don’t know if unzip is actually installed. So we will put declaration that we expecting it is.

12. Last but not least. We need to read file that have created and pass it to the mysq_database resource with action :query

TODO12 : myapp_db/recipe/load_data.rbsql { ::File.open("#{Chef::Config[:file_cache_path]}/world_innodb.sql").read }

If you know ruby then it will be very familiar string

Activity 1

$ vagrant destroy –force$ vagrant up$ vagrant ssh

Now we are ready to start convergence

12. Refresh VM (if you have old one)

13. Let’s do some manual tests

vagrant $ mysql --versionvagrant $ netstat -atn | grep 3306tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTENvagrant $ ps –ef grep | mysqlmysql 5345 1 0 11:13 ? 00:00:04 /usr/sbin/mysqldvagrant $ mysql –u chuck –ptopsecret –D worldmysql> select * from City limit 0,10;+----+----------------+-------------+---------------+------------+| ID | Name | CountryCode | District | Population |+----+----------------+-------------+---------------+------------+| 1 | Kabul | AFG | Kabol | 1780000 || 2 | Qandahar | AFG | Qandahar | 237500 |...

Activity 2Enable Test Kitchen suite

Activity 2Go to /activity2/myapp_db

.kitchen/ logs/kitchen.logtest/integration/default.kitchen.yml

$ kitchen init --driver=kitchen-vagrant

1. First we must bootstrap test kitchen

2. Explore new files appeared

.kitchen.yml describes test suite to execute and backed by Vagrant

example

Activity 2: explain .kitchen.ymldriver Default “vagrant”. Test kitchen will be using

Vagrant as container management tool. Shipped with gem: kitchen-vagrant

provisioner Default “chef_solo” (simpliest). Tells how to run Chef. Other viable option “chef_zero”

platforms List of operating systems to support. Platforms may vary (OS version, 32/64bit, cloud env etc)

suites Specifies run lists for test suite.

---driver: name: vagrantprovisioner: name: chef_soloplatforms: - name: ubuntu-13.04suites: - name: default run_list: - recipe[mysql::server] attributes:

.kitchen.yml

Activity 23.a. Customize .kitchen.yml. For now leave just one platform

platforms: - name: ubuntu-14.04

3.b. In the same file customize chef runlist and add attributes for mysql cookbook

.kitchen.ymlsuites: - name: default run_list: - recipe[apt] - recipe[mysql::server] - recipe[myapp_db::default] attributes: mysql: server_root_password: "rootpass"

Activity 24. Validate platform

$ kitchen listInstance Driver Provisioner Last Actiondefault-ubuntu-1404 Vagrant ChefSolo <Not Created>

5. Proceed with test kitchen creation$ kitchen create default-ubuntu-1404-----> Starting Kitchen (v1.0.0)-----> Creating <default-ubuntu-1404>...

This takes too long (downloads box file). Interrupt it (CTRL+C).

Activity 26. Add vagrant box from file cached locally

$ vagrant box add opscode-ubuntu-14.04 C:/.../opscode_ubuntu-14.04_chef-provisionerless.box

$ vagrant box listopscode-ubuntu-14.04 (virtualbox)

7. Now let’s start the action (be careful with VM naming)

$ kitchen create default-ubuntu-1404-----> Starting Kitchen (v1.2.1)-----> Creating <default-ubuntu-1404>... Bringing machine 'default' up with 'virtualbox' provider... [default] Importing base box 'opscode-ubuntu-14.04'...... Finished creating <default-ubuntu-1404> (0m53.91s).

$ kitchen listInstance Driver Provisioner Last Actiondefault-ubuntu-1404 Vagrant ChefSolo Created

Berksfile

Activity 28. Run test suite (it should fail)

$ kitchen converge default-ubuntu-1404>>>>>> Converge failed on instance <default-ubuntu-1404>.>>>>>> Please see .kitchen/logs/default-ubuntu-1404.log for more details

8. Check log file (as described in error message) or manually inspect VM with command$ kitchen login default-ubuntu-1404$ sudo less /tmp/kitchen/cache/chef-stacktrace.out

HINT: Chef cannot find community cookbooks.

9. Create Berkshelf file to deliver cookbooks

site :opscode

cookbook "apt"cookbook "mysql"cookbook "database"cookbook "myapp_db", path: "."

Activity 2

10. Run convergence

$ kitchen convergeChef Client finished, 39/45 resources updated in 396.203151814 secondsFinished converging <default-ubuntu-1404> (6m45.29s).

10. We almost there… Add vagrant-cachier support. Modify Vagrantfile

$ kitchen diagnose...vagrantfile_erb: ...gems/kitchen-vagrant-0.15.0/templates/Vagrantfile.erb...

11. We will specify custom template for Vagrantfile that will have enabled vagrant-cachier plugin. Updated Vagrant.erb has been available from the git. So you just need t oupdate .kitchen.yml file

.kitchen.yml---driver: name: vagrant vagrantfile_erb: Vagrantfile.erb

Activity 211. Now let’s do a manual test

$ kitchen login...vagrantfile_erb: ...gems/kitchen-vagrant-0.15.0/templates/Vagrantfile.erb...

12. Looks familiar isn’t it?

vagrant $ mysql --versionvagrant $ netstat -atn | grep 3306tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTENvagrant $ ps –ef grep | mysqlmysql 5345 1 0 11:13 ? 00:00:04 /usr/sbin/mysqldvagrant $ mysql –u chuck –ptopsecret –D worldmysql> select * from City limit 0,10;+----+----------------+-------------+---------------+------------+| ID | Name | CountryCode | District | Population |+----+----------------+-------------+---------------+------------+| 1 | Kabul | AFG | Kabol | 1780000 || 2 | Qandahar | AFG | Qandahar | 237500 |...

Activity 3Write Automated tests

BEFORE START!!!Read about Ice-Cream anti-pattern: http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/

Test Kitchen

<COOKBOOOK-PATH>/test/integration/<TEST-SUITE-NAME>/<PLUGIN-NAME>

Test kitchen finds tests by following convention

Test Kitchen plugins

chefspec Unit testing framework that runs cookbook locally without actually converging it

bats Bash Automated Testing System. Very simple and easy to use. Great for quick-start but not saleable enough.

S erverspec Bash Automated Testing System. Very simple and easy to use. Great for quick-start but not saleable enough.

Prep

<COOKBOOOK-PATH>/test/integration/<TEST-SUITE-NAME>/<PLUGIN-NAME>

Busser is a Test suite adapter for Test Kitchen. It provides an abstraction between test suite and testing framework itself (plugin)

Test Kitchen plugins

chefspec Unit testing framework that runs cookbook locally without actually converging it

bats Bash Automated Testing System. Very simple and easy to use. Great for quick-start but not saleable enough.

serverspec Bash Automated Testing System. Very simple and easy to use. Great for quick-start but not saleable enough.

cucumber Acceptance Spec framework. A very nice way how to formulate implementable requirements

Activity 3Go to /activity3/cookbooks/myapp_db

$ busser plugin listPlugin Versionserverspec 0.2.6

1. Check if you have serverspec plugin installed

or$ gem list | grep busser-serverspec

If you wish to uninstall busser plugin. You should do it via gem uninstall command

$ kitchen destroy

2. We can clean up our test kitchen (OPTIONAL)

Activity 3

myapp_db\test\integration\default\serverspec\mysql_spec.rb

3. Create serverspec file

This means we will write an integration based on “serverspec” for test suite with the name “default”

Activity 3mysql_spec.rb

require 'serverspec'require 'pathname'

include Serverspec::Helper::Execinclude Serverspec::Helper::DetectOS

RSpec.configure do |c| c.before :all do c.os = backend(Serverspec::Commands::Base).check_os c.path = '/sbin:/usr/sbin' endend

describe service('mysql') do it { should be_enabled } it { should be_running }end

describe port(3306) do it { should be_listening }end

Activity 3

$ kitchen converge$ kitchen verifyService "mysql" should be enabled should be running

Port "3306" should be listening

Finished in 0.10853 seconds3 examples, 0 failures

4. Run test suite

$ kitchen listInstance Driver Provisioner Last Actiondefault-ubuntu-1404 Vagrant ChefSolo Verified

5. Check our VM

Retrospective