technical university/symposia materials may not be reproduced in whole or in part without the prior...

44
Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM Deep Dive Tips & Tricks Benoît Créau Aix System Engineer @chmod666 chmod666org http://chmod666.org [email protected] rg

Upload: jonas-park

Post on 17-Jan-2016

237 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.9.0

PowerVC for PowerVM Deep DiveTips & Tricks

Benoît CréauAix System Engineer

@chmod666 chmod666org http://chmod666.org [email protected]

ben
Page 2: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

whoami

• AIX/PowerVM System Engineer working in Large french banks• Passionate about AIX and Power Systems• Blogger since Feb 2010

– http://chmod666.org

• Creating tools for AIX an Power– Github : chmod666org

• Sharing my daily life as a sysadmin on Twitter (tips and tricks, blog post)– @chmod666

• Recently Redbook author: PowerVC 1.2.3 Introduction and configuration (www.redbooks.ibm.com/redpieces/abstracts/sg248199.html?Open)

© Copyright IBM Corporation 2015 2

Page 3: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Agenda

• Part 1 - Network management• Part 2 - Storage management• Part 3 - Datasource/activation• Part 4 - Disks PVID • Part 5 - cloud-init • Part 6 - sys0 attributes ghostdev/clouddev• Part 7 - Using openstack commands • Part 8 - Rest api• Part 9 - Linked clones• Part 10 - Dynamic Resource Optimizer

© Copyright IBM Corporation 2015 3

Page 4: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

PowerVC network management

• Shared Ethernet Adapters:– PowerVC add and remove vlans from the Shared Ethernet Adapters.– If a VLAN is needed on a host PowerVC add the VLAN on the Shared

Ethernet Adapter:• At deploy time• At migrate time (Live Partition Mobility)• At restart time (Remote Restart Operation)

– If a VLAN is not needed anymore PowerVC removes the VLAN from the Shared Ethernet Adapter:• Same conditions as above (deploy, migrate, restart)

• PowerVC can create Virtual Ethernet Adapters and add it in the Shared Ethernet adapters if needed

• PowerVC can remove Virtual Ethernet Adapter from the Shared Ethernet Adapter

© Copyright IBM Corporation 2015 4

Page 5: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

PowerVC Network Management : Forget your conventions

• Don’t prepare a slot numbering convention as PowerVC can add or remove the Virtual Ethernet Adapters

• Advice: Do not create SEA with more than two VEA.

Automatically added by PowerVC (PVID 4094) No custom buffers on

this interface

Output of lsseas korn shell script (https://github.com/chmod666org/lsseas)

© Copyright IBM Corporation 2015 5

Page 6: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

PowerVC network management : SEA buffers

• Using the chdef way:# chdef -a min_buf_tiny=4096 -c adapter –s vdevice –t IBM,l-lan# chdef -a max_buf_tiny=4096 -c adapter –s vdevice –t IBM,l-lanSame for min_buf_small=4096 max_buf_small=4096 min_buf_medium=512 max_buf_medium=512 min_buf_large=128 max_buf_large=128 min_buf_huge=128 max_buf_huge=128

• Using the Virtual I/O servers rules ( >= 2.2.4):# rules -o modify -t adapter/vdevice/IBM,l-lan -a min_buf_tiny=4096# rules -o modify -t adapter/vdevice/IBM,l-lan -a max_buf_tiny=4096# rules –o list[..]adapter/vdevice/IBM,l-lan max_buf_tiny 4096adapter/vdevice/IBM,l-lan min_buf_tiny 4096# rules –o diff –s# rules –o deploy[1] Warning #7 (0590-206 A manual post-operation is required for the changes to take effect): Please reboot the system# shutdown -force -restart

WARNING:

Some attributes

are today m

issing

in vios rules

© Copyright IBM Corporation 2015 6

Page 7: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Disabling vlan addition/removal (automated_powervm_vlan_cleanup)

• PowerVC 1.2.3.1/2 bug: PowerVC forget to keep at least two VEA for Shared Ethernet Adapter in sharing mode

• Some customers wants to disable this feature:– Keep their numbering convention– Avoiding removal of the next to last VEA

• Using openstack-config:

# openstack-config --set /etc/nova/nova.conf DEFAULT automated_powervm_vlan_cleanup False# openstack-config --set /etc/nova/nova-828642A_10D6D5T.conf DEFAULT automated_powervm_vlan_cleanup False

• Modifying nova.conf :# grep automated_powervm_vlan_cleanup /etc/nova/nova.confautomated_powervm_vlan_cleanup = False# grep automated_powervm_vlan_cleanup /etc/nova/nova-9117MMD_659C5FF.confautomated_powervm_vlan_cleanup = False

WARNING: don’t

forget the change

for existin

g hosts !

© Copyright IBM Corporation 2015 7

Page 8: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Disabling vlan addition/removal (automated_powervm_vlan_cleanup)

• Any configuration files modification = RESTART POWERVC# powervc-services restart

© Copyright IBM Corporation 2015 8

Page 9: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

VLANs: be careful when adding new hosts!

• Adding a host on PowerVC does not automatically set which SEA will be utilized for a particular VLAN.

• When adding a host, be carefull modifying all your existing VLAN to match the SEA configuration !

• PowerVC will use the first founded one !• This can result in propagating a VLAN in the wrong network

vswitch (deploy, LPM errors)

Wrong vswitch

Wrong SEA

© Copyright IBM Corporation 2015 9

Page 10: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Zoning: How many zones will be created

• Number of zones created depends of:– Storage connectivity group (easy to understand)

• NPIV Fabric Access Requirement.

• VIOS Redundency for volume Connectivity

– Storage Templates:• Use all available WWPNs for attachement (very very important option)

( next slide)

This will decide how many ports will be

zoned on your storage array

© Copyright IBM Corporation 2015 10

Page 11: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Zoning: « Use all available WWPNs » (SVC)

• Option not checked– Both nodes will be zoned but not for each WWPNs– One zone per WWPN– For instance 4 NPIV adapters = 4 zones– No spof / Multi fabric– Warning on the SVC

Best practice on AIX: Use 4 or 8 paths, exceptionally 16 (too much paths can result in huge performance issues)

© Copyright IBM Corporation 2015 11

Page 12: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Zoning: « Use all available WWPNs » (SVC)

• Option checked– Both nodes will be zoned for each WWPNs– Many zones per WWPN– All SVC available ports will be zoned– No spof / Multi fabric– Too many zones ?

© Copyright IBM Corporation 2015 12

Page 13: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Zoning: FE Ports spreading

• By using a algorithm based on:– The number of ports that can be zoned– The name of the Virtual Machines

• PowerVC will spread the zoning on each available ports:

Virtual Machine 1 Virtual Machine 2

• In version <= 1.2.3.2 PowerVC just use the four first ports available on the SVC.

• Ask for patch, open a PMR to get it.© Copyright IBM Corporation 2015 13

Page 14: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Storage: Good to know

• Machines with no storage templates (adopt using « manage existing » button):– Use the default storage template (depending on the storage provider)– When you’ll add a new LUN (from a different storage template) it

adds new zones to an existing VM

• Be very careful when creating VMs:– PowerVC can create new zones on a existing VM but can not remove

a part of this zones (ie. all zones will be deleted at the VM deletion, but you can’t remove zones if you made a mistake when choosing a storage connectivity group or a storage template).

– If you notice that you choose the wrong storage template for a VM you can add a new lun in the new storage template and all the existing lun will use the new zones (just cfgmgr).

© Copyright IBM Corporation 2015 14

Page 15: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Openstack data-source used by PowerVC 1/4

1. The user enter ip address, the hostname, … in the GUI

2. A iso image is created on the PowerVC host (glance is doing that)

3. This iso is transfered on one of the Virtual I/O Server (POST rest api)

4. By using Virtual Optic devices the iso is attached to the virtual machine

5. The machine is created

6. cloud-init or activation engine is mounting and reading the virtual cdrom and can use the information entered in the PowerVC GUI to setup the Virtual Machine

PowerVC server

User

VM creation

IP, hostname, activation input ….

- IP- hostname- ae- ….

VIOS1

VIOS2

Loadbalanced(to be verified)

Iso copy

Iso copy

VMLibary

vscsi

mount

Cloud-init/ae

1

2

34 5

6

© Copyright IBM Corporation 2015 15

GLA

NC

E

Created VM

Page 16: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Openstack data-source used by PowerVC 2/4

• For each Virtual Machines created PowerVC create a vscsi adapter used for the virtual optic data source

• This adapter can be removed by the user if he wants to. This one will not be utilized anymore after the VM creation

• Be careful with VIOS rootvg size (be sure to have enough space in /var to copy the iso, had issue when moving on P8 boot on san VIOS)

• What’s inside this iso ?

# lsrepName File Size Optical Accessvopt_754df319e3f14cbc979011e2360fc723 1 None ro# ls –l /var/vio/VMLibrary | grep vopt-r--r----- 1 root system 425984 Aug 3 12:36 vopt_754df319e3f14cbc979011e2360fc723# loopmount -i vopt_754df319e3f14cbc979011e2360fc723 -l loop0 -o "-V cdrfs -o ro" -m /mnt © Copyright IBM Corporation 2015 16

Page 17: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Openstack data-source used by PowerVC 3/4

• Network configuration (cloud-init format/debian style):# cat /mnt/openstack/content/0000# Injected by Nova on instance boot[..]auto loiface lo inet loopback

auto eth0iface eth0 inet static address 10.10.10.11 hwaddress ether fa:34:9b:4c:fb:20 netmask 255.255.255.0 broadcast 10.10.10.254 pre-up [ $(ifconfig eth0 | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}') = "fa:34:9b:4c:fb:20" ] gateway 10.10.10.254 dns-nameservers 8.8.8.8 8.8.8.9 dns-search lab.chmod666.org

Yes ! Cloud-init use the debian style interface format.

Don’t worry it’s his job to translate it for

AIX (mktcpip)

© Copyright IBM Corporation 2015 17

Page 18: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Openstack data-source used by PowerVC 4/4

• User data (user-data):

# cat /mnt/openstack/latest/user_dataruncmd: - root_home=$(grep ^root /etc/passwd | awk -F ':' '{print $6}') ; echo 'ssh-dss AAAABBBBCCCC[…]'>> ${root_home}/.ssh/authorized_keys - mount deckard:/export/nim/postinstall /mnt - /mnt/postintall.ksh - /mnt/install_was.ksh - /mnt/install_jre.ksh

Running post-install scripts

Adding an ssh key

This was a Websphere VM

© Copyright IBM Corporation 2015 18

Page 19: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

The PVID topic 1/4

• To capture the Virtual Machines PowerVC is using:– Snapshot (Shared Storage Pools)– Flashcopies (SVC, Storwise)

• To create a new Virtual Machines PowerVC is using these Snapshots/Flashcopies– After the deployments your VM will have the same PVIDs for both

rootvg and datavg– This is not an issue! But some customers may want to change this

(for instance storage billing can be based on PVID … or stuff like that)# for i in $(cat myvms) ; do echo "==$i==" ; ssh -q $i "lspv | grep -E \"rootvg|appsvg\"" ; done==rachael==hdisk0 00c30e8e42c879b9 rootvg activehdisk1 00c30e8e42d0aae5 appsvg active==tyrell==hdisk0 00c30e8e42c879b9 rootvg activehdisk1 00c30e8e42d0aae5 appsvg active

© Copyright IBM Corporation 2015 19

Page 20: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

The PVID topic 2/4

• Solution for the rootvg • Did you know changing the sys0 « ghostdev » to 2 and

rebooting the VM change rootvg PVID !

• After rebooting the lpar ghostdev attribute will automatically be reset to 0, and rootvg PVID will be changed

• Documentation about ghostdev : https://www-01.ibm.com/support/knowledgecenter/#!/ssw_aix_61/com.ibm.aix.osdevice/change_pvid_vgid.htm

• AIX 6.1/7.1/7.2

# lsattr -El sys0 -a ghostdevghostdev 0 Recreate ODM devices on system change / modify PVID True# chdev -l sys0 -a ghostdev=2sys0 changed# lsattr -El sys0 -a ghostdevghostdev 2 Recreate ODM devices on system change / modify PVID True

© Copyright IBM Corporation 2015 20

Page 21: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

The PVID topic 3/4

• Solution for other VG, use the « recreatevg » command to change all the PVIDs of a volume group

• cloud-init shell solution

© Copyright IBM Corporation 2015 21

Page 22: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

The PVID topic 4/4

• cloud-init cloud-config solution :

© Copyright IBM Corporation 2015 22

Page 23: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

cloud-init 1/5

• Cloud-init needs a lot of prerequistes. There is a script doing the job for you (https://github.com/transt/cloud-init-0.7.5/blob/master/cloudinst.sh)# cloudinst.sh

• You need an access to internet to run the script. If you don’t have any internet access, install it by hand. (rpm)# rpm -ivh --nodeps gettext-0.17-8.aix6.1.ppc.rpm[..]gettext ################################################### for rpm in bzip2-1.0.6-2.aix6.1.ppc.rpm db-4.8.24-4.aix6.1.ppc.rpm expat-2.1.0-1.aix6.1.ppc.rpm gmp-5.1.3-1.aix6.1.ppc.rpm libffi-3.0.11-1.aix6.1.ppc.rpm openssl-1.0.1g-1.aix6.1.ppc.rpm zlib-1.2.5-6.aix6.1.ppc.rpm gdbm-1.10-1.aix6.1.ppc.rpm libiconv-1.14-1.aix6.1.ppc.rpm bash-4.2-9.aix6.1.ppc.rpm info-5.0-2.aix6.1.ppc.rpm readline-6.2-3.aix6.1.ppc.rpm ncurses-5.9-3.aix6.1.ppc.rpm sqlite-3.7.15.2-2.aix6.1.ppc.rpm python-2.7.6-1.aix6.1.ppc.rpm python-2.7.6-1.aix6.1.ppc.rpm python-devel-2.7.6-1.aix6.1.ppc.rpm python-xml-0.8.4-1.aix6.1.ppc.rpm python-boto-2.34.0-1.aix6.1.noarch.rpm python-argparse-1.2.1-1.aix6.1.noarch.rpm python-cheetah-2.4.4-2.aix6.1.ppc.rpm python-configobj-5.0.5-1.aix6.1.noarch.rpm python-jsonpointer-1.0.c1ec3df-1.aix6.1.noarch.rpm python-jsonpatch-1.8-1.aix6.1.noarch.rpm python-oauth-1.0.1-1.aix6.1.noarch.rpm python-pyserial-2.7-1.aix6.1.ppc.rpm python-prettytable-0.7.2-1.aix6.1.noarch.rpm python-requests-2.4.3-1.aix6.1.noarch.rpm libyaml-0.1.4-1.aix6.1.ppc.rpm python-setuptools-0.9.8-2.aix6.1.noarch.rpm fdupes-1.51-1.aix5.1.ppc.rpm ; do rpm -ivh $rpm ;done[..]python-oauth ##################################################python-pyserial ##################################################python-prettytable ##################################################python-requests ##################################################libyaml ##################################################

© Copyright IBM Corporation 2015 23

Page 24: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

cloud-init 2/5

• The documentation is aweful (https://cloudinit.readthedocs.org/en/latest/)

• Configuration file is located in /opt/freeware/etc/cloud.cfg• For debugging purposes you may want to re-run cloudinit by

hand, delete somes files before re-running cloud-init:

# rm –rf /opt/freeware/var/lib/cloud/instances/*

• Default logs file located in: – /var/log/cloud-init-output.log– /var/log/cloud-init.log

• User data file located in /opt/freeware/var/lib/cloud/instances/<XXX>/user-data.txt#cloud-configruncmd: - mount nim:/export/nim/postinstall /mnt - /mnt/install_oracle.ksh - export LIBPATH=/usr/lib ; /mnt/script_install_was.ksh

© Copyright IBM Corporation 2015 24

Page 25: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Cloud-init 3/5

• Step 1 (local): cd is loaded, ip, hostname are set.

• Step 2 (init): All init modules are launched (rmc, bootlist, vg)

• Step 3 (config): All config modules are launched (mounts, chef run, runcmd)

• Step 4 (final) : All final modules are launched (phone_home, messages, shutdown)

Come join me in my session about Chef

on AIX (intermediate/expert)

© Copyright IBM Corporation 2015 25

Page 26: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

cloud-init 4/5

• Files:

write_files: - path: /tmp/cloud-init-started content: | cloud-init was started on this server permissions: '0755' - path: /var/log/cloud-init-sub.log content: | starting chef logging permissions: '0755'final_message: "The system is up, cloud-init is finished"

© Copyright IBM Corporation 2015 26

Page 27: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

cloud-init 5/5• Power_state:

power_state: delay: "+5" mode: poweroff message: cloud-init mandatory reboot for sddpcm timeout: 5• chef:

chef: force_install: false server_url: "https://cserver.chmod666.org/organizations/chmod666" validation_name: "chmod666-validator" validation_key: | -----BEGIN RSA PRIVATE KEY----- -----END RSA PRIVATE KEY----- run_list: - "role[aix7]" © Copyright IBM Corporation 2015 27

Page 28: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Ghostdev ? Clouddev ? 1/2

• The problem:– When a new machine is created, its rootvg was cloned. To remove the old

devices and wipe the ODM the activation is taking care of the ghostdev/clouddev attribute (it’s a sys0 attribute)

• If you are using the activation engine:– ghostdev is set to 1, after activation, ghostdev is automatically set back to 0– It works on all versions of AIX.

• If you are using cloud-init:– If you are using AIX >= 7100-03-05 or >=6100-09-05

• Clouddev is set to 1, after rebooting, clouddev is automatically set back to 0

– If you are using another version of AIX• Ghostdev is set to 1, but after rebooting, ghostdev is not automatically set back

to 0• Set manually (or using the activation input) ghostdev to 1• If you forget to do that any remote restart operation will result in an ODM wipe.• All AIX versions (6.1/7.1) will be supported in the next release of PowerVC

© Copyright IBM Corporation 2015 28

Page 29: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Ghostdev / Clouddev 2/2

• clouddev is using a flag is the nvram to check if the ODM needs to be wiped or not

# errpt | moreIDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTIONCB4A951F 0803171015 I S SRC SOFTWARE PROGRAM ERROR D872C399 0803171015 I O sys0 Partition ID changed and devices recreat• cloud-init on unsupported os:

# cloud-configruncmd: - chdev –l sys0 –a ghostdev=0

© Copyright IBM Corporation 2015 29

Page 30: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Using openstack command on PowerVC 1/2

• As PowerVC is based on Openstack, there is a way to use the openstack commands on the PowerVC host.

• Create a powervcrc file:# cat powervcrcexport OS_USERNAME=rootexport OS_PASSWORD=mypasswdexport OS_TENANT_NAME=ibm-defaultexport OS_AUTH_URL=https://powervc.lab.chmod666.org:5000/v3/export OS_IDENTITY_API_VERSION=3export OS_CACERT=/etc/pki/tls/certs/powervc.crtexport OS_REGION_NAME=RegionOneexport OS_USER_DOMAIN_NAME=Default export OS_PROJECT_DOMAIN_NAME=Default

powervc root password

powervc url

• Source it:# source powervcrc

• You can now use openstack commands:# nova list+--------------------------------------+-----------------------+--------+------------+-------------+------------------------+| ID | Name | Status | Task State | Power State | Networks |+--------------------------------------+-----------------------+--------+------------+-------------+------------------------+| 394ab4d4-729e-44c7-a4d0-57bf2c121902 | deckard | ACTIVE | - | Running | vlan666=10.16.66.60 || cd53fb69-0530-451b-88de-557e86a2e238 | priss | ACTIVE | - | Running | vlan666=10.16.66.61 || 64a3b1f8-8120-4388-9d64-6243d237aa44 | rachael | ACTIVE | - | Running | || 2679e3bd-a2fb-4a43-b817-b56ead26852d | batty | ACTIVE | - | Running | || 5fdfff7c-fea0-431a-b99b-fe20c49e6cfd | tyrel | ACTIVE | - | Running | |+--------------------------------------+-----------------------+--------+------------+-------------+------------------------+

© Copyright IBM Corporation 2015 30

Page 31: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Using openstack commands on PowerVC

• Reboot a machine:# nova reboot deckard

• Migrate a machine (lpm):# nova live-migration deckard 828641A_YYYYYY

• List hosts:# nova hypervisor list+----+---------------------+-------+---------+| ID | Hypervisor hostname | State | Status |+----+---------------------+-------+---------+| 21 | 828641A_XXXXXXX | up | enabled || 23 | 828641A_YYYYYYY | up | enabled |+----+---------------------+-------+---------+

• Maintenance mode:# nova maintenance-enable --migrate active-only --target-host 828641A_XXXXXX 828641A_YYYYYYY

• Create a machine:# nova create --image 7100-03-04-cic2-chef --flavor powervm.tiny --nic net-id=5fae84a7-b463-4a1a-b4dd-9ab24cdb66b5,v4-fixed-ip=10.16.66.51 novacreated

• WARNING : No PowerVC extension with openstack cmd !

© Copyright IBM Corporation 2015 31

Page 32: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

PowerVC Lun order

• To remove a rootvg lun from a Virtual Machine this one needs to be in « position » 2. (not 1)

• For instance when I’m moving a new machine to PowerVC (from an old host) I’m creating a VM with PowerVC (for profiles,zoning, …). Then I’m mapping « old » luns to this machine

© Copyright IBM Corporation 2015 32

Page 33: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Rest API 1/2

• The best way to scripts things on PowerVC is to use the rest API• I warn you: for a system administrator using a rest API can be (at the

begining) difficult• My advise is to use Python (json trees can be treated as dict)• I’ve created a few example of how to use the PowerVC rest API :

– pvcgrowlun: growing a existing lun (https://github.com/chmod666org/pvcmd/blob/master/pvcgrowlun)

– pvcmkvm: creating a Virtual machine with the rest API (https://github.com/chmod666org/pvcmd/blob/master/pvcmkvm)

# pvcgrowlun -v multi-vol-bf697dfa-0000003a-828641A_21AFF8V-data-1 -s 84 -p powervc.lab.chmod666.org -u root -P mypassword# cat myvmname:myvmip_address:10.10.10.2vlan:vlan666image:aix7100-03-05-chefstorage_connectivity_group:npiv_4portsvirtual_processor:1entitled_capacity:0.1memory:8192 storage_template:svc# pvcmkvm -f myvm -p powervc.lab.chmod666.org -u root -P mypassword

© Copyright IBM Corporation 2015 33

Page 34: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Rest API 2/2• The most difficult part with rest API is to find the json body you need to create• My advice is to test every calls to the API with RESTClient (firefox plugin)

• Check json body with firefox firebug (example with network creation):

© Copyright IBM Corporation 2015 34

Page 35: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Copying images between storage providers• When you have a lot of different storage providers (in my case I have three different

SVC) you have to synchronize your images between each one.• Step 1: Zone all your storage provider on a host• Step 2: Map your OS Image lun on this host (from the source Storage Provider)

• Step 3: Map a new lun (from the desitnation Storage Provider)• Step 4: Use dd to copy the data from on lun to one another. (note the block size)

– /dev/mapper/mpathe -> lun of the OS Image (source Storage Provider)– /dev/mapper/mpathc -> new lun (destination Storage Provider)

# dd if=/dev/mapper/mpathe of=/data/download/aix7100-03-04-cloudinit-chef bs=4M# dd if=/data/download/aix7100-03-04-cloudinit-chef of=/dev/mapper/mpathc bs=4M• Step 5: Import the new image on PowerVC

– Manage the new lun with « Manage Existing »

# powervc-volume-image-import --name rhel64 --os rhel --volume volume_capture2 --activation-type ae Password: Image creation complete for image id: e3a4ece1-c0cd-4d44-b197-4bbbc2984a34

© Copyright IBM Corporation 2015 35

Page 36: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Linked clones 1/2

• PowerVC is the only way to created « linked clone » VM• You need to have a Shared Storage Pool as a storage

provider. How does it work:– Step 1: The captured rootvg underlying disk is a Shared Storage Pool Logical Unit.– Step 2: When the image is captured the rootvg Logical Unit is copied and is known as a “Logical

(Client Image) Unit”.– Step 3: When deploying a new machine a snapshot is created from the Logical (Client Image) Unit.– Step 4: A “special Logical Unit” is created from the snapshot. This Logical Unit seems to be a

pointer to the snapshot. We call it a clone.– Step 5:The machine is booted and the activation engine is running and reconfiguring the network– Step 6:When a block is modified on the new machine this one is duplicated and modified on one

new block on the Shared Storage Pool.– Step7: This said if no blocks are modified all the machines created from this capture are sharing

the same blocks on the Shared Storage Pool.– Step 8: Only modified blocks are not shared between Linked Clones. The more things you will

change on your rootvg the more space you will use on the Shared Storage Pool.– Step 9:That’s why these machines are called Linked Clones : they all are connected by the same

source Logical Unit.– Step 10:You will save TIME (just a snapshot creation for the storage side)

and SPACE (all rootvg will be shared by all the deployed machines) by using Linked Clones.

© Copyright IBM Corporation 2015 36

Page 37: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Linked clones 2/2

© Copyright IBM Corporation 2015 37

Page 38: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Debugging PowerVC

• Best things to do is to check logs:– All openstack logs are located in /var/log

• Nova• Cinder• …

• Things I’ve learned by experience:– Check VIOS rootvg /var space (PowerVC need space in /var)– PowerVC needs responsive VIOS. By this I mean if some commands

are too long to answer, PowerVC will consider this VIOS as busy and will try a new one. (And indicate it may be an RMC problem in the log)

– In the Virtual I/O Server check the logs located in /home/ios/logs• viosvc.log• viod_CM.log (if you have .FATAL files you can check them to find the

issue)

© Copyright IBM Corporation 2015 38

Page 39: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Backup/Restore strategy

• My advice:– Create two Linux hosts (on different sites)– Install PowerVC at the same level on each hosts– If you update one PowerVC update the backup too– On the production one run weekly backups:

– Copy this backup on a third host or backup it with TSM– If you have an outtage of the first PowerVC restore the configuration

on the second one

# crontab –l0 0 * * 6 rm -rf /var/opt/ibm/powervc/backups/* ; /opt/ibm/powervc/bin/powervc-backup --targetDir /var/opt/ibm/powervc/backups --noPrompt

# powervc-restore --targetDir /var/opt/ibm/powervc/backups/Continuing with this operation will stop all PowerVC services and overwrite critical PowerVC data in both the database and the file system. Do you want to continue? (y/N):yPowerVC services stopped.[..]

© Copyright IBM Corporation 2015 39

Page 40: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Dynamic Resource Optimizer 1/2

• Had the chance to participate in the beta test program• DRO is checking host utilization and optimize (rebalance) the

placement of Virtual Machine, or of CPU:– By using Live Partition Mobility– By using Power Entreprise Pool (mobiles cores)

• DRO can by run in:– Active mode: VM will be moved, cores will be moved.– Advise mode: DRO will advise you in the logs

• Specific machines, host, or host group can be excluded from DRO.

• For the moment DRO is the checking the CPU usage:– Criteria : Threshold, stabilization, run interval

© Copyright IBM Corporation 2015 40

Page 41: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Dynamic resource Optimizer 2/2

© Copyright IBM Corporation 2015 41

Page 42: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Redbook & resources

• My blog posts about PowerVC : http://chmod666.org/index.php/tag/powervc/

• Redbook: http://www.redbooks.ibm.com/redpieces/pdfs/sg248199.pdf

• Linked-in PowerVC group: https://www.linkedin.com/grp/home?gid=6610502

• PowerVC DW Wiki (best place for documentation): https://www.ibm.com/developerworks/community/wikis/home?lang=en_us

• Twitter:– @IBMPowerVC

© Copyright IBM Corporation 2015 42

Page 43: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

Thank you !

Special thanks to (for their support or help)- All the PowerVC team in Austin (Joe, Christine, Sam, Gerald)- Jay Kruemcke @chromeaix @cloudrancher- Philippe Hermes @phhermes- Nigel Griffiths @mr_nmon / Gareth Coates @power_gaz- Chris Gibson @cgibbo- Rosa Davidson @radavids1- Eddie Shvartsman

© Copyright IBM Corporation 2015 43

Page 44: Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM. 9.0 PowerVC for PowerVM

© Copyright IBM Corporation 201544

© Copyright IBM Corporation 2015

Continue growing your IBM skills

ibm.com/training provides acomprehensive portfolio of skills and careeraccelerators that are designed to meet all your training needs.

• Training in cities local to you - where and when you need it, and in the format you want– Use IBM Training Search to locate public training classes

near to you with our five Global Training Providers

– Private training is also available with our Global Training Providers

• Demanding a high standard of quality – view the paths to success – Browse Training Paths and Certifications to find the

course that is right for you

• If you can’t find the training that is right for you with our Global Training Providers, we can help.– Contact IBM Training at [email protected] Global Skills Initiative