Doc : Dom0 Upgrade Execution Process

Version : 1.4

--------------------------------------------------------------------------------------------------------------------------

Objective :-

This Document describes the steps to be followed to perform the OVMS upgrade or Dom0 upgrade.

Pre check to be performed 1 hr before the schedule of Execution

1.Ensure : Shift Lead takes Pre HC for the Dom0-VM using Url

2.Ensure : /mig-os is mounted (script will auto mount during execution phase)

3.Mig-os mount source details :-

mkdir /mig-os ; mount -o noacl rm02stor29-nas:/export/roh_29a_hwmig/rmdc_dom0 /mig-osmkdir /mig-os ; mount -o noacl adc08ntap17-bkp:/vol/aoh_17a_hwmig/adc_dom0 /mig-os mkdir /mig-os ; mount -o noacl sl05stor02-nas:/export/sloh_02a_hwmig/sldc_dom0 /mig-osmkdir /mig-os ; mount -o noacl tvp01stor05-nas:/export/tvpoh_05a_hwmig/tvpdc_dom0 /mig-osmkdir /mig-os ; mount -o noacl epc002oodstor02-nas:/export/epcoh_02a_hwmig/epdc_dom0 /mig-osmkdir /mig-os ; mount -o noacl syc002oodstor01-nas:/export/sycoh_01a_hwmig/sydc_dom0 /mig-os

4.Validate Prep Data and Current Environment

============== TEMPLATE ==============

SR# :

Source Dom0 :

Target Dom0 :

Source Dom0 VM list :

Nos of VMs from PREP :

Nos of VMs from Portal :

Are the VM names Listed in Prep the same as current VM List [Y/N] :

============== TEMPLATE ==============

Use This as reference through out activity

5.To Take: Source Dom0 – Pre Health checks and paste the output in the SR

Run:

a) /ptsadmin/ateam/bin/ovms-go_live_check.pl

b) /ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o pre -rfc 3-TESTII

Above script covers:

HW_model of the Dom0BIOS version ,ILOM versionunamecat /etc/enterprise-releasemig-os is mountedreset ilom passwordShows Total vcpus of all running VMs ,Shows Total Memory of all running VMsChecking the vm.cfg for vcpu_max parameterCheck if vms are on localdiskIdentify the container to snapEM agent status ,Ovs-agent statusLdap statusIfconfig,bond,MTU,route details

c). Ensure you are able to access the ilom console of the Dom0

d). Ensure that /etc/ovms-provision.conf exists. If it does not, perform the following step:

cat > /etc/ovms-provision.conf << EOF

DATACENTER=<Valid values found in Appendix 7.1 of OMCS_Hypervisor_Provisioning.pdf>

ZONE=<Dom0s Network ZONE>

PROVISIONING_TICKET=<SR/RFC Number>

EOF

vi /etc/ovms-provision.conf

e) Replace the values assigned to DATACENTER, ZONE, and PROVISIONING_TICKET with the relevant information for the patching ticket at hand.

6. Set BO and Disable Pinger

EM : https://odem3.oracle.com/

Pinger :https://globaldc.oracle.com/host/detail/vmfsnchpg029

On Temp/Stage Dom0

7. To Take: Target Dom0 – Validations

Run:

/ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o target -o validate -rfc 3-TESTII

- This will check same Bridges from Source are available on the temp Dom0- Ensure the same container is available.

SA Has to :-

- Ensure VLANs tagged with the same VLANS as Source Dom0 –check the related SR to confirm the VLAN is already tagged.

Search for SR based on <sub> check the SR is closed and included the VLANs in use on the current Dom0 ( based on output captured in the above prep script.

***Proceed to live migrate ,if your answer is yes for the below conditions ****

- the same bridges are on Temp domo - VLAN are tagged - Source container is mounted on temp Dom0 - Source Swap container is mounted on the temp Dom0

Execution Phase :-

1. Perform Live migration using the script

On Source Dom0

Run:

/ptsadmin/os_migration/bin/migVMs.sh

Sample output--

[root@rmc002oodhost201 pchandru]# /ptsadmin/os_migration/bin/migVMs.sh

--------------------------------------------a3e9b7c74290475e8bf70a00fa14e6f0 64bit VM--------------------------------------------

/tmp/migVMs.lst was generated <<<<<<<<Please review it, and modify if necessary

To start migration, run

Take confirmation form SL on HC by this time.

Initiate a ping to the VMs and take Uptime

# /ptsadmin/os_migration/bin/migVMs.sh /tmp/migVMs.lst [target Dom0]

Monitor Logs on source and target : tail -f /var/log/xen/xend.log

Validate by running xm list so that no vms are there on the source dom0 now.

Access/Verify each VM after migrating to Temp dom0.

2.Ugrade : (this steps performs Disable uptrack and upgrade )

Run:

Time /ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o upgrade -rfc 3-TESTII

Log location for the upgrade: Ex /var/log/od-provision/2015-08-08-20\:40\:13.log

3. Bounce the Dom0

Check grub.conf – we should see the new kernel

Power off the Dom0

Use: /ptsadmin/os_migration/bin/system_restart.pl -s --From consoleTake current power status of host : show /SYS … if should of off Start Dom0 from Console : start /SYS--

4.Once the Dom0 is up run the below

/ptsadmin/ateam/bin/fix_firmware --- (go with defaults )

if it suggests : Firmware upgrade needed ..allow it proceed and perform below steps

b) INFORM A TEAM LEAD (this item is being tracked)

c) Shutdown EM Agent

d) Shutdown Dom0

e) Power off Dom0 host

f) Reboot ILOM by executing reset /SP in an ILOM ssh session

g) Power on Dom0 host and await boot-up

h) rerun : /ptsadmin/ateam/bin/fix_firmware ... it should not show firmware upgrade needed

Post checks on Source Dom0

To Take: Source Dom0 – Post checks

Ensure to Mount : /mig-os

1.

Run:

/ptsadmin/ateam/bin/ovms-go_live_check.pl

/ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o post -rfc 3-TESTII

2.Ensure that OVM discovery completes successfully through OVM UI

- You can identify the OVM manager for a Dom0 by running the following as root on the Dom0 (no value means that the Dom0 is not currently managed by OVMM):

ovs-agent-db read_item server manager_core_api_url | \ cut -f2 -d@ | cut -f1 -d:

Ex:

[root@auc026oodhost147 pchandru]# ovs-agent-db read_item server manager_core_api_url | cut -f2 -d@ | cut -f1 -d:

10.224.94.217

Connect to the OVMM server and follow steps given in the below doc

3.If the upgrade has gone correctly and all verifications passed, then remove uptrack RPMs

yum erase -y uptrack-libyaml uptrack-python-pycurl uptrack-PyYAML uptrack

4.Bringback the VMs to original Dom0/Flip them back

Note:

On the Source– MOTD if we find a note as below do not move back the VMs to SRC

----------------------------------------------------------------------------------------------------------

===================================IMPORTANT NOTE===================================

----------------------------------------------------------------------------------------------------------

THIS SERVER WILL BE USED EXCLUSIVELY FOR OVS UPGRADES AND LIVE MIGRATIONS.

PLEASE DO NOT USE IT FOR ANY CUSTOMER PROVISIONING OR STANDARD MAINTENANCE WORK ORDERS.

If you have any queries/concerns over this dom0s usage please contact Girish Ahuja / Vaibhav Chhabra

----------------------------------------------------------------------------------------------------------

===================================IMPORTANT NOTE===================================

----------------------------------------------------------------------------------------------------------

Login to Source Dom0 : take xm list

Login to Stage/Temp Dom0 : take xm list

Move the VMs back From Temp/Target Dom0 to Source

-Login to Temp /Target dom0

Run:

xm migrate –l <your vm> <source-Dom0>

Repeat the same for all the VMs which you had moved before.

Take xm list on the temp/stage dom0 : No VMs should be running now

Take xm list on the Source dom0 : VMs should be running now

Login to each VM and take uptime

On Source Dom0:

Once All VMs are up and moved back – restore the : /etc/xen/auto_OVMSUPGBKP.<ddMMYY> backup folder

As /etc/xen/auto

Inform Shift Lead to take Post Health checks

…cont

SR final Update Template/Release note

The scheduled maintenance activity has been completed successfully.should you encounter any issue, Please contact:

SA1 Contact:------------Name : Devesh KumarMobile : +91 9611523300Email ID : devesh.d.kumar@oracle.com

Management Escalation :---------------------Name : karuppiah rama Email ID : karuppiah.rama@oracle.comMobile : +91 98456 33725

Name : Hari Yalavarthy Email ID : hari.yalavarthy@oracle.comMobile : +91 9845902356

Regards,

******END******

Doc : Dom0 Upgrade PREP Process

Version : 1.4

--------------------------------------------------------------------------------------------------------------------------

OVMS Dom0 Upgrade Prep Procedure

1.Capture the Details in to the Template

============== TEMPLATE ==============

Source Dom0 :

Target Dom0 :

Source Dom0 VM list :

Nos of VMs from Portal :

Nos of VMs currently running :

Are the VM names Listed in SR the same as current VM List [Y/N] :

NETWORK SR# :

============== TEMPLATE ==============

Mount the mig-os on the Source and Target Dom0s

RMDC :mkdir /mig-os ; mount -o noacl rm02stor29-nas:/export/roh_29a_hwmig/rmdc_dom0 /mig-osADC :mkdir /mig-os ; mount -o noacl adc08ntap17-bkp:/vol/aoh_17a_hwmig/adc_dom0 /mig-os SLDC :mkdir /mig-os ; mount -o noacl sl05stor02-nas:/export/sloh_02a_hwmig/sldc_dom0 /mig-osTVP :mkdir /mig-os ; mount -o noacl tvp01stor05-nas:/export/tvpoh_05a_hwmig/tvpdc_dom0 /mig-osEPDC :mkdir /mig-os ; mount -o noacl epc002oodstor02-nas:/export/epcoh_02a_hwmig/epdc_dom0 /mig-osSYDC :mkdir /mig-os ; mount -o noacl syc002oodstor01-nas:/export/sycoh_01a_hwmig/sydc_dom0 /mig-os

On Source Dom0 :-

Mount : /mig-os

RUN:

/ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o prepwork -rfc 3-TESTII

This script will cover below steps

-Takes a List of the VMs running on the Source Dom0

-Takes the Total Memory and VCPU used/assigned to the VMS on Source Dom0 -Identify the bridges of the VMs running on the Source Dom0 - Identify the Storage Container and swap container on the Source dom0 - Reset ILOM Password on Source Dom0- check if VM is on local disk

On Target Dom0 :- Mount : /mig-os

/ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o target -o prepvalidate -rfc 3-TESTII

This will compare:-

Same Storage and swap container is Available on Target dom0

Same bridges of source Dom0 Vms are available on the Target Dom0

Show if enough free memory is available for VM migration

VLAN Tagging:-

1.Check in portal for vlan tagging SR and confirm the required vlan

Confirm it is completed.

update the vlan tagging page

2.if vlan tagging is not there, then identify all the VM's in a given server pool and get the vlan details of each VM and raise network vlan tagging SR

stepsssh -l admin localhost -p 10000OVM> show serverPool name=rmc002oodpool001OVM> show serverPool name=rmc002oodpool001

update the vlan tagging page

3.IF we find more than Two Bridges for a VM – check it has –rac interface

In that case we need to take ./getnet for –rac also and get it tagged

Run for :

-rac

-vip

-Frontend

Login : hqsun1 - take the vlan id details of the VMs

EX:

bash-2.03$ /home/vvarughe/getnet vmohsadvg501

Request: vmohsadvg501

FQDN: vmohsadvg501.oracleoutsourcing.comIP: 148.87.205.163Netmask: 255.255.255.240

vlan id: rmdc-z2-advg-v160vlan note: NAIR-DB-Advantage-Sales-and-Marketing-(sr3-5756856194)

Network: 148.87.205.160/28First IP: 148.87.205.161Last IP: 148.87.205.174Broadcast IP: 148.87.205.175

To Get the Switch port details and : PFE and SFE

Use portal : http://pnvcapp01.oracle.com/cgi-bin/switches.cgi

Rasie SR to : Network Team

Sample SR#3-11199598771

Follow up till completion of the VLAN tagging SR

Validate all VLANS from the list given are tagged.