opalisrobot™ demonstration . actual run book procedure actual data center run book procedure...
TRANSCRIPT
OpalisRobot™ Demonstration
www.opalis.com
Actual Run Book Procedure
Actual Data center Run Book Procedure documenting for Level 1 staff how to both VERIFY and then RESOLVE a SQL service failure
Demo: Resolve SQL Failure Alert
• Acknowledge the alert• Assign the alert to a level 1 technician• Open a new trouble ticket• Notify users: “System may be down”• Place troubled device “off-line”
Execute the repetitive tasks associated with performing all maintenance procedures
SetupSetup
Running under VMware, the NT service for
Microsoft SQL server fails. This SQL service is the
backend for Cognos
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
SetupSetup TestTest ResolveResolve CloseClose
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
The network management event monitor (in this case NetIQ AM) sees the SQL service is down and a new AM Alert is generated
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
SetupSetup TestTest ResolveResolve CloseClose
OpalisROBOT sees the new alert and assigns itself responsibility for the alert by changing Status from “Open” to “Acknowledged”
OpalisROBOT opens a new Remedy trouble ticket and updates the Remedy work log as each step of the procedure is performed
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
OpalisROBOT takes the troubled machine offline by setting Maintenance Mode for the device to ON
OpalisROBOT updates the Remedy case log for the ticket after every step of the procedure, documenting the date, time and results
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
• Acknowledge the alert• Assign the alert to a level 1 technician• Open a new trouble ticket• Notify users: “System may be down”• Place troubled device “off-line”
Execute the repetitive tasks associated with performing all maintenance procedures
SetupSetup
• Is it really down?• PING the server IP address• Check the VM is not frozen• Run a test query• Test the SQL service is not frozen
95% of the time it’s a known issues. Test for them. Else it’s an EXCEPTION - call expert!
TestTest
Demo: Resolve SQL Failure Alert
PING completes, and OpalisROBOT continues by performing a status check on the virtual machine in VMWARE
The status check of the NT service for MS SQL server shows the problem is that SQL service is down.
OpalisROBOT follows the standard Level 1 procedure to diagnose the problem by first performing a PING to see if IP address is hung
The status check on the virtual machine in VMWARE shows no problems, and OpalisROBOT now checks the NT service for SQL server
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
For every step, OpalisROBOT has updated the Remedy case log history ensuring compliance, best practices, and audit requirements
• Acknowledge the alert• Assign the alert to a level 1 technician• Open a new trouble ticket• Notify users: “System may be down”• Place troubled device “off-line”
Execute the repetitive tasks associated with performing all maintenance procedures
SetupSetup
• Is it really down?• PING the server IP address• Check the VM is not frozen• Run a test query• Test the SQL service is not frozen
95% of the time it’s a known issues. Test for them. Else it’s an EXCEPTION - call expert!
TestTest
• Restart the VM• Restart the SQL service• Run a test SQL Query• Ensure the service is back up• Notify Expert if it’s an exception
Knows issue, then known fix. Did it resolve? No? Then it’s an EXCEPTION - call expert
ResolveResolve
Demo: Resolve SQL Failure Alert
OpalisROBOT follows the documented Level 1 resolution procedure restarting both Windows and the VM
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
Under VMware, the instance of Windows 2003 that was running the troubled SQL service is completely shut down
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
Following the Level 1 restoration procedure OpalisROBOT backs up the physical machine, initiating Veritas Backup
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
Following the Level 1 restoration procedure OpalisROBOT backs up the physical machine, initiating Veritas Backup
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
As part of the procedure OpalisROBOT waits until the Veritas Backup completes and tests the backup set for validity
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
Continuing the procedure OpalisROBOT invokes SMS to deploy the standard configuration files before restarting SQL
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
The restart phase of the procedure begins by Opalis controlling VMware to start a new virtual machine
Windows 2003 server completes it’s normal boot process and is started inside the new VMware virtual machine
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppMangerNetIQ AppManger Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
The NT service for Microsoft SQL server successfully re-starts
automatically as part of the Windows boot process
The service is now restored
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
• Acknowledge the alert• Assign the alert to a level 1 technician• Open a new trouble ticket• Notify users: “System may be down”• Place troubled device “off-line”
Execute the repetitive tasks associated with performing all maintenance procedures
SetupSetup
• Is it really down?• PING the server IP address• Check the VM is not frozen• Run a test query• Test the SQL service is not frozen
95% of the time it’s a known issue. Test for these. Else it’s an EXCEPTION - call an expert!
TestTest
• Restart the VM• Restart the SQL service• Run a test SQL Query• Ensure the service is back up• Notify Expert if it’s an exception
Known issue, then known fix. Did this resolve it? No? Then it’s an EXCEPTION - call expert
ResolveResolve
• Close the alert• Close the trouble ticket• Place machine back “on-line”• Notify users: “System is back up”• Notify Level 2 Expert
Perform the repetitive tasks associated with closing all maintenance procedures
CloseClose
Demo: Resolve SQL Failure Alert
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
Completing the procedure OpalisROBOT puts the machine back into production by removing the AM Maintenance Mode flag
SetupSetup TestTest ResolveResolve CloseClose
VMwareVMware NetIQ AppManagerNetIQ AppManager Veritas NetBackupVeritas NetBackup BMC RemedyBMC Remedy Microsoft SMSMicrosoft SMS
Finally, OpalisROBOT closes the Alert and updates the Remedy case log once more before sending an email with the results to the admin.
• Acknowledge the alert• Assign the alert to a level 1 technician• Open a new trouble ticket• Notify users: “System may be down”• Place troubled device “off-line”
Execute the repetitive tasks associated with performing all maintenance procedures
SetupSetup
• Is it really down?• PING the server IP address• Check the VM is not frozen• Run a test query• Test the SQL service is not frozen
95% of the time it’s a known issues. Test for them. Else it’s an EXCEPTION - call expert!
TestTest
• Close the alert• Close the trouble ticket• Place machine back “on-line”• Notify users: “System is back up”• Notify Level 2 Expert
Perform the repetitive tasks associated with closing all maintenance procedures
CloseClose
• Restart the VM• Restart the SQL service• Run a test SQL Query• Ensure the service is back up• Notify Expert if it’s an exception
Knows issue, then known fix. Did it resolve? No? Then it’s an EXCEPTION - call expert
ResolveResolve
Demo: Resolve SQL Failure Alert
Actual Run Book Procedure
Actual Data center Run Book Procedure documenting for Level 1 staff how to both VERIFY and then RESOLVE a SQL service failure
The resulting OpalisROBOT documentation produced as part of the execution of the above run book procedure
For more information visit www.opalis.com
www.opalis.com