em12c high availability without slb and rac

High Availability Cloud Control Recipes - EM12C

Session ID# 941

Mike Gangler

Ithaka - www.ithaka.org

http://www.ithaka.org/

Mike Gangler• Ann Arbor, Michigan• Database Specialist at Ithaka.org – aka “JSTOR”• Multi-platform and Database DBA• Specializing in Performance, Tuning and management of

databases and environments• Working With Oracle since Versus 4 ….• Board of Directors for SEMOP (Southeast Michigan Oracle

Professionals), MOUS (Michigan Oracle User Summit, and charter member (1993) of the Board of Directors of the International Oracle Users Group(1OUG).

What this presentation will cover• EM12 Architecture High Availability

options

• Setting up an Level 3 MAA Environment

• Installation of a Standby OMS – Silent Mode

• If Time---Notifications Best Practices

Audience Experience• Grid Control Experience

• Used the old 9i / Workgroup version ?

• 10G Grid Control

• 11G Grid Control

• 12C Cloud Control

Other Presentations• Enterprise Manager SIG – 9:30 – 10:30am

• Wednesday – MileHigh 4B ?

Presentation Goals • Goals

• Learn the Different MAA architectures

• How to run a installation of a Standby OMS in silent Mode

• Notifications Tips / Best Practices

Ithaka’s Missionaka JSTOR

ITHAKA is a not-for-profit organization dedicated to helping the academic community take full advantage of rapidly advancing information and networking technologies. We serve scholars, researchers, and students by providing the content, tools, and services needed to preserve the scholarly record and to advance research and teaching in sustainable ways. We are committed to working in collaboration with other organizations to maximize benefits to our stakeholders.

www.ithaka.org - www.jstor.org

http://www.ithaka.org/

http://www.jstor.org/

EM12C ArchitectureOptions

• Level 1 – Single Site – No Failover

• Level 2 – Single Site • 2 OMS – Active / Passive Mode• VIP Failover


• Level 3 • Single Site• Multiple OMS (Active/Active)• Load Balancer• RAC Data Guard Primary / Standby DB


• Level 4 • Multiple Sites• Primary OMS (Active/Active) w/ Load Balancer• Load Balancer• RAC Data Guard Primary / Standby DB• Multiple Standby OMS w/ Load Balancer

• .

EM12C ArchitectureSuggestions

• Setup an additional / separate OMS for Dev/QA targets

• Setup Level 3.5 Maximum Availability Architecture (MAA)• Multiple Sites• Multiple OMS (Primary / Standby)• Oracle Data Guard with Fast Start Failover setup.• Using VIP’s / DNS alias help with data Guard rollover.• Use Load Balancer (optional) for OMS.

EM12C Standby OMS Architecture (level 3 MAA)

Monitoring at JSTOR• Production Oracle Databases – 25 –

24x7• Development Oracle Databases – 27 –

14x5• Microsoft SqlServer Databases - 21 –

14x5• MySql Databases – 9 – 14x5

• Total 240 Production Targets (Agents, hosts, db, etc.)

• Total 276 Dev/QA Targets (Agents, hosts, db, etc.)

Ithaka OMS LayoutDATACENTER

1DATA

CENTER 2DATACENTER

3

DataCenter 1

Secondary OMS

DataCenter 2

Primary OMS

Datacenter1

StandbyDataGuard

DB

Primary DB

DNS

DNS

Pre-requisites for High Availability•Setup VIP or DNS Alias for multiple

sites.

•Create Data Guard database on other site.

•NFS Shared Library

•Primary OMS Setup

•Plugins Installed on Primary Site

Pre-requisites for High Availability

•Optional setups:

•Fast Start Failover Data guard Setup(Requires additional hardware)

•Load Balancer NOT REQUIRED – Although the documentation disagrees

•RAC DB’s– Great if you can afford it.

Requirements Standby OMS

•Machine Required for Standby OMS/DB (Linux, Sparc, HPUX, Microsoft)

•Plugins need to be the same version

•Exact same directory structure

•Load Balancer (optional)

•NFS Drive ….

Configuration Steps:1.Install Primary OMS / DB

2.Add targets to Primary

3.Setup NFS Drive – Software Library

4.Add All Plugins required to Primary OMS

5.Add Dataguard - Standby Database

6.Create Standby OMS

7.Convert Standby Database to RAC (IF Available)

Installation Steps: (Silent)

Verify NFS Software Library on Standby Machine

Install a “Software-Only” Installation on Standby OMS Machine

(Do NOT run the ConfigureGC.sh or ConfigureGC.bat)

Installation Steps: (Silent)• Ensure same Plugins as the source OMS

SOURCE_OMS_pgrid01>l 1 SELECT epv.plugin_id, epv.version, epv.rev_version 2 FROM em_plugin_version epv, em_current_deployed_plugin ecp 3 WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE','INSTALL_HOME') 4 AND ecp.dest_type='2' 5* AND epv.plugin_version_id = ecp.plugin_version_idSYSMAN_pgrid01>/

PLUGIN_ID VERSION REV_VERSION-------------------- ------------ -----------oracle.sysman.mos 12.1.0.2.0 0oracle.em.satc 12.1.0.1.0 0oracle.em.smss 12.1.0.2.0 0oracle.sysman.emct 12.1.0.3.0 0oracle.sysman.ssa 12.1.0.4.0 0oracle.sysman.db 12.1.0.2.0 20120804oracle.sysman.xa 12.1.0.3.0 0oracle.sysman.emas 12.1.0.3.0 0oracle.sysman.vt 12.1.0.3.0 0pythian.mysql.prod 12.1.0.1.2 0

Installation Steps: (Silent)Plug-in Definitions:

PLUGIN_ID DEFINITION-------------------- -----------------------oracle.sysman.mos My Oracle Supportoracle.em.satc Apache Tomcatoracle.em.smss Microsoft SQL Serveroracle.sysman.emct Charge Back oracle.sysman.ssa Cloud Application oracle.sysman.db Oracle Databaseoracle.sysman.xa Exadataoracle.sysman.emas Fusion Middlewareoracle.sysman.vt Virtualisationpythian.mysql.prod MySQL

Installation Steps: (Silent)Plug-ins (Continued)

1. To Install Additional Plug-ins – Manually download the plug-ins from OTN to Accessible location (i.e., /tmp)

a.Verify that they are the same version as primary OMS

2. http://www.oracle.com/technetwork/oem/grid-control/downloads/index.html

3.Run $OMS_HOME/sysman/install/PluginInstall.sh –pluginLocation /tmp - Choose the plugins you want to install (screen print – next page)

PluginInstall.sh

Installation Steps: (Silent)1.Apply all patches that were applied to the first OMS

a. To Check - $OMS_HOME/OPatch/opatch lsinventory

2.Export the Configuration Details from the First OMS

a. $OMS_HOME/bin/emctl exportconfig oms –dir <Path Name>

b.Creates bka file – i.e. opf_ADMIN_<DATE>_<TIME>.bka

c. Copy bka file to standby OMS Machine

Installation Steps: (Silent)oracle@pr2prodora01 ~]$ cd /u01/app/oracle/OracleHomes2/oms/bin/[oracle@pr2prodora01 bin]$ ./emctl exportconfig oms -dir $HOMEOracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password : ExportConfig started...Machine is Admin Server host. Performing Admin Server backup...Exporting emoms properties...Exporting secure properties...

Export has determined that the OMS is not fronted by an SLB. The local hostname was NOT exported. The exported data can be imported on any host but resecure of all agents will be required. Please see the EM Advanced Configuration Guide for more details.

Exporting configuration for pluggable modules...Preparing archive file...Backup has been written to file: /home/oracle/opf_ADMIN_20130125_101524.bka

The export file contains sensitive data. Please ensure that it is kept secure.

ExportConfig completed successfully![oracle@pr2prodora01 bin]$ cd $HOME[oracle@pr2prodora01 ~]$ scp opf_ADMIN_20130125_101524.bka [email protected]:/home/[email protected]'s password: opf_ADMIN_20130125_101524.bka 100% 45MB 5.0MB/s 00:09 [oracle@pr2prodora01 ~]$


OMSCA Breakdown on Standby database using the backup file created

• Uses the backup file just generated = bka file• Admin Server HTTPS Port = 7101• Managed Server Port=7202• Managed Server HTTPS Port=7301• EM Note Manager Port = 7403• EM Upload Port = 4889• EM Upload HTTPS Port=4900• EM Console Port = 7788• EM Console HTTPS Port= 7799• Config_home = /u01/oracle/product/OracleHomes2/gc_inst • EM_INSTANCE_HOST aa2prodora11.jstor.org (Standby machine)


Of all these ports the following are relevant to the load Balancer:

•EM Upload HTTP Port: 4889•EM UPLOAD HTTPS Port: 4900•EM Central Console HTTP Port: 7788•EM Central Console HTTPS Port: 7799


• Run OMSCA on Standby database using the backup file created

<OMS_HOME>/bin/omsca recover -ms -backup_file /opt/oracle/product/backup/opf_ADMIN_20120504_031016.bka -AS_HTTPS_PORT 7101 -MSPORT 7202 -MS_HTTPS_PORT 7301 -EM_NODEMGR_PORT 7403 -EM_ UPLOAD_PORT 4889 -EM_UPLOAD_HTTPS_PORT 4900 -EM_CONSOLE_PORT 7788 -EM_ CONSOLE_HTTPS_PORT 7799 -config_home /opt/oracle/product/omsmdw/gc_inst -EM_INSTANCE_HOST example.com

!!! Many errors at this stage, due Failed attempts

Installation Issues: MSEVERE: OMSCA-ERR:Post "Deploy and Repos Setup" operations failed

Check the following items :

-- Check RAM on Standby Machine – needs 6GB – to Install

-- If using 32 Bit Linux – Modify – create_domain.py (under omsca/scripts/wls

directory)Change - -Xmx1524m to –Xmx1324mChange –XX:MaxPermsize=612M to –

X:MaxPermsize=412M

-- Check for Lock in repository

Installation Issues• Check for lock in the repository

• Due to multiple tries there was a lock in the repository during the plugin deployment stage.

• select host_url, oms_id, status, is_first_oms, em_release_version from gc_oms_info;

• select mp_guid,file_id,file_name,file_type from mgmt_mp_files;

• select * from em_plugin_lock_track;

Installation Issues:

• First Issue – wrong version of the database plugin

• I upgraded the database plugin but didn’t include it in the standby installation

• Copied latest “DB” plugin and reran pluginInstall.sh

Installation Issues:• 2nd Issue – Plugin Issue and multiple attempts

caused lock in EM database (SYSMAN)

• Removed record from lock table em_plugin_lock_track

• Ran $OMS_HOME/bin - ./pluginca –action deploy –isFirstOMS false –plugins

• Started OMS

Final Installation Steps

If you haven’t installed the agent on the standby host:

$<AGENT_HOME>/sysman/install/agentDeploy.sh AGENT_BASE_DIR=<middleware_ home>/agent OMS_HOST=<second_oms_host_name> EM_UPLOAD_PORT=<second_oms_ port> AGENT_REGISTRATION_PASSWORD=<password> -configOnly

Note: If you have Server Load Balancer (SLB) configured, then directly enter the host name and port number of the SLB. If SLB is not configured, then enter the secure upload port of the first OMS.

Final Installation Steps After agent is installed and deployed

• Deploy the required plug-ins on the management agent

Secure the agent - $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks (If applicable)

In the new OMS Grid control (standby) – discover the Oracle Weblogic services

• Targets, Middleware, click EMGC_DOMAIN

• From EMGC_DOMAIN home page

• Select Farm Menu, Refresh Weblogic Domain, click continue till 2nd management server is discovered on the standby OMS Host.

Installation logs / help• Plugin Logs:

• $OMS_HOME/cfgtoollogs/pluginca/*

• $OMS_HOME/cfgw/CfmLogger_<timestamp>.log

• Repository Configuration logs

• $OMS_HOME/cfgw/CfmLogger_<timestamp>.log

• $OMS_HOME/sysman/log/schemamanager/m_<timestamp>/*

• MDS Schema Configuration Assistant

• $OMS_HOME/sysman/log/schemamanager/m_<timestamp>/*

• $OMS_HOME/cfgtoollogs/cfgfw/emdcreate*.log

• More Log file and Troubleshooting tips:

http://docs.oracle.com/cd/E24628_01/install.121/e24080/appdx_troubleshooting.htm#CEGFAHIG

What We Covered

•High Available architecture options

•Silent Standby OMS installations and gotchas

References

Deploying a Highly Available EM12C Cloud Control – An Oracle White Paper – May 2012 - # 1631423.pdf

www.dbakevlar.com – Kellyn Pot’Vin’s blog contains many great ideas about EM12c

e24089.pdf – Cloud Control Advanced Installation and Configuration Guide (section 5 - installing Add'l OMS in silent mode)

References

https://forums.oracle.com/forums/category.jspa?categoryID=70 – OTN Discussion on EM

www.oracle.com – Documentation is not bad for many of the basic items with EM12c

Keeping your OMS Tidy – Ray Smith - IOUG Tips and Best Practices – 7th edition - www.ioug.org

http://oramanageability.wordpress.com - Ray Smiths Blog on Cloud Control

http://www.ioug.org/

http://oramanageability.wordpress.com/

References

www.pythian.com – MySQL plug-in and instructions

oemgc.wordpress.com – Rob Zoeteweij – Good information on Incident management and ITL, including a blog

www.slideshare.net/Enkitec/em12c-monitoring-metric-extensions-and-performance-pages#btnPrevious – Em12C Monitoring, Metric Extensions and Performance pages – By Kellyn Pot’Vin – Enkitec

EM12C Log Cleanup – Script to clean up logs and information

Questions? Comments?

Thank You for Attending

Please fill out your Evaluations

www.ioug.org/eval

Cloud Control Recipes for EM12C Session # 941

Mike Gangler•Email: [email protected]

• Twitter: @mjgangler• Blog: mjgangler.wordpress.com

mailto:[email protected]

em12c high availability without slb and rac

Technology