rac troubleshooting and diagnosability sangam2016

Troubleshooting and Diagnosing Oracle RAC in the Private CloudSandeshRao,SeniorDirector,RACDevelopment

SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

Confidential– OracleRestricted 2

Agenda

• ArchitecturalOverview• TroubleshootingScenarios• ProactiveandReactivetools• Q&A

GridInfrastructureOverview

• OracleClusterwareisrequiredfor11gR2+RACdatabases• OracleClusterwarecanmanagenonRACdatabaseresourcesusingagents• OracleClusterwarecanmanageHAforanyBusinessCriticalApplicationwithagentinfrastructurealsocalledXAG–OraclepublishesBundledAgentsforsomenonRACDBresources

• SAP,GoldenGate,Siebel,Apache..

OracleGridInfrastructure

• GridInfrastructureisthenameforthecombinationof:-–OracleClusterReadyServices(CRS)–OracleAutomaticStorageManagement(ASM)

• TheGridHomecontainsthesoftwareforbothproducts• CRScanalsobeStandaloneforASMand/orOracleRestart• CRScanrunbyitselforincombinationwithothervendorclusterware• GridHomeandRDBMShomemustbeinstalledindifferentlocations

– TheinstallerlockstheGridHomepathbysettingrootpermissions.

• CRSrequiressharedOracleClusterRegistry(OCR)andVotingfiles–MustbeinASMorCFS–OCRbackedupevery4hoursautomaticallyGIHOME/cdata– Kept4,8,12hours,1day,1week– Restoredwithocrconfig– VotingfilebackedupintoOCRateachchange.– Votingfilerestoredwithcrsctl

• FornetworkCRSrequires–One/multiplehighspeed,lowlatency,redundantprivatenetworkforinternodecommunications

– Thinkofinterconnectasamemorybackplaneforthecluster– Shouldbeaseparatephysicalnetwork ormanagedconvergednetwork– VLANSaresupported– Usedfor:-

• Clusterwaremessaging• RDBMSmessagingandblocktransfer• ASMmessaging• HANFSforblocktraffic

• OnlyonesetofClusterwaredaemonscanrunoneachnode• TheCRSstackisspawnedfromOracleHAServicesDaemon(ohasd)• OnUnixohasd runsoutofinittab withrespawn• Anodecanbeevictedwhendeemedunhealthy

–MayrequirerebootbutatleastCRSstackrestart(rebootless restart)– IPMIintegrationordiskmon incaseofExadata

• CRSprovidesClusterTimeSynchronizationservices– Alwaysrunsbutinobservermodeifntpd configured

GridInfrastructureProcesses11.2+Agentschangeeverything.• Multi-threadedDaemons• Managemultipleresourcesandtypes• Implementsentrypointsformultipleresourcetypes

– Start,stop check,clean,fail

• oraagent,orarootagent,applicationagent,scriptagent,cssdagent• SingleprocessstartedfrominitonUnix(ohasd)• Diagrambelowshowsallcoreresources

GridInfrastructureProcesses

Level1

Level2a

Level2b

Level3

Level4a

Level4b

Level0

GridInfrastructureProcessesInitScripts• /etc/init.d/ohasd (locationO/Sdependent)

– RCscriptwith“start”and“stop”actions– InitiatesOracleClusterware autostart– ControlfilecoordinateswithCRSCTL

• /etc/init.d/init.ohasd (locationO/Sdependent)–OHASDFrameworkScriptrunsfrominit/upstart– ControlfilecoordinateswithCRSCTL– NamedpipesyncswithOHASD

• Level1:OHASDSpawns:– cssdagent - AgentresponsibleforspawningCSSD– orarootagent- Agentresponsibleformanagingallrootownedohasd resources– oraagent - Agentresponsibleformanagingalloracleownedohasd resources– cssdmonitor - MonitorsCSSDandnodehealth(alongwiththecssdagent)

• Level2a:OHASDrootagent spawns:– CRSD- Primarydaemonresponsibleformanagingclusterresources.– CTSSD- ClusterTimeSynchronizationServicesDaemon– Diskmon (Exadata)– ACFS(ASMClusterFileSystem)Drivers

• Level2b:OHASDoraagent spawns:–MDNSD– MulticastDNSdaemon– GIPCD– GridIPCDaemon– GPNPD– GridPlugandPlayDaemon– EVMD– EventMonitorDaemon– ASM– ASMinstancestartedhereasmayberequiredbyCRSD

• Level3:CRSDspawns:– orarootagent - Agentresponsibleformanagingallrootownedcrsd resources.– oraagent - Agentresponsibleformanagingallnonroot ownedcrsd resources.

• OneisspawnedforeveryuserthathasCRSresourcestomanage.

• Level4:CRSDoraagent spawns:– ASMResouce - ASMInstance(s)resource(proxyresource)– Diskgroup- Usedformanaging/monitoringASMdiskgroups.– DBResource- UsedformonitoringandmanagingtheDBandinstances– SCANListener- Listenerforsingleclientaccessname,listeningonSCANVIP– Listener- NodelistenerlisteningontheNodeVIP– Services- Usedformonitoringandmanagingservices– ONS- OracleNotificationService– eONS - EnhancedOracleNotificationService(pre11.2.0.2)– GSD- For9ibackwardcompatibility– GNS(optional)- GridNamingService- Performsnameresolution

Startup Sequence

TroubleshootingScenariosClusterStartup ProblemTriage(11.2+)

StartupSequence

ps –ef|grep init.ohasdps –ef|grep ohasd.bin Running?

NO crsctl config crsohasd.log Obvious?

NO EngageOracleSupportEngageSysadminTeam

ClusterStartupDiagnosticFlow

TFACollector

Running?

EngageSysadminTeam

ohasd.logagentlogsprocesslogs

Obvious?YES

EngageSysadminTeam

EngageOracleSupportSysadminTeam

TFACollectorohasd.logOLRperms

Comparereferencesystem

Obvious?YESNO

TFACollector EngageSysadminTeam

EngageOracleSupportSysadminTeam

• MulticastDomainNameServiceDaemon(mDNS(d))– UsedbyGridPlugandPlaytolocateprofilesinthecluster,aswellasbyGNStoperformnameresolution.ThemDNS processisabackgroundprocessonLinuxandUNIXandonWindows.

– Usesmulticastforcacheupdatesonserviceadvertisementarrival/departure.– Advertises/servesonallfoundnodeinterfaces.– LogisGI_HOME/log/<node>/mdnsd/mdnsd.log

Troubleshooting ScenariosCluster Startup Problem Triage

<?xmlversion="1.0"encoding="UTF-8"?>

<gpnp:GPnP-ProfileVersion="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile"xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profilegpnp-profile.xsd"ProfileSequence="6" ClusterUId="b1eec1fcdd355f2bbf7910ce9cc4a228"ClusterName="staij-cluster"PALocation="">

<gpnp:Network-Profile><gpnp:HostNetwork id="gen"HostName="*"><gpnp:Network id="net1"IP=”192.168.1.0"Adapter="eth0"Use="public"/><gpnp:Network id="net2"IP=”192.168.2.0"Adapter="eth1“Use="cluster_interconnect"/></gpnp:HostNetworkcss"></gpnp:Network-Profile>

<orcl:CSS-Profileid="DiscoveryString="+asm"LeaseDuration="400"/>

<orcl:ASM-Profileid="asm"DiscoveryString=""SPFile="+SYSTEM/staij-cluster/asmparameterfile/registry.253.693925293"/>

<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethodAlgorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethodAlgorithm="http://www.w3.org/2001/10/xml-exc-c14n#"><InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"PrefixList="gpnp orclxsi"/></ds:Transform></ds:Transforms><ds:DigestMethodAlgorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>x1H9LWjyNyMn6BsOykHhMvxnP8U=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>N+20jG4=</ds:SignatureValue></ds:Signature>

</gpnp:GPnP-Profile>

• cssdagentandmonitor– Samefunctionalityinbothagentandmonitor– Functionalityofseveralpre-11.2daemonsconsolidatedinboth

• OPROCD– systemhang• OMON– oracleclusterwaremonitor• VMON– vendorclusterwaremonitor

– Runrealtime withlockeddownmemory,likeCSSD– Providesenhancedstabilityanddiagnosability– Logsare

• GI_HOME/log/<node>/agent/oracssdagent_root/oracssdagent_root.log• GI_HOME/log/<node>/agent/oracssdmonitor_root/oracssdmonitor_root.log• 12.1– ORACLE_BASE/diag/node/agent/..

NodeEvictions

EvictionScenario

Clusteralertocssd.log

NHB?1050693.11534949.11546004.1

Engagenetworking teamYES

DHB?1549428.11466639.1 YES

Obvious?

Fenced?Resourcestarvation

NodeEvictionDiagnosticFlow

Troubleshooting Scenarios

ResourceStarvation?

EngageOracleSupport

Engagesysadminteam

Engagestorageteam

1531223.11328466.1Systemlog

Engageappropriate

Resolved?NO

Freememory?CPUload?

NodeResponse?

TFACollector

MissingNetworkHeartbeat(1)• ocssd.logfromnode1

• ===>sendingnetworkheartbeatsothernodes.Normally,thismessageisoutputonceevery5messages(seconds)

• 2016-08-1317:00:20.023:[CSSD][4096109472]clssnmSendingThread:sendingstatusmsg toallnodes

• 2016-08-1317:00:20.023:[CSSD][4096109472]clssnmSendingThread:sent5statusmsgs toallnodes

• ===>Thenetworkheartbeatisnotreceivedfromnode2(drrac2)for15consecutiveseconds.

• ===>Thismeansthat15networkheartbeatsaremissingandisthefirstwarning(50%threshold).

• 2016-08-1317:00:22.818:[CSSD][4106599328]clssnmPollingThread:nodedrrac2(2)at50%heartbeatfatal,removalin14.520seconds

• 2016-08-1317:00:22.818:[CSSD][4106599328]clssnmPollingThread:nodedrrac2(2)isimpendingreconfig,flag132108,misstime 15480

• ===>continuingtosendthenetworkheartbeatsandlogmessagesonceevery5messages

• ===>75%thresholdofmissingnetworkheartbeatisreached.Thisissecondwarning.

MissingNetworkHeartbeat(2)• ===>continuingtosendthenetworkheartbeatsandlogmessagesonceevery5messages

• ===>continuingtosendthenetworkheartbeats,butthemessageisloggedafter4messages

• ===>Lastwarningshowsthat90%thresholdofthemissingnetworkheartbeatisreached.

• ===>Theevictionwilloccurin2.49seconds.

• 2016-08-1317:00:34.841:[CSSD][4106599328]clssnmPollingThread:nodedrrac2(2)at90%heartbeatfatal,removalin2.490seconds,seedhbimpd 1

• ===>Evictionofnode2(drrac2)started

• 2016-08-1317:00:37.337:[CSSD][4106599328]clssnmPollingThread:Removalstartedfornodedrrac2(2),flags0x2040c,state3,wt4c0

• ===>Thisshowsthatthenode2isactivelyupdatingthevotingdisks

• 2016-08-1317:00:37.340:[CSSD][4085619616]clssnmCheckSplit:Node2,drrac2,isalive,DHB(1281744040,1396854)morethandisktimeoutof27000afterthelastNHB(1281744011,1367154)

MissingNetworkHeartbeat(3)• ===>Evictingnode2(drrac2)

• 2016-08-1317:00:37.340:[CSSD][4085619616](:CSSNM00007:)clssnmrEvict:Evictingnode2,drrac2,fromtheclusterinincarnation169934272,nodebirthincarnation169934271,deathincarnation169934272,stateflags 0x24000

• ===>Reconfiguredtheclusterwithoutnode2

• 2016-08-1317:01:07.705:[CSSD][4043389856]clssgmCMReconfig:reconfigurationsuccessful,incarnation169934272with1nodes,localnodenumber1,masternodenumber1

MissingNetworkHeartbeat(4)• ocssd.logfromnode2:• ===>Loggingthemessagetoindicate5networkheartbeatsaresenttoothernodes

• ===>Firstwarningofreaching50%thresholdofmissingnetworkheartbeats

• 2016-08-1317:00:26.213:[CSSD][4073040800]clssnmPollingThread:nodedrrac1(1)isimpendingreconfig,flag394254,misstime 15460

• ===>Loggingthemessagetoindicate5networkheartbeatsaresenttoothernodes

• ===>Secondwarningofreaching75%thresholdofmissingnetworkheartbeats

MissingNetworkHeartbeat(5)• ===>Loggingthemessagetoindicate4networkheartbeatsaresent

• ===>Thirdwarningofreaching90%thresholdofmissingnetworkheartbeats

• 2016-08-1317:00:38.236:[CSSD][4073040800]clssnmPollingThread:nodedrrac1(1)at90%heartbeatfatal,removalin2.460seconds,seedhbimpd 1

• ===>Loggingthemessagetoindicate5networkheartbeatsaresenttoothernodes

• ===>Evictionstartedfornode1(drrac1)

• 2016-08-1317:00:40.702:[CSSD][4073040800]clssnmPollingThread:Removalstartedfornodedrrac1(1),flags0x6040e,state3,wt4c0

• ===>Node1isactivelyupdatingthevotingdisk,sothisisasplitbraincondition

• 2016-08-1317:00:40.706:[CSSD][4052061088]clssnmCheckSplit:Node1,drrac1,isalive,DHB(1281744036,1243744)morethandisktimeoutof27000afterthelastNHB(1281744007,1214144)

• 2016-08-1317:00:40.706:[CSSD][4052061088]clssnmCheckDskInfo:Mycohort:2

• 2016-08-1317:00:40.707:[CSSD][4052061088]clssnmCheckDskInfo:Survivingcohort:1

MissingNetworkHeartbeat(6)• ===>Node2isabortingitselftoresolvethesplitbrainandensuretheclusterintegrity

• 2016-08-1317:00:40.707:[CSSD][4052061088](:CSSNM00008:)clssnmCheckDskInfo:Abortinglocalnodetoavoidsplitbrain.Cohortof1nodeswithleader2,drrac2,issmallerthancohortof1nodesledbynode1,drrac1,basedonmaptype2

• 2016-08-1317:00:40.707:[CSSD][4052061088]###################################

• 2016-08-1317:00:40.707:[CSSD][4052061088]clssscExit:CSSDaborting fromthreadclssnmRcfgMgrThread

• 2016-08-1317:00:40.707:[CSSD][4052061088]###################################

MissingNetworkHeartbeat(7)• Observations1. Bothnodesreportedmissingheartbeatsatthesametime2. Bothnodessentheartbeatstoothernodesallthetime3. Node2aborteditselftoresolvesplitbrain

• Conclusion1. Thisislikelyanetworkproblem,engagenetworkteam2. CheckOSWatcher output(netstat andtraceroute)

1. Configureprivate.net file,notconfiguredbydefault

3. CheckCHMOS4. Checksystemlog

VotingDiskAccessProblem(1)

ocssd.log:

===>Thefirsterrorindicatingthatitcouldnotreadvotingdisk-- firstmessagetoindicateaproblemaccessingthevotingdisk

2016-08-1318:31:19.787:[SKGFD][4131736480]ERROR:-9(Error27072,OSError(LinuxError:5:Input/outputerror

Additionalinformation:4

Additionalinformation:-1)

2016-08-1318:31:19.787:[CSSD][4131736480](:CSSNM00060:)clssnmvReadBlocks:readfailedatoffset529of/dev/sdb8

2016-08-1318:31:19.802:[CSSD][4131736480]clssnmvDiskAvailabilityChange:votingfile/dev/sdb8nowoffline

VotingDiskAccessProblem(2)====>Theerrormessagethatshowsaproblemaccessingthevotingdiskrepeatsonceevery4seconds

2016-08-1318:31:23.782:[CSSD][150477728]clssnmvDiskOpen:Opening/dev/sdb8

2016-08-1318:31:23.782:[SKGFD][150477728]Handle0xf43fc6c8fromlib:UFS::fordisk:/dev/sdb8:

2016-08-1318:31:23.782:[CLSF][150477728]Openedhdl:0xf4365708fordev:/dev/sdb8:

VotingDiskAccessProblem(3)

====>Thelasterrorthatshowsaproblemaccessingthevotingdisk.

====>Notethatthelastmessageis200secondsafterthefirstmessage

====>becausethelongdisktimeout is200seconds

2016-08-1318:34:37.423:[CSSD][150477728]clssnmvDiskOpen:Opening/dev/sdb8

2016-08-1318:34:37.423:[CLSF][150477728]Openedhdl:0xf4336530fordev:/dev/sdb8:

VotingDiskAccessProblem(4)====>Thismessageshowsthatocssd.bintriedaccessingthevotingdiskfor200seconds

2016-08-1318:34:38.205:[CSSD][4110736288](:CSSNM00058:)clssnmvDiskCheck:NoI/Ocompletionsfor200880msforvotingfile/dev/sdb8)

====>ocssd.binabortsitselfwithanerrormessagethatthemajorityofvotingdisksarenotavailable.Inthiscase,therewasonlyonevotingdisk,butifthreevotingdiskswereavailable,aslongastwovotingdisksareaccessible,ocssd.binwillnotabort.

2016-08-1318:34:38.206:[CSSD][4110736288](:CSSNM00018:)clssnmvDiskCheck:Aborting,0of1configuredvotingdisksavailable,need1

2016-08-1318:34:38.206:[CSSD][4110736288]###################################

2016-08-1318:34:38.206:[CSSD][4110736288]clssscExit:CSSDabortingfromthreadclssnmvDiskPingMonitorThread

2016-08-1318:34:38.206:[CSSD][4110736288]###################################

• ConclusionThevotingdiskwasnotavailable,engagestorageteam

• Timesynchronisationissue• ClusterTimeSynchronisationServicesdaemon

– ProvidestimemanagementinaclusterforOracle.• ObservermodewhenVendortimesynchronisations/wisfound

– LogstimedifferencetotheCRSalertlog• ActivemodewhennoVendortimesyncs/wisfound

Node Eviction TriageTroubleshooting Scenarios

• ClusterReadyServicesDaemon– TheCRSDdaemonisprimarilyresponsibleformaintainingtheavailabilityofapplicationresources,suchasdatabaseinstances.CRSDisresponsibleforstartingandstoppingtheseresources,relocatingthemwhenrequiredtoanothernodeintheeventoffailure,andmaintainingtheresourceprofilesintheOCR(OracleClusterRegistry).Inaddition,CRSDisresponsibleforoverseeingthecachingoftheOCRforfasteraccess,andalsobackinguptheOCR.

– LogfileisGI_HOME/log/<node>/crsd/crsd.log• Rotationpolicy10-50M• Retentionpolicy10logs• Dynamicin12.1andcanbechanged

• CRSDoraagent– CRSD’soraagent manages

• alldatabase,instance,serviceanddiskgroupresources• nodelisteners• SCANlisteners,andONS

– IftheGridInfrastructureownerisdifferentfromtheRDBMShomeownerthenyouwouldhave2oraagents eachrunningasoneoftheinstallationowners.Thedatabase,andserviceresourceswouldbemanagedbytheRDBMShomeownerandotherresourcesbytheGridInfrastructurehomeowner.

– Logfileis• GI_HOME/log/<node>/agent/crsd/oraagent_<user>/oraagent_<user>.log

• CRSDorarootagent– CRSD’srootagent manages

• GNSandit’sVIP• NodeVIP• SCANVIP• networkresources.

– Logfileis• GI_HOME/log/<node>/agent/crsd/orarootagent_root/oraagent_root.log

• Agentreturncodes– Checkentrymustreturnoneofthefollowingreturncodes:

• ONLINE• UNPLANNED_OFFLINE

– Target=online,mayberecoveredfailedover• PLANNED_OFFLINE• UNKNOWN

– Cannotdetermine,ifpreviouslyonline,partialthenmonitor• PARTIAL

– Someofaresourcesservicesareavailable.Instanceupbutnotopen.• FAILED

– Requirescleanaction

§ Importantlogsandtraces§ 11.2– DatabasesonlyuseADR

• GridInfrastructurefilesin$GI_HOME/log/<node_name>/<component_name>– $GI_HOME/log/myHost/cssd– $GI_HOME/log/myHost/alertmyHost.log

§ 12.1– GridInfrastructureandDatabaseuseADR§ DifferentlocationsforGridInfrastructureandDatabases§ GridInfrastructure

• Alert.log,cssd.log,csrd.log,etc

§ Databases§ Alert.log,backgroundprocesstraces,foregroundprocesstraces

Automatic Diagnostic Repository (ADR)Troubleshooting Scenarios

• Whatifissuesweredetectedbeforetheyhadanimpact?

• Whatifyouwerenotifiedwithaspecificdiagnosisandcorrectiveactions?

• WhatifresourcebottlenecksthreateningSLAswereidentifiedearly?

• Whatifbottleneckscouldbeautomaticallyrelievedjustintime?

• Whatifdatabasehangsandnoderebootscouldbeeliminated?

Oracle’sDatabaseandClusterwareTools

Cluster Verification

Utility

ORAchkCluster Health

Monitor

Trace File Analyzer

Quality of Service

Management

Hang Manager

EXAchk

Cluster Health

Advisor

Memory Guard

Confidential– OracleRestricted

–Automatedriskidentificationandproactivenotificationbeforebusinessisimpacted

–HealthChecksbasedonmostimpactfulreoccurringproblemsacrossOraclecustomerbase

–Runsinyourenvironment– noneedtosendanythingtoOracle

–ScheduledemailHealthCheckreports

–Findingscanbeintegratedintoothertoolsofchoice

Oracle EXAchk/Orachk (Proactive)

EngineeredSystems

NonEngineeredSystems

OracleEXAchk

OracleORAchk

CommonFramework

Lightweight&nonintrusiveOracleStackHealthChecks

1. IncludedinbaseimageandlatestOEDA

2. DownloadlatestversionfromMyOracleSupport(install<1min)1070954.1

3. Autoupdatewhenlaterversionavailable

RollOut&MaintainEXAchk

1. Downloadtheorachk.ziptoyourlocalmachinefromMOSNote1268927.2

2. TransfertoadirectoryonthetargetSystem

3. Unziporachk.zipo Asowneroforacle

databaseorgridhome

Installation

• Profilesprovidelogicalgroupingofchecks whichareaboutsimilartopics

• Runonlychecksinaspecificprofile

• Runeverythingexceptchecksinaspecificprofile

Profiles

./exachk –profile <profile>

./exachk –excludeprofile <profile>

Profile Descriptionasm ASMChecksavdf AuditVaultConfigurationchecks

clusterware Oracleclusterwarecheckscontrol_VM ChecksonlyforControlVM(ec1-vm,ovmm,db,pc1,pc2).

Nocrossnodecheckscorroborate Exadatachecksneedsfurtherreviewbyusertodetermine

passorfaildba DBAChecksebs OracleE-BusinessSuitechecks

eci_healthchecks EnterpriseCloudInfrastructureHealthchecksecs_healthchecks EnterpriseCloudSystemHealthchecks

goldengate OracleGoldenGatecheckshardware HardwarespecificchecksforOracleEngineeredsystems

maa MaximumAvailabilityArchitectureChecksovn OracleVirtualNetworking

platinum Platinumcertificationcheckspreinstall Pre-installationchecksprepatch Checkstoexecutebeforepatchingsecurity Securitychecks

solaris_cluster SolarisClusterChecksstorage OracleStorageServerChecksswitch Infinibandswitchchecks

sysadmin Sysadminchecksuser_defined_checks Runuserdefinedchecksfromuser_defined_checks.xml

• Profilesprovidelogicalgroupingofchecks whichareaboutsimilartopics

• Runonlychecksinaspecificprofile

• Runeverythingexceptchecksinaspecificprofile

Profiles

./orachk –profile <profile>

./orachk –excludeprofile <profile>

Profile Descriptionasm ASMChecks

bi_middleware OracleBusinessIntelligencechecksclusterware Oracleclusterware checks

dba DBAChecksebs OracleE-BusinessSuitechecks

emagent Cloudcontrolagentchecksemoms CloudControlmanagementserverem Cloudcontrolchecks

goldengate OracleGoldenGate checkshardware HardwarespecificchecksforOracleEngineeredsystems

oam OracleAccessManagerchecksoim OracleIdentifyManagerchecksoud OracleUnifiedDirectoryserverchecksovn OracleVirtualNetworking

peoplesoft Peoplesoft bestpracticespreinstall Pre-installationchecksprepatch Checkstoexecutebeforepatchingsecurity Securitycheckssiebel SiebelChecks

solaris_cluster SolarisClusterChecksstorage OracleStorageServerChecksswitch Infiniband switchchecks

sysadmin Sysadmin checksuser_defined_checks Runuserdefinedchecksfromuser_defined_checks.xml

EnterpriseManagerIntegration

•CheckresultsintegratedintoEMcomplianceframeworkviaplugin

•ViewresultsinnativeEMcompliancedashboards

•Relatedchecksgroupedintocompliancestandards

•Viewtargetschecked,violations&averagescore

•Drilldownintocompliancestandardtoseeindividualcheckresults

•Viewbreakdownbytarget

• IntegrationisviatheEnterpriseManagerORAchkHealthchecksplugin withthefollowingSupport:

• Thefollowingprerequisitesmustbemetbeforeyoucandeploytheplug-in:o VerifythatyourEngineeredSystemshardwareandsoftwareareatthesupportedlevelasdescribedin SupportedHardwareandSoftwareVersions

o AllEngineeredSystemplug-insshouldbedeployed

o InfiniBand switchesandstoragecellsshouldbeanEnterpriseManager-managedtargetfortherespectiveengineeredsystem

o Expectpackageshouldbeinstalledonthehosts

EnterpriseManagerPluginPrerequisites

HardwareTypes SupportedByPlugin

Exadata(physicalconfigurationonly) YesExadata(virtualconfiguration) NoRecoveryappliance YesExalogic(physicalconfiguration) YesExalogic(virtualizedconfiguration) YesOracleSuperCluster NoOraclePrivateCloudMachine No

JSONOutputtoIntegratewithKibana,ElasticSearchetc

OracleHealthCheckCollectionManagerDashboard

OracleStackCoverage• EngineeredSystems

• OracleExadataDatabaseMachine• OracleSuperCluster• OraclePrivateCloudAppliance• OracleDatabaseAppliance• OracleBigDataAppliance• OracleExalogicElasticCloud• OracleExalyticsIn-MemoryMachine• OracleZeroDataLossRecoveryAppliance• OracleZFSStorageAppliance

• ASR

• Systems• OracleSolaris• Crossstackchecks• SolarisCluster• OVN

• Oracle Database• StandaloneDatabase• GridInfrastructure&RAC• Maximum AvailabilityArchitecture(MAA)

Scorecard• Upgrade ReadinessValidation• Golden Gate

• EnterpriseManagerCloudControl• Repository• Agent• OMS

• Middleware• ApplicationContinuity• OracleIdentifyandAccessManagement

Suite(OracleIAM)

• E-BusinessSuite• OraclePayables• OracleWorkflow• OraclePurchasing• OracleOrderManagement• OracleProcessManufacturing• OracleReceivables• OracleFixedAssets• OracleHCM• OracleCRM• OracleProjectBilling

• Siebel• Databasebestpractices

• PeopleSoft• Databasebestpractices

• SAP

• EXAdatabestpractices

GeneratesDiagnosticMetricsViewofClusterandDatabasesClusterHealthMonitor(CHM)

ologgerd(master)

osysmond

12cGridInfrastructureManagementRepository

• Alwayson- Enabledbydefault• ProvidesDetailed OSResourceMetrics• AssistsNodeevictionanalysis• Locallylogsallprocessdata• Usercandefinepinnedprocesses• ListenstoCSSandGIPCevents• Categorizesprocessesbytype• Supportsplug-incollectors(ex.traceroute,netstat,ping,etc.)

• NewCSVoutputforeaseofanalysis

OSData OSData

OSData

Confidential– OracleInternal/Restricted/HighlyRestrictedConfidential– OracleRestricted

Oclumon CLIorFullIntegrationwithEMCloudControlClusterHealthMonitor(CHM)

Confidential– OracleInternal/Restricted/HighlyRestrictedConfidential– OracleRestricted

WhyTFA?(Proactiveandreactive)

Providesoneinterfaceforalldiagnosticneeds

Collectsdataacrosstheclusterandconsolidatesitinoneplace

Collectsallrelevantdiagnosticdataatthetimeoftheproblem,withonlywhatisneededtodiagnosetheproblem

Reducestimerequiredtoobtaindiagnosticdata,whichsavesyourbusinessmoney

50OracleConfidential– Internal

• AllmajorOperatingSystems aresupported– Linux(OEL,RedHat,SUSE,Itanium&zLinux)

–OracleSolaris(SPARC&x86-64)– AIX– HPUX(Itanium&PA-RISC)–Windows

• AllOracleDatabase&Gridversions10.2+aresupported

• YouprobablyalreadyhaveTFAinstalledasitisincludedwith:–OracleGridInfrastructure:

• 11.2.0.4+• 12.1.0.2+• 12.2.0.1+

–OracleDatabase:• 12.2.0.1+

• AlsoavailablefromDoc1513912.251

SupportedPlatformsandVersions

MonitoringByTFA&AutomatedCollections

Automaticallydetectevent

Collect&packagerelevant

diagnostics

NotifyrelevantDBAandorSysAdminby

UploadcollectiontoOracleSupportforfurtherhelp

Significantproblemoccurs

TFADBA(s)/SysAdmin(s)

OracleGridInfrastructure&Database(s)

• Trim&collectallimportantlogfilesupdatedinthepast12hours:

• CollectaproblemspecificServiceRequestDataCollection(SRDC):

Collect

tfactl diagcollect

• Collectionsstoredintherepository directory• Changediagcollecttimeframewith–since<n>h|d• Forlistoftypesofsrdc collectionsusetfactldiagcollect-srdc help

tfactl diagcollect -srdc ora600

TFAdbglevel profiles• Example

– tfactl dbglevel -setnode_eviction

–wouldbeusedforenhancingdiagnosticswhennode evictions arethebeinginvestigatedandwouldperformthefollowingoperationinternally• crsctl setlogcss "CSSD=4"• crsctl setlogcss "CSSDNMC=4"• crsctl setlogcss "CLSF=4"• crsctl setlogcss "CSSDGMCC=4"• crsctl setlogcss "CSSDGMPC=4"

• Toreverttotheoriginalordefaultlogginglevelsthefollowingcommand– $tfactl dbglevel -unsetnode_eviction

• wouldperformthefollowingoperationsinternally• crsctl setlogcss "CSSD=2"• crsctl setlogcss "CSSDNMC=2"• crsctl setlogcss "CLSF=0"• crsctl setlogcss "CSSDGMCC=2"• crsctl setlogcss "CSSDGMPC=2"

• Inthiswayofsettingthelogginglevelsadegreeofautomationandsimplificationis

OracleConfidential– Internal/Restricted/HighlyRestricted 54

LogManagement• TFAwillbelogmanagementinterface forsoftwarestack– Rotatelogs– Archivelogs– Purgeoldlogs

• Intelligentlogmanagementbasedonunderstandingofwhatisinlogsandwhatisstillimportant

OracleConfidential– Internal/Restricted/HighlyRestricted 55

TFALogManagement AllLogsAcross

SoftwareStack

Rotate

Archive

Actionthroughpredictionoruser

SetEmailNotificationAddresses

tfactl set notificationAddress=john.doe@oracle.com

Automaticallydetectevent

Collect&packagerelevant

diagnostics

NotifyrelevantDBAandorSys

Adminbyemail

Uploadcollectionto

OracleSupportforfurtherhelp

Significantproblemoccurs

TFADBA(s)/SysAdmin(s)

OracleGridInfrastructure&Database(s)

tfactl set notificationAddress=oracle:another.person@oracle.com

• TFAcansendemailnotificationwhensignificantproblemsaredetected

• Tosetnotificationemailforanyproblemdetected:

• TosetnotificationemailforspecificORACLE_HOMEsincludetheOSowner:

OracleConfidential– Internal

• Analyzeallimportantrecentlogentries: • Searchrecentlogentries:

Analyze

tfactl analyze –since 1d tfactl analyze -search “ora-00600" -since 8h

• TFAincludesallkeydatabasesupporttools• tfactlprovidesasingleinterfacetothemall

Analyze

MostoftheseSupporttoolsareonlyavailableintheMyOracleSupportdownload,theyarenotincludedinthebaseGridorDatabaseinstall

Tool Description DetailsORAchk OracleStackHealthChecksonnon-engineered

systems1268927.2

EXAchk OracleStackHealthChecksonEngineeredSystems

1070954.1

oswatcher CollectandarchiveOSmetrics,usefulforinstance/nodeevictions&performanceIssues

301137.1

procwatcher Automate&capturedatabaseperformancediagnostics&sessionlevelhangs

459694.1

oratop Nearreal-timedatabasemonitoring 1500864.1sqlt CaptureSQLtracedateusefulfortuning 215187.1

alertsummary ProvidessummaryofeventsforoneormoredatabaseorASMalertfilesfromallnodes

ls ListsallfilesTFAknowsaboutforagivenfilenamepatternacrossallnodes

Tool Descriptionpstack Generateprocessstackforspecifiedprocessesacrossallnodes

grep Searchalertortracefileswithagivendatabaseandfilenamepattern,forasearchstring.

summary Highlevelsummaryoftheconfigurationvi Openalertortracefilesforviewingagivendatabaseandfile

namepatterninthevieditortail Runatailonanalertortracefilesforagivendatabaseandfile

namepatternparam ShowalldatabaseandOSparametersthatmatchaspecified

patterndbglevel SetandunsetmultipleCRStracelevelswithonecommandhistory Showtheshellhistoryforthetfactlshellchanges Reportanynotedchangesinthesystemsetupoveragiven

timeperiod.Thisincludesdatabaseaparameters,OSparameters,patchesappliedetc

• Usesrdc <incidenttype>:• Tospecifysid use–sid <oraclesid>• Tospecifydatabaseuse–db<dbname>• Tospecifyincidentdate&timeuse–inc_date <YYYY-MM-DD>-inc_time <HH:MM:SS>

• TouploaddirectlytotheSRuse–sr<SR#>

• Fordbperf usetheseparameterstospecifythegood&badperformanceperiodstocompare:

IncidentBasedCollectionswithSRDC

tfactl srdc ora4030

IncidentType Descriptionora4030 ForORA-04030errorsora4031 ForORA-04031errorsdbperf Forbasicdbperformanceproblemsora600 For ORA-00600errorsora700 For ORA-00700errorsora7445 For ORA-07445errors

tfactl srdc ora4030 -sid orcl –db RDBMS121 \-inc_date 2016-06-15 -inc_time 02:48:23 \-sr 3-123456789

Parameter Descriptionperf_base_sd Startdateforagoodperformanceperiodperf_base_st Starttimeforagoodperformanceperiodperf_base_ed Enddateforagoodperformanceperiodperf_base_et Endtimeforagoodperformanceperiodperf_comp_sd Startdateforabadperformanceperiodperf_comp_st Starttimeforabadperformanceperiodperf_comp_ed Enddateforabadperformanceperiodperf_comp_et Endtimeforabadperformanceperiod

tfactl srdc dbperf –db RDBMS121 \–perf_base_sd 2016-06-15 –perf_base_st 01:30:00 \–perf_base_ed 2016-06-15 –perf_base_et 02:00:00 \–perf_comp_sd 2016-06-16 –perf_comp_st 09:30:00 \–perf_comp_ed 2016-06-16 –perf_comp_et 10:00:00

ClusterHealthAdvisor(CHA)*DiscoversPotentialCluster&DBProblems- NotifieswithCorrectiveActions

OSData

• Alwayson- Enabledbydefault• Detectsnodeanddatabaseperformanceproblems

• Provides early-warningalertsandcorrectiveaction

• Supports on-sitecalibrationtoimprovesensitivity

• Integrated intoEMCCIncidentManagerandnotifications

• StandaloneInteractiveGUITool

DBData

NodeHealth

PrognosticsEngine

DatabaseHealth

PrognosticsEngine

*RequiresandIncludedwithRACorR1NLicense

Oracle12cHangManager

• Alwayson- Enabledbydefault

• Reliablydetectsdatabasehangsanddeadlocks

• Autonomouslyresolvesthem

• SupportsQoSPerformanceClasses,RanksandPoliciestomaintainSLAs

• Logsalldetectionsandresolutions

• NewSQLinterfacetoconfiguresensitivity(Normal/High)andtracefilesizes

AutonomouslyPreservesDatabaseAvailabilityandPerformance Session

EVALUATE

DETECT

ANALYZE

VERIFY

Victim

QoSPolicy

FullResolutionDumpTraceFileandDBAlertLogAuditReportsOracle12cHangManager

Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trcOracle Database 12c Enterprise Edition Release 12.2.0.0.0 - 64bit BetaWith the Partitioning, Real Application Clusters, OLAP, Advanced Analyticsand Real Application Testing optionsBuild label: RDBMS_MAIN_LINUX.X64_151013ORACLE_HOME: …/3775268204/oracleSystem name: LinuxNode name: slc05kyrRelease: 2.6.39-400.211.1.el6uek.x86_64Version: #1 SMP Fri Nov 15 13:39:16 PST 2013Machine: x86_64VM name: Xen Version: 3.4 (PVM)Instance name: hm62Redo thread mounted by this instance: 2Oracle process number: 19Unix process pid: 12656, image: oracle@slc05kyr (DIA0)

*** 2015-10-13T16:47:59.541509+17:00*** SESSION ID:(96.41299) 2015-10-13T16:47:59.541519+17:00*** CLIENT ID:() 2015-10-13T16:47:59.541529+17:00*** SERVICE NAME:(SYS$BACKGROUND) 2015-10-13T16:47:59.541538+17:00*** MODULE NAME:() 2015-10-13T16:47:59.541547+17:00*** ACTION NAME:() 2015-10-13T16:47:59.541556+17:00*** CLIENT DRIVER:() 2015-10-13T16:47:59.541565+17:00

2015-10-13T16:47:59.435039+17:00Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353):ORA-32701: Possible hangs up to hang ID=1 detectedIncident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc2015-10-13T16:47:59.506775+17:00DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2

due to a GLOBAL, HIGH confidence hang with ID=1.Hang Resolution Reason: Automatic hang resolution was performed to free a

significant number of affected sessions.DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1.

In the alert log on the instance local to the session (instance 2 in this case), we see the following:

2015-10-13T16:47:59.538673+17:00Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753):ORA-32701: Possible hangs up to hang ID=1 detectedIncident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc

2015-10-13T16:48:04.222661+17:00DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1

requested by master DIA0 process on instance 1Hang Resolution Reason: Automatic hang resolution was performed to free a

significant number of affected sessions.by terminating session sid:40 with serial # 43179 (ospid:13031)

Oracle12cDomainServicesCluster(DSC)

• HostsFrameworkasServices• Reduceslocalresourcefootprint• Centralizesmanagement• Speedsdeploymentandpatching• OptionalSharedStorage• Supportsmultipleversionsandplatformsgoingforward

DeployswithMinimumFootprintandMaximumManageability

ApplicationMemberCluster

DatabaseMemberCluster

OracleDomainServicesCluster

ORACLECLUSTERDOMAIN

Management Repository ServiceTrace File Analyzer ReceiverORAchk Collection ServiceGrid Names ServiceStorage ServicesRapid Home Provisioning Service

OracleDomainServicesCluster

OracleClusterDomain

IOServiceACFSServices

ASMService

UsesASMService

UsesIO&ASMServiceofDSC

MgmtRepository(GIMR)Service

GIonly

UseslocalASM

SharedASM

AdditionalOptionalServices

RapidHomeProvisioning

(RHP)Service

PrivateNetwork

rac troubleshooting and diagnosability sangam2016

Documents

ca workload automation de troubleshooting guide ca workload...

advanced rac troubleshooting

1 juliandyke.com © 2008 julian dyke rac troubleshooting web...

reinventing upgrades, platform changes, rac and more with...

a genetic algorithm approach for diagnosability analysis ·...

diagnosability versus the cloud, redwood shores 2011-08-30

120r-rac-01 and 120r-rac-02

diagnosability versus the cloud

diagnosability ii technometrics 2003.doc

first failure fault diagnosability - nocoug - northern...

rac best practices - rac sig 9dec05[1]

plan navegación aérea para colombia · rac 5 reglamento...

an approach for diagnosability analysis and sensor...

rac cc - cc rac - rac c - c rac - rac cc honda...

rac agenda – april 2020 electronic meetings only · 2020....

rac agenda – september 2018 - wildlife.utah.gov · rac...

configuring 11g r2 rac to rac dataguard

improving software diagnosability via log enhancement

diagnosability analysis and fdi system design for

riyaj advanced rac troubleshooting rmoug 2010 doc