Oracle 11 g R1/R2 Real

Application Clusters Essentials

Design, implement, and support complex Oracle 11 g

RAC environments for real-world deployments

Ben Prusinski

Syed Jaffer Hussain

PUBLISHING

BIRMINGHAM - MUMBAI

] so88

Table of Contents

Preface 1

Chapter 1: High Availability.

J_

High availability concepts 7

Planned versus unplanned downtime 8

Service Level Agreements for high availability 8

High availability interpretations 9

Recovery time and high availability 10

System design for high availability 11

Business Continuity and high availability 11

Disaster Recovery 12

Business Continuity and Disaster Recovery guidelines 13

Fault-tolerant systems and high availability 14

Requirements for implementing fault tolerance 15

Fault tolerance and replication 16

High availability solutions for Oracle 17

Oracle Data Guard 17

Oracle Streams 17

Oracle Application Server Clustering 18

High availability: Oracle 11g R1 Real Application Clusters (RAC) 19

High availability: Oracle 11g R2 Real Application Clusters (RAC) 19

Summary 20

Chapter 2: Oracle 11g RAC Architecture 21

Oracle 11g RAC architecture 22

Certification matrix for Oracle 11g RAC architecture 23

Hardware architecture for Oracle 11 g RAC 25

Server configurations for Oracle 11 g RAC 26

CPU processors 26

Choosing between 32-bit and 64-bit CPU architectures 27

Dual core and multicore processors 29

Table of Contents

Network architecture for Oracle 11 g RAC 30

The private network and the Oracle 11 g RAC interconnect 30Choices for private interconnect and 11 g RAC 31

Redundancy for Ethernet interconnects with 11g RAC 33

Network bonding (NIC teaming) 39

Storage architecture for Oracle 11 g RAC 39

RAID configurations for Oracle 11g RAC 40

RAID 0 (striping) 42

RAID 1 (mirroring) 42

RAID 5 (striped with parity) 42

RAID 10 (striped mirrors) 43

Third-party RAID implementations 44IBM AIX LPAR disk volume management for RAID 44

Linux volume management for RAID configuration 45

Storage protocols for RAC 45

SCSI 46

Fibre Channel 47

Point-to-Point (FC-P2P) 47

Fibre Channel Arbitrated Loop (FC-AL) 47

Fibre Channel Switched Fabric (FC-SW) 48

Which Fibre Channel topology is best? 48

iSCSI 50

Which storage protocol is best for RAC? 50Asynchronous I/O versus Direct I/O for Oracle 11 g RAC 51

Oracle 11g RAC components 53

Voting Disk 53

Oracle Cluster Registry 53

Oracle 11g R1 RAC background processes 54ACMS Atomic Controlfile to Memory Service 54

GTXO-j Global Transaction Process 54LMON Global Enqueue Service Monitor 54LMD Global Enqueue Service Daemon 54

LMS Global Cache Service Process 54

LCK0 Instance Enqueue Process 55

RMSn Oracle RAC Management Processes 55RSMN Remote Slave Monitor 55

Oracle 11g R2 RAC background processes 56Grid Plug and Play 56

Grid Interprocess Communication 56

Multicast Domain Name Service 56

Oracle Grid Naming Service 56

How RAC differs from Oracle 11 g single-instance implementations 57

New ASM features and RAC 58

New SYSASM privilege for Oracle 11 g R1 ASM 58

Oracle 11 g R2 ASM features 59

Table ofContents

OCR and Voting Disk stored in ASM 59Oracle Automatic Storage Management Cluster Filesystem (Oracle ACFS) 59

New Oracle 11 g ASM Disk Group compatibility features 60

Summary 61

Chapter 3: Clusterware Installation 63

Preparing for a cluster installation 63

Server (node) requirements 64Network requirements 64Kernel parameters 65

Operating system packages 66OS groups and users 68OS user settings 68

Configuring Secure Shell (SSH) 69

Verifying prerequisites with the CLUVFY utility 71

Oracle 11 g R1 Clusterware installation 72

Initiating Oracle Universal Installer for Oracle 11 g R1 Clusterware 73What happens when orainstRoot.sh and root.sh is run? 82

Oracle 11 g R1 Clusterware post-installation checks 86

Installing Oracle 11g R1 RAC software 88

Initiating Oracle Universal Installer for Oracle 11 g R1 RAC software 88

Post-installation tasks 90Oracle 11 g R2 Clusterware installation 91

Initiating Oracle Universal Installer for 11 g R2 Clusterware 92What happens when the root.sh is run? 99Oracle 11 g R2 Clusterware post-installation checks 100

Installing Oracle 11g R2 RAC software 102

Initiating Oracle Universal Installer for Oracle 11 g R2 RAC software 103

Post-installation tasks 108

Oracle 11 g R2 Clusterware new features highlights 109

Removing/Reconfiguring a Grid Infrastructure configuration 110

Removing a successful Grid Infrastructure configuration 110

Reconfiguring a failed Grid Infrastructure configuration 111

Summary 112

Chapter 4: Automatic Storage Management 113

Overview of Automatic Storage Management (ASM) 114

Filesystem versus ASM storage architecture 115ASM disk 116ASM disk group 117

ASM instance configuration and management 118ASM initialization parameters 119

Creating an ASM instance 120Initializing DBCA 120

ASM background processes 122

Tabic of Contents

ASM dynamic views 122

V$ASM_DISK 123

V$ASM_DISKGROUP 124

V$ASM_OPERATION 124

V$ASM_DISK_STAT 124

V$ASM_DISKGROUP_STAT 124

V$ASM_CLIENT 125

ASM instance startup/shutdown 125

ASM disk group administration 126

Creating a disk group 127

Altering a disk group 128

Dropping a disk group 129

Overview of ASMCMD 130

ASMCMD in action 131

ASM 11 g R1 new features 133

ASM fast mirror resync 133

ASM preferred mirror read 134

ASM fast rebalance 135

ASM disk group compatibility attributes 135

ASM performance enhancements 136

New SYSASM role 136

ASM 11 g R2 new features 137

Automatic Storage Management Configuration Assistant (ASMCA) 137

Initiating ASMCA 137

Configuring a new ASM instance 138

Managing an ASM instance 140

Managing an ASM disk group 142

Creating an ASM disk group 143

Creating an ASM disk group in silent mode 144

Automatic Storage Management Dynamic Volume Manager (ADVM) 145

ADVM volume trivia 145

Creating ASM volumes 146

Automatic Storage Management Cluster File System 152

(ACFS) 152

Using ACFS as Oracle database home 153

Oracle ACFS drivers 153

Prerequisites for creating ACFS 154

ACFS creation methods 154

ACFS mount registry 161

Managing ACFS 161

ASM new background processes to support ACFS 163

Querying V$ASM views to obtain information about ACFS 164

ACFS snapshots 164

Creating a snapshot 166

Removing a snapshot 168

[iv]

Table of Contents

ASM Intelligent Data Placement (IDP) 169

Managing IDP settings 169

Finding IDP setting information 169

ASMCMD enhancements 170

ASM backup strategies 171

mdjoackup and md_restore commands 171

Summary 173

Chapter 5: Managing and Troubleshooting Oracle 11 g

Clusterware 175

Oracle 11 g RAC Clusterware administration 175

About Oracle Clusterware 176

Oracle 11 g Clusterware concepts 176

Oracle Cluster Registry 177

Voting disk 177

Initialization and shutdown scripts for Clusterware 179

Oracle 11 g Clusterware background processes 180

Cluster Ready Services Daemon 180

Additional background processes for Oracle 11g Clusterware 181

Fatal Clusterware processes and Oracle 11g RAC 184

Managing Oracle 11g Clusterware utilities 185

CRSCTL 185

CRS_STAT 188

OCRCHECK 190

OCRCONFIG 192

CLSCFG 193

CLUVFY 196

Troubleshooting Oracle 11g Clusterware 197

Failed, missing, or offline 11 g Clusterware resources 198

Offline Clusterware resources for Oracle 11 g RAC 199

Problems with the Voting disk and OCR 200

Vote disk issues with 11 g Clusterware resources 200

Failed or corrupted Vote Disks 201

Failed or corrupted OCR 202

How to recover the OCR from backup 202

Steps to perform recovery of lost and/or corrupted OCR 203

Check status 11 g RAC Clusterware 204

Root cause analysis 11 g RAC 205

Oracle 11 g RAC node reboot issues 206

Oracle 11 g RAC Clusterware processes—node reboot issues 207

Root cause analysis for solving node reboots with 11 g RAC 208

OCSSD Reboots and 11 g RAC 210

OPROCD failure and node reboots 211

OCLSOMON-RAC node reboot 211

[v]

Table of Contents

Hardware, storage, and network problems with RAC 212

Hardware, storage, and network resolutions 215

New features in Oracle 11 g R2 Clusterware 215

Oracle Real Application Clusters one node (Oracle RAC one node) 216

Improved Oracle Clusterware resource modeling 216

Policy-based cluster and capacity management 217

Cluster time synchronization service 217

Oracle Cluster Registry and voting disks within Oracle ASM 217

New features for upgrading to Oracle 11 g Clusterware 218

Oracle 11 g R2 Cluster Verification Utility new features 218

Zero downtime patching for Oracle Clusterware 218

Summary 219

Chapter 6: RAC Database Administration and Workload

Management 221

RAC database configuration and creation 222

Creating a database using DBCA 222

Choosing database storage options 231

We have created a database using DBCA—now what? 241

What's new in Oracle 11g R1 and R2 databases? 243

Automatic Memory Management 244

New AMM dynamic performance V$ views 245

Tuning AMM 246

Database Smart Flash Cache 248

Configuring Smart Flash Cache 248

Instance caging 249

New background processes in Oracle 11 g 249

Finding the alert.log file location in Oracle 11g 250

Automatic Diagnostic Repository 251

V$DIAG_INFO view 252

RAC database administration 253

Using the Server Control Utility 253

Automatic Workload Management 255

Overview of services 255

Creating and managing services 256

What's new in Oracle 11 g services' behavior? 262

Scalability (Load Balancing) 263Client Side Connect Time Load Balance 263

Server Side Listener Connection Load Balance 264

Transparent Application Failover 265

Configuring Transparent Application Failover 266

Fast Connection Failover 267

Configuring Fast Connection Failover 267

Summary 268

Table of Contents

Chapter 7: Backup and Recovery 269

An overview of backup and recovery 270

An overview of Recovery Manager (RMAN) 270

RMAN architecture 272

RMAN performance tuning tips 273

Backup types and methods 274

Logical backup 274

Physical backup 275

ONLINE RMAN backups 275

OFFLINE RMAN backups 275

RMAN new features in 11g R1 and 11g R2 275

Database Recovery Advisor 276

Multisection backups for very large datafiles 276

Undo tablespace backup optimization 277

Faster backup compression 277

Active database duplication 278

Archivelog deletion policy enhancements 280

Automatic Block Recovery (ABR) 280

Tablespace point-in-time recovery enhancements 281

RMAN best practices for RAC 281

Configuring the Flash Recovery Area for a RAC database 282

Instance recovery versus Crash recovery in RAC 283

Parallelism for backup and recovery in RAC 287

Backing up a RAC database with RMAN 287

Configuring multiple channels 289

OCR and Voting disk backup and recovery strategies 290

Adding a Mirror location for the OCR and Voting disk 291

OCR automatic backups 291

Performing OCR manual backups 293

Voting disk manual backups 294

Restoring OCR 294

Restoring the Voting disk 297

Summary 298

Chapter 8: Performance Tuning 299

Tuning differences: single instance versus RAC 300

Oracle 11 g single instance database 300

Oracle RAC 11g database 300

New Oracle 11g performance tuning features 300

Database Replay 301

SQL Performance Analyzer 302

Database Health Monitor 302

Table of Cantenh

PL/SQL Native CompilationServer Result Cache

Client Side Result Cache

SQL Tuning Advisor

New performance features in Oracle 11gR2In-Memory Parallel Execution

Analyzing the Cache Fusion impact on RAC performance

Cache Fusion

Latency statistics

RAC wait events

Monitoring RAC cluster interconnect performanceOracle cluster interconnects

Monitoring RAC wait events

Summary

Chapter 9: Oracle 11g Clusterware UpgradeOverview of an upgrade 323

Upgrade sequence 324

Upgrading Oracle 10g R2 Clusterware to Oracle 11g R1 324

Kernel parameter values 325

Packages required on Linux 5 326

Oracle 11 g R1 Clusterware upgrade steps 326

Performing preinstallation checks with cluvfy 327

Executing runlnstaller sh script 327

Post-upgrade steps for 11 g R1 Clusterware 332

Upgrading to Oracle 11 g R2 Clusterware 333

Overview of our environment 333Upgrading nodes 334

11g R2 upgrade changes and restrictions 334

Kernel parameter values 335

Packages required on Linux 5 335

Performing preinstallation checks with cluvfy 336

Oracle 11g R2 Clusterware upgrade steps 336Executing the runlnstaller.sh script 337Post-upgrade checks for 11 g R2 Clusterware 349

Post-upgrade steps for 11g R2 Clusterware 351

Downgrading Oracle Clusterware after an upgrade 352

Summary 353

Chapter 10: Real-world Scenarios 355

Adding a new node to an existing cluster 356

Performing prechecks with the cluvfy utility 357addNode sh 360

Adding a node in silent mode in Oracle 11 g R2 363

303

304

305

306

307

307

307

308

309

310

315

315

316

321

323

Table of Contents

Post-installation status checks for Clusterware 364

OCR file manual backup syntax 368

Voting Disk backup syntax 368

Installing ASM and RDBMS software using addNode.sh script 368

Cloning ASM software using addNode.sh script on Oracle 11 g R1 369

Post-node addition steps 370

Removing a node from the cluster 370

Adding an RAC database instance 373

Adding a new instance using DBCA 374

We have added an instance—what next? 379

Verifying new instance status 379

Using DBCA in silent mode to add an instance 380

Post-add instance steps 380

Deleting an RAC database instance 380

Using DBCA in silent mode to delete an instance 381

Converting a single-instance database to an RAC database 382

Overview of RCONFIG command-line tool 383

What you need to accomplish the conversion task 384

Sample of a modified ConvertToRAC.xml input file 384

How to test a conversion without actually performing the conversion 386

Converting a single-instance database to an RAC database 387

How to resume a failed rconfig operation 389

Checking log files 389

How to optimize rconfig to run faster 389

Post-conversion steps 390

Relocating an RAC database and instances across nodes 390

Relocating the instance 391

Adding the instance example 392

Workaround when a database and instance are configured on the

same node 392Adding the database example 393

Post-relocation steps 394

Summary 394

Chapter 11: Enabling RAC for EBS 395

EBS architecture 396

Oracle 11 g RAC suitability 399

Installing EBS 12.1.1 400

EBS implementation on Oracle 11g RAC 407

RAC-enabling EBS 12.1.1 410

Configuration prerequisites 410

ASM and RAC-enabling the EBS database with the rconfig utility 412

Running AutoConfig 418

Table of Contents

Copying AutoConfig to the new RDBMS ORACLE_HOME for

Oracle 11g R1 11.1.0.7 419

Generating your database context file 422

Preparing for AutoConfig by completing the following AutoConfig steps 422

Generating and applying AutoConfig configuration files 423

Executing AutoConfig on all database nodes in the cluster 424

Performing Init file, tnsnames, and listener file activities 424

Establishing applications environment for Oracle RAC 426

Setting up load balancing 427

Configuring Parallel Concurrent Processing 428

Prerequisites for setting up Parallel Concurrent Processing 428

Cloning EBS concepts in brief 429

Preparing the source system 430

Copying the source system 430

Configuring the target system 431

Adding a new node to an existing EBS system 431

Setting up Parallel Concurrent Processing 433

Setting up Transaction Managers 434

Setting up load balancing on concurrent processing nodes 435

Summary 435

Chapter 12: Maximum Availability 437

Oracle 11 g Streams for RAC 438

Oracle 11g Streams architecture for RAC 438

Capture 438

Staging 439

Propagation 439

Consumption 439

Default apply 440

User-defined function apply 440

Explicit de-queue 440

Understanding Oracle Streams rules 440

Transformations and Streams 440

Capture and Apply processes in an RAC instance 441

Streams in the RAC environment 441

New features in Oracle 11g Streams 442

Synchronous Capture 442

Splitting and merging of a Stream Destination 442

Tracking LCRs through a Stream 443

Streams Topology and Performance Advisor 443

Combined Capture and Apply 443

Best practices for Streams in an RAC environment 444

Additional configuration of RAC environments for a Source Database 444

Queue ownership 445

Propagation restart 446

Table of Contents

Changing the GLOBAL_NAME of the source database 447

Additional configuration for RAC environments for the Apply Database 448

Changing the GLOBAL_NAME of the Target Database 448

New features for Streams in Oracle 11 g R2 448

XStream 449

Statement DML Handlers 449

Ability to record table changes 449

SQL generation 449

Support for compressed tables 450

Support for SecureFile LOBs 450

Automatic splitting and merging 450

New Apply process parameter 450

Monitoring jobs 451

New 11 g R2 Oracle Streams view 451

Oracle 11 g Data Guard and RAC 451

New features for Oracle 11g Data Guard 451

Active Data Guard 452

Snapshot Standby 452

Configuring Data Guard Physical Standby for 11g RAC 452

Configuring Oracle RAC primary database to send redo data 453

Design considerations in an Oracle RAC environment 453

Format for archived redo log filenames 453

Troubleshooting Oracle 11 g Data Guard and RAC 454

Switchover fails in an Oracle 11 g RAC configuration 455

How to recover from corrupt datafile on standby 455

How to recover from a corrupt block on standby 457

Automatic repairing of corrupt data blocks 457

New features for Data Guard in Oracle 11 g R2 458

New Oracle Data Guard 11g R2 features for Redo Apply 458

New Oracle 11 g R2 Data Guard features for SQL Apply 459

Summary 460

Appendix: Additional Resources and Tools for the Oracle

RAC Professional 461

Sample configurations 461

Reviewing and resolving manual configuration changes 464

adchkcfg utility 464

Oracle RAC commands and tips 467

Cluster deconfig tool for Oracle RAC 468

Using the cluster deconfig tool 469

Limitations of the cluster deconfig tool 470

Problems and limitations of the cluster deconfig tool 470

Starting the cluster deconfig tool 471

Silent mode operations using cluster deconfig 471

Table ofContents

Manual cleanup for RAC 474

Repairing the RAC environment without reinstalling 476

Reinitializing OCR and Voting Disks without reinstalling RAC 476

Using ROOTDELETE.SH in debug mode 478

Using rootdeinstall.sh 480

Reinstalling CRS on the same cluster in another CRS_HOME 480

Stopping CRS processes 480

Reinstalling CRS on same cluster in another CRSJHOME 481

Oracle 11g R2 cluster removal tools for RAC 481

Tracing RAC issues with Oradebug 482Using Oradebug to trace Oracle 11g Clusterware 485

Server Control Utility 486

Oracle 11 g R2 SRVCTL commands 486

Managing Oracle Clusterware with the CRSCTL utility 487Differences between 11 g R1 and 11 g R2 syntax for CRSCTL 487

Operating system-level commands for tuning and diagnosis 492

Strace 492

Truss 495

GDB 496

Additional references and tips 499

Clusterware startup sequence for Oracle 11g R2 500

Index 503

[xii]