dba 101 : calling all new database administrators (wp)

COLLABORATE 14 – IOUG Forum Database

1 | P a g e “DBA 101 : Ca l l i ng a l l New Database Admin i s t r a to r s” Wh i t e Pape r

DBA 101: Calling all New Database Administrators

Gustavo René Antúnez, The Pythian Group

ABSTRACT

TARGET AUDIENCE

This paper will be beneficial for anyone who is beginning to use Oracle’s Database Technology and for those who are already using it, but want to understand the several new features of Oracle 12c.

EXECUTIVE SUMMARY

BACKGROUND Most of us who are Database Administrators, or at least the ones I know personally, did not initially start as one; we were either programmers, sysadmins or even functional users who had some knowledge of how the RDBMS worked or maybe even none but we thought we did. We were thrown into this world because the previous DBA left and the company didn’t want to hire a new DBA or because you had proven yourself enough in the company to be given this trusted task. The real problem begins if you were ever in this situation, is that you discover that you really had very little knowledge and you are asked to recover the database to the day before yesterday and you have to provide a solution in a very tight time frame. Here we will discuss the basics of how the Oracle 12c Database is comprised of, and how you can take this knowledge and grow from here to become a successful DBA.

Today, many businesses around the world are using an Oracle product and in many of these, at the core there is an Oracle Database. Many of us who started as a Database administrator where put in this position because we were good PL/SQL programmers or good Sysadmins, but knew very little of what it took to be a DBA. In this session you will learn the core architecture of an Oracle Database in 12c as well as what it takes to administer and apply this new knowledge the day you go back to your office.

Learner will be able to: • Obtain the knowledge of the core architecture of a 12c Oracle Database.

• Learn what are the common causes of a performance problem and how to tackle it. • What is a High Availability environment and what it takes to administer it.

• Where is the DBA role moving towards to.



WHAT IS AN ORACLE DATABASE SERVER ? To be able to understand what are we to manage, first we need to define what is an Oracle Database Server. Making a

strict definition of how it is comprised, we need to define the following terms:

• Database. - It is a set of files that store data and metadata; these files are saved on a physical disk. • Database Instance or Instance. – It is a set of memory structures and a set of background processes that

manage database files.

Each of those is independent, meaning that you can have one without the other. But for you to be able to work with the database, you need to have an active database, meaning that you need to have at least one instance associated to one database. To better understand what a database is, it has to be understood that there is physical and logical viewpoint, and each have their own structures.

• Physical o Control Files. – Binary file that is normally multiplexed due to the importance of its contents. It contains the

metadata which specifies the physical organization of your database, as well it holds the following information:

§ Database Name § Checkpoint information, which it indicates the System Change Number in the redo stream where

recovery needs to begin. Every committed change before a SCN checkpoint is guaranteed to be written to disk.

§ Current Online Redo Log Sequence § RMAN Backup Information

o Online Redo Logs. – Set of files that record altered data that has not been written to the datafiles. Oracle requires that there be a minimum of two. The online redo log for a database instance is called Redo Thread, and each Database Instance will have it’s own Redo thread. Only one multiplexed online redo log file is used at a time per instance. A redo log is made up of redo records and these records include the following metadata:

§ SCN and time stamp of the change § Transaction ID of the transaction that generated the change § SCN and time stamp when the transaction committed (if it committed) § Type of operation that made the change § Name and type of the modified data segment

o Data files. – File that contains system and user data, the space used in these files are broken down into the following:

§ Data File Header. –Contains metadata, as SCN checkpoint and file size, as well as a unique file identifier that relates it to the database, a tablespace identifier that relates it to a tablespace and a container id, which relates it to the correct container.

§ Used § Free (Formatted,Never Used) § Free (Previously Used,Currently Unused)



• Logical o Data block. – It is the smallest logical unit; data is stored in data blocks. One data block corresponds to a

specific number of bytes. It is structured in the following way: § Block header. - General information about the block, including disk address and segment type § Table directory. - Metadata about tables whose rows are stored in this block § Row directory. - Describes the location of rows in the data portion of the block. § Row Data. –Contains user data/system data. Data is stored in rows and is formatted in the following

way: • Row Header

• Column Data The Database uses a rowid to uniquely identify a row, and the rowid has a four-piece format that is the following:

• OOOOOO. - Data Object number • FFF. - Tablespace-Identifies the data file that contains the row

• BBBBB. - Block number that contains the row • RRR. – Identifies the row in the block

o Extent. – A single allocation of a number of logically bordering data blocks. An extent can exist in only one data file.

o Segment. – A set of extents allocated for user, undo or temporary data. The first data block of every segment contains a directory of the extents in the segment.

o Tablespace. –This is the container of segments, defined as a storage unit that groups related logical structures. One tablespace needs at least one datafile. A database must have the SYSTEM and SYSAUX tablespaces. Below you will find the tablespace types:

§ The SYSTEM tablespace has the following: • Data dictionary

• Compiled stored objects § The SYSAUX tablespace is an auxiliary to the SYSTEM tablespace and is the default tablespace for

many Oracle Database features. § Permanent Tablespaces are recommended to store application data. § The UNDO tablespace is a locally managed tablespace reserved for system-managed undo

tablespaces. Undo data is the copy of the original, premodified , data and is retained until at least the transaction is ended.

§ The temporary contains data that persists only for the duration of a session; it is used for multiple sort operations that can’t be fitted into memory.

WHAT IS AN ORACLE DATABASE INSTANCE ? An Oracle Database Instance is a set of memory structures and processes that manage database files. Every active database has at least one Database Instance associated to it. When an Instance is started, it allocates the System Global Area and starts one or more background processes. An Instance is identified by a SID, which is a unique system identifier on a host.



To continue defining a Database Instance, first we have to define the basic memory structures that compose it:

• UGA (User Global Area). - Memory allocated for session variables, it must be available to a database session during the lifetime of the session.

• PGA (Program Global Area). – Memory heap that contains session-dependent variables required by a server process. It is subdivided into different areas

o Private SQL Area. - Holds information about a parsed SQL statement and other session-specific information for processing

o SQL Work Area. - Private allocation of PGA memory used for memory-intensive operations.

• SGA (System Global Area). - A group of shared memory structures that contain data and control information for one database instance. The most important components of the SGA are the following:

o Database Buffer Cache. – Memory area that stores copies of data blocks read from data files. It is divided into different pools:

§ Default pool. - Location where blocks are normally cached. § Keep pool. - Intended for blocks that are normally cached. § Recycle pool. - Intended for blocks that are infrequently used. § Non Standard Block pools. – These are pools for tablespaces that contain nonstandard

block sizes. o Redo Log Buffer. – It is a circular buffer that caches redo entries describing changes made to the

database until it can be written to the online redo log files. Redo entries contain the information necessary to reconstruct, or redo, changes made to the database by DML or DDL operations.

o Shared pool. – Portion of the SGA that provides critical services for sharing objects among large number of users, it is used to support the execution of shared SQL and PL/SQL packages. This pool is divided into several areas:

§ Library cache. – It stores executable SQL and PL/SQL code. • Shared SQL Area. – Contains the hash value and execution plan for a SQL

statement. § Data Dictionary Cache or Row Cache. – Holds information about database tables and views.

The data dictionary cache is also known as the row cache because it holds data as rows instead of buffers, which hold entire data blocks.

§ Reserved Pool. - Memory area that Oracle Database can use to allocate large contiguous chunks of memory.

It is imperative to understand that we have only briefly described the basic memory structures, but the Oracle Database Instance has more memory structures than the ones mentioned.

Now we need to define the mandatory processes that are needed for a Database to become an Active Database. You

will find these mandatory background processes in every database configuration:

• Process monitor (PMON). - Monitors other background process and performs process recovery. Responsible for cleaning up the buffer cache.

• Listener Registration Process (LREG). – Notifies the listeners about instances, services, handlers and endpoints. In releases before 12c, PMON performed these tasks.

• System Monitor (SMON). – Process in charge of several system-level cleanup tasks o Perform Instance Recovery at instance startup (If Necessary) o Maintains the undo tablespace o Cleans up temporary segments

• Database Writer (DBW). – Writes the contents of the modified database buffers in the buffer cache to the datafiles. There can be up to 100 database writers.



• Log Writer (LGWR). – Responsible for writing the redo entries from one or more redo log buffer strands, which speeds up the performance of each redo write requested. The LGWR writes all redo entries since the last write in all of these situations:

o A user commits o An Online Redo Log switch o 3 seconds since the last time it wrote o Redo Log Buffer is 1/3 full o DBW must write buffers to disk; one thing to keep in mind is that the LGWR must write the redo

records before the DBW writes the dirty buffers to disk.

• Checkpoint Process (CKPT). – It updates the control file and data file headers with the checkpoint information that includes checkpoint position and SCN and signals the DBW to write blocks to disk.

• Manageability Monitor (MMON and MMNL). – Performs tasks related to the Automatic Workload Repository.

• Recovery Process (RECO). – In a distributed database, it resolves failures in a distributed transaction.

WHAT IS THE MULTITENANT ARCHITECTURE?

Oracle Database 12c introduced what it calls the multitenant architecture, and what this licensable feature brings, is that it allows you to have one container database (CDB) that is an Oracle database that includes zero, one, or many customer-created Containers or better called Pluggable Databases.

The CDB has:

• One ROOT container (CDB$ROOT) containing SYSTEM, SYSAUX, UNDO, and TEMP tablespaces, Controlfiles and Redologs.

• The CDB has one ROOT and only one ROOT container

• One SEED container (PDB$SEED) containing SYSTEM, SYSAUX, TEMP, EXAMPLE tablespaces, used as a template to create new PDBs

A pluggable Database (PDB) is a user-created container holding the data and code for your specific applications. A

PDB is uniquely named within the CDB and:

• Has SYSTEM, SYSAUX, and TEMP tablespaces.

• Contains any number of other user created tablespaces. • Writes to the container UNDO tablespace, controlfiles and redologs. • Undo and redo is annotated with details of the PDB that they belong to.

• Each PDB have a separate namespace, allowing you to have a similarly named schema between PDB, also this allows you to have a uniquely named object in each PDB, like database directories.



In this multitenant environment, the data dictionary in each PDB in a CDB is independent, but for a CDB the data dictionary is split between the PDB and the CDB, allowing a reduction of duplication. In a CDB, there now exists a CDB_view for every DBA_view.

For us to be able to identify where we are in the multitenant environment, a container ID was introduced.

Container ID Rows Belong to

0 Whole CDB

1 CDB$ROOT

2 PDB$SEED

All other IDs User created PDBs

You can view this when connected to SQL*plus with the SHOW CON_ID and SHOW CON_NAME command

SQL> COLUMN NAME FORMAT A8 SQL> SELECT NAME, CON_ID, DBID, CON_UID, GUID FROM GV$CONTAINERS ORDER BY CON_ID; NAME CON_ID DBID CON_UID GUID -------- ---------- ---------- ---------- -------------------------------- CDB$ROOT 1 585028037 1 DD7C48AA5A4504A2E04325AAE80A403C PDB$SEED 2 4077104999 4077104999 EDDDB886A1191F07E043344EB2C0BE27 PDB1 3 3342226823 3342226823 EDDDC5E35CF7216DE043344EB2C0AB6F SQL> show con_id CON_ID ------------------------------ 1 SQL> show con_name CON_NAME ------------------------------ CDB$ROOT

CDB_view : All of the objects in the CDB across all PDBs.

DBA_view: All of the objects in a CDB or PDB

All_view Objects accessible by the current user

User_view : Objects owned by the current user



We have also a new type of users, called Common and Local users

• Common User. – User that has the same identity within the CDB and the PDB. It can connect and perform operations within the root and every existing and future PDBs.

• Local User. – Specific user that can only operate within the PDB. A restriction is that a user cannot begin with c## or C##

BASIC UNDERSTANDING OF INDEXES Oracle has 2 ways of accessing data; one is doing a Full Table Scan, which this reads every row in the table, or access one row at a time via the ROWID. Index-based reads are single block reads, so what you will try to do when using an index is to reduce the number of single block reads. If you insert into or delete from a table, Oracle has to insert or delete rows in the index. If you update an indexed column, Oracle has to maintain the indexes that are on the columns being updated, so if you have a lot of indexes, it may speed up select statements, but it will slow down inserts, updates, and deletes.

AUTOMATIC WORKLOAD REPOSITORY The Automatic Workload Repository is a licensable option within the Oracle Database, it periodically makes a snapshot of essential statistics and more important of workload information. These statistics is the basis for detecting most of the problems and self-tuning means that the Oracle Database Server provides. It is controlled by the parameter STATISTICS_LEVEL and CONTROL_MANAGEMENT_PACK_ACCESS. The repository has the following components:

• SGA In-Memory AWR statistics. - Some are Object statistics, some statistics collected in V$SYSSTAT and V$SESSTAT views,Time Model statistics based on usage or High load SQL statements.

• AWR snapshots. - These are In-Memory statistics that are flushed by the MMON to the SYSAUX tablespace by default every hour. These are retained by the snapshot retention policy that is 10,080 minutes (7 days).

But now that you have this repository, the only way for it to be useful to you, is if you can view, exploit its data and produce reports out of it. One-way to do this is to use the Oracle Produce SQL Scripts, which can be located in the $ORACLE_HOME/rdbms/admin directory. Some of the types of reports that you can produce are the following:

• Report for the local Database (awrrpt.sql) • Report for a specific Database (awrrpti.sql)

• RAC Report for the local Database (awrgrpt.sql) • RAC report for a specific Database (awrgrpti.sql)

• Report for SQL Statement for the local Database (awrsqrpt.sql) • Report for SQL Statement for a specific Database (awrsqrpi.sql)

All of these reports use the package DBMS_WORKLOAD_REPOSITORY, this package allows you to manage the repository, like create baselines, manage the retention period and even create reports.

As stated before, AWR provides a time based model, so to start looking for performance problems, focusing in the

AWR reporting the “Top 5 wait events” for a system would be the first place to start looking, and some of the most common performance problems are in the following wait events but not constrained to as there would be to many to focus in this paper:

• DB File Sequential Read. - Wait commonly related to a single-block read (Index Read for example), so a good way to look is for unselective index scans

• DB File Scattered Read. – Wait commonly related to Full Table Scans or Fast Full Index Scans, so a good place to start would be to see if partitioning can be used or see if there is a way to have a far more selectivity of your data.



• Buffer Busy Wait. – Wait happens when a session wants to access a block in the buffer cache but it can’t because it is busy. Depending on your block type (Data,Segment Header,Undo Header,Index Block) your course of action will vary.

• Log File Sync. – Commit requests are waiting for a log file switch, start by checking if redologs are large enough.

• Direct Path Read. – These are generally used by Oracle when reading directly into PGA memory (as opposed to into the buffer cache). Start by ensuring that no disks are I/O bound and verify that OS asynchronous I/O is configured correctly.

Also there are events that are considered Idle as the server process is waiting because it has no work. This normally implies that the bottleneck is not for database resources, a few of these are below:

• pmon timer

• SQL*Net message from client • rdbms ipc message

• PX Idle Wait • pipe get

• dispatcher timer • PX Idle Wait

As part of the Diagnostics and Tuning Pack, comes the Automatic Database Monitor (ADDM, commonly known as

ADAM) an advisor that uses the statistics captured in the AWR, it proactively diagnoses performance issues and as well provides recommendations based on an all-inclusive approach on the time-wait model to fix these issues and offers the rationale as to why it’s providing the recommendations. In the 12c AWR report, at the end you will find an ADDM report as well.

Another integral part of this pack is the Active Session History (ASH), this samples the active sessions every second

and stores the data in a circular buffer within the SGA and is normally used for shorted lived problems that can’t be diagnosed with AWR reports or ADDM. You can view ASH information using the V$ACTIVE_SESSION_HISTORY view. In Oracle 12c, the ASH samples are stored in the root and a common user that the current container is the PDB, can only view the PDB information, if the current container is root, it can view ASH data for the root and for the PDBs. We can also generate reports for ASH from scripts that are located in $ORACLE_HOME/rdbms/admin

• Report on a specific database instance (ashrpt.sql)

• Report for Oracle RAC or a specific Database (ashrpti.sql)

ORACLE DATABASE AND HIGH AVAILABILITY Availability is percentage of uptime in a year that a designed system in which it guarantees that it will be operational during a pre-established service level of agreement. What is considered a Highly Available system, depends on the SLA established, but a highly desired but difficult to achieved standard is the “Five 9’s” (99.999%) which is 5.26 minutes.

Availability % Downtime per year Downtime per month Downtime per week

90% ("one nine") 36.5 days 72 hours 16.8 hours

95% 18.25 days 36 hours 8.4 hours

97% 10.96 days 21.6 hours 5.04 hours

98% 7.30 days 14.4 hours 3.36 hours

99% ("two nines") 3.65 days 7.20 hours 1.68 hours



99.5% 1.83 days 3.60 hours 50.4 minutes

99.8% 17.52 hours 86.23 minutes 20.16 minutes

99.9% ("three nines") 8.76 hours 43.8 minutes 10.1 minutes

99.95% 4.38 hours 21.56 minutes 5.04 minutes

99.99% ("four nines") 52.56 minutes 4.32 minutes 1.01 minutes

99.999% ("five nines") 5.26 minutes 25.9 seconds 6.05 seconds

99.9999% ("six nines") 31.5 seconds 2.59 seconds 0.605 seconds

99.99999% ("seven nines") 3.15 seconds 0.259 seconds 0.0605 seconds

According to Steve Mills “At a high level, the essence of the HA (High Availability) requirement is captured in the

following flow fault management cycle below. Given a healthy system, we can follow the flow as follows:

• Fault detection – a fault is declared as some fault criteria (ex. health-check or heartbeat rules) has been detected • Fault isolation – determine the scope of the fault given the nature or place of the fault. Is it just a single process, a

collection of processes, or an entire node?

• Fault recovery – engage an automated fault recovery policy that often must be completed in a sub-second time frame

• Fault repair – if possible, automatically bring faulted resource back to health and re-integrate into the system to again provide protection

• Health – optimal system functionality and protection in place”

A highly available architecture should have the following characteristics:

• Tolerate Failures • Prevent data loss

• Achieve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) • Detect errors in a timely manner

Oracle provides several solutions to achieve your availability service level agreements. As you will start to deploy your

Maximum Availability Architecture, you need to evolve it around your RTO and your RPOs, while keeping in mind your Cost of Downtime and your Return of Investment (ROI). Below you will find several options that comprise Oracle Maximum Availability Architecture:

• Oracle Real Application Cluster. - This is an architecture that has a one Database to many Instances relationship; a RAC database can have up to 100 instances. All instances must use the same private network using a switch (or multiple switches) that only the nodes in the cluster can access. This private network is called interconnect network. Oracle RAC requires Oracle Clusterware for the infrastructure to bind multiple servers. Oracle Clusterware along with Oracle Automatic Storage Management (ASM) compose what is called Oracle Grid Infrastructure. All datafiles,control files and redo logs must be in a cluster-aware disks, which can be any of the following options:

o Oracle ASM o A certified cluster system

§ Oracle ACFS § 3rd Party Vendor that is certified for RAC

o Certified network file system (NFS)

• Oracle Data Guard. – Set of services where you can create, manage and monitor one or up to 30 geographically dispersed standby databases. Data Guard only propagates the redo data in the logs, so no disk corruptions are introduced in the standby. There are 3 types of Standby Databases

o Physical Standby. - Physically identical copy of the primary database on a block-by-block basis, that is synchronized through redo apply.



o Logical Standby. –Contains the same logical information as the primary, and is synchronized through SQL Apply

o Snapshot Standby. –It is an updateable standby that receives archived redolog logs, but does not apply them, allowing it to be opened in read-write mode.

§ It cannot be a target for a switchover/failover; it has to be first converted into physical standby. Data Guard has 3 protection modes:

o Maximum Availability. – Provides the highest level of data protection without compromising the availability of the primary. Transactions don’t commit until all redo has been received in memory or written to the standby redo log.

o Maximum Performance. –This is the default protection method. It allows transaction to commit as all redo generated by those transactions has been written to the online redo log. Redo data is written asynchronously to the standby.

o Maximum Protection. – Redo must be written to the standby to both online redo log and standby redo log before the transaction commits. To ensure that there is no data loss, the primary database will shutdown if it can’t write to any of its standby databases.

• Oracle Golden Gate. – Logical log based change data capture (CDC) and replication that enables the exchange and manipulation at the transaction level amongst heterogeneous platforms. It allows for both source and target databases to be in read/write mode. Replication may be uni-directional, bi-directional or multi-dimensional. The following considerations should be made:

o Database in archive log mode o Supplemental logging at the source o Sufficient disk space to store trail files o Use a Data Pump when the target connection is via TCP/IP. o If using RAC, Golden Gate should reside on SAN shared by the RAC servers.

Oracle Golden Gate has the following components:

o Extract. – It runs on the source system and it is the extraction method for Golden Gate. o Data pump. – Is a secondary extract group, it adds storage flexibility and serves to isolate the primary extract

from TCP/IP activity. It can perform data filtering, mapping and conversion. o Replicat. –It runs on the target system, it reads the trails and reconstructs the DML or DDL operations and

applies them. o Trails or extract files. – Series of files that store records of captured changes, which support the extraction

and replication that are stored on disk. o Checkpoints. – Store the current read and write positions of a process, prevent redundant processing and loss

of data. o Manager. – Control process, which should be running on each system that is configured with Golden Gate

and needs to be running before Extract or Replicat is running. o Collector. – Process that runs in the background on the target system, when continuous, online change is

active.

Oracle’s generally recommended Maximum Availability Architecture (MAA) is to use Oracle Golden Gate for any advanced or granular replication requirements, combined with Oracle RAC and Data Guard. Taking in mind that if you only have a RAC system is normally not considered as a highly available system, as an example, if your SAN network goes down all of your instances become unusable.



WHERE IS THE DBA JOB GOING? According to Forbes, Database as a Service (DBaaS) annual revenues will grow from 150 million in 2012 to 1.8 billion by 2016. This a trending technology where in my point of view the Oracle DBA job is moving towards to, being a Cloud administrator. You are actually seeing in some shops where Exadata Machine exists that the Database Administrator is already moving towards being a Data Machine Administrator, making this just a pebble stone moving this job towards being a cloud administrator.

You can see why IT shops around will adopt this trend, as end users can request a database service, use it for the needed lifecycle and return the resources as they are no longer needed. This technology will allow a faster deployment and lower costs in the process. You can see the ease of this move with Oracle’s 12c Database multitenant option, where you can easily setup a seed DB for the HR department, another Seed for the Customer application, and use EM12c to allow the end user to easily clone and plug and unplug their databases without the need of a Database administrator having to duplicate the source database as it used to be done in previous versions.

REFERENCES Why Database As A Service (DBaaS) Will Be The Breakaway Technology of 2014 http://www.forbes.com/sites/oracle/2013/12/05/why-database-as-a-service-dbaas-will-be-the-breakaway-technology-of-2014/ Delivering Database as a Service (DBaaS) using Oracle Enterprise Manager 12c http://www.oracle.com/technetwork/oem/cloud-mgmt/dbaas-overview-wp-1915651.pdf Deploying Oracle Maximum Availability Architecture with Exadata Database Machine http://www.oracle.com/technetwork/database/features/availability/exadata-maa-131903.pdf Oracle® Database High Availability Overview 12c Release 1 (12.1) http://docs.oracle.com/cd/E16655_01/server.121/e17601/toc.htm

Oracle® Database Administrator's Guide 12c Release 1 (12.1) http://docs.oracle.com/cd/E16655_01/server.121/e17636/toc.htm Oracle® Database Performance Tuning Guide 12c Release 1 (12.1) http://docs.oracle.com/cd/E16655_01/server.121/e15857/pfgrf_instance_tune.htm#TGDBA024