<Insert Picture Here>

Controlling resources in an Exadata environment

Agenda

• Exadata Security

• Flash Cache and Log

• Storage Indexes

• Parallel Execution

Agenda

• Exadata Security • Flash Cache and Log

• Storage Indexes

• Parallel Execution

EXADATA SECURITY

Exadata Security Model

• Exadata Storage Interconnect uses IP and InfiniBand

• Limit the storage connection to specific adapters

• Defined when creating the cell

• This is physical security, robust and easy to maintain

• If systems (DB nodes) are connected to InfiniBand

• See Exadata cells

• See / use (empty) grid disks unless Exadata Security is setup

• Per-database access-control is possible

• Open Security, ASM-Scoped, Database-Scoped Security

Security setup on Exadata

• Database nodes are customer responsibility

• No firewall setup

• Password limitations in place

• 90 days, case restrictions, sizes, 5 attempts etc

• Exadata storage cells are secured

• Password restrictions

• 90 days, case restrictions, sizes, 5 attempts etc

• Firewall setup

• Only SSH access through Mgmt and IB network

• OS activity monitored

• Automatic log-off of SSH sessions

Open Security (default)

• No access restrictions

• Useful for environments without security requirements.

• Default security mode after creating new storage cell

• No security setup required

• Works “out of the box”.

• No ACLs or access-limitations

• Any database can access all grid-disks

• Normal ASM “protection” is in place through headers

• Mistakes (force addition) could happen

ASM-Scoped Security

• Security per ASM cluster

• ASM cluster identified by unique key in local file on DB node

• Alias is defined in Exadata cell for each unique key

• Griddisks are enabled for specific alias(es)

• Only one key needs to be set up per ASM cluster

• Same key on all nodes in of the same ASM cluster

• Every Griddisk needs to be enabled

• Aliases in properties of Griddisk (field AvailableTo)

Database-Scoped Security

• Each database needs different key

• Each database, not each instance !

• Make sure all DB in a cluster have same key setup

• If no key has been setup, fallback to ASM security

• If wrong key has been setup, access is denied

• Access per GridDisk (availableTo) field

• Each cell gets alias for unique key

• Access by setting alias(es) in the availableTo field

• Multiple entries are allowed in the availableTo field

• ASM cluster used must have access too !

• Both Database and ASM must have access to GridDisk

Creating a Security Key

• A new key can be generated on any cell:

CellCLI> CREATE KEY

66e12adb996805358bf82258587f5050

• Use result on all cells en DB nodes

• Do not generate a key on each cell, one key is enough

• Do not generate a key for each instance, just each database

• User is responsible for copying the key to the right locations

Setting up ASM-Scoped Security

1. Start with Open Security, tested and working

2. Shutdown all databases and ASM

3. Create a new key on any cell

4. Copy key file cellkey.ora to all hosts in cluster

5. Copy key to all cells in the storage pool

6. Set up ACLs to allow ASM on all cells

7. Restart all databases and ASM

• All databases using this ASM can access Griddisks.

• No other user can access any of the Exadata storage.

ASM-Scoped Security How to setup ?

• On any cell • CellCLI> CREATE KEY

66e12adb996805358bf82258587f5050

• Lookup the name of the ASM cluster • SQL> SHOW PARAMETER db_unique_name

• Create cellkey.ora file on ASM node

• Located in /etc/oracle/cell/network-config/cellkey.ora

• Permissions 640

#cellkey.ora file for ASM

key=<unique key> (66e12adb996805358bf82258587f5050)

asm=<db_unique_name> (+asm)

realm=<any name, optional> (my_realm)

ASM-Scoped Security How to setup ? (continued..)

• On all of the the cells (not just one) • ASSIGN KEY FOR +asm='66e12adb996805358bf82258587f5050'

• Create new Grid disks with ACL by default • CREATE GRIDDISK ALL PREFIX=sales, availableTo='+asm'

• Or change ACL for existing cells • ALTER GRIDDISK ALL availableTo='+asm'

• ALTER GRIDDISK sales_CD_1_cell01, sales_CD_2_cell01,

sales_CD_3_cell01, sales_CD_4_cell01,

sales_CD_5_cell01, sales_CD_6_cell01 availableTo='+asm'

Setting Up Database-Scoped Security

1. Start with ASM-Scoped Security, tested and working.

2. Shutdown all databases and ASM

3. Create a key for each database (not each instance)

4. Copy key file cellkey.ora to each database home

5. Copy key to all cells in the storage pool

6. Set up ACLs to allow each database, as needed.

7. Restart all databases and ASM

Each database can access its own grid-disks.

If a key is defined, no other user can access any of the

Exadata storage.

Database-Scoped Security How to setup ?

• Create new key on any cell • CellCLI> CREATE KEY

51a826646ebe1f29e33c6ed7c4965c9a

• Create cellkey.ora file on Database node

• Located in

$ORACLE_HOME/admin/<db_name>/pfile//cellkey.ora

• Permissions 640

#cellkey.ora file for Database key=<unique key> (51a826646ebe1f29e33c6ed7c4965c9a)

asm=<db_unique_name_for_the_ASM> (+asm)

realm=<any name, optional> (my_realm)

Database-Scoped Security How to setup ? (continued..)

• On all of the cells (not just one !) • ASSIGN KEY FOR PROD='51a826646ebe1f29e33c6ed7c4965c9a'

• Use DB_UNIQUE_KEY for key alias !

• Create new Grid disks with ACL by default • CREATE GRIDDISK ALL PREFIX=sales, availableTo='+asm,PROD'

• Or change ACL for existing cells • ALTER GRIDDISK ALL availableTo='+asm,PROD'

• ALTER GRIDDISK sales_CD_1_cell01, sales_CD_2_cell01,

sales_CD_3_cell01, sales_CD_4_cell01,

sales_CD_5_cell01, sales_CD_6_cell01

availableTo='+asm,PROD‘

• Make sure you repeat existing values in the availableTO ! • Alter GRIDDISK will overwrite every value previous set

Adding/Removing Databases

• Keep cellkey.ora in each database directory

• It doesn‟t hurt, makes the environment more secure

• Retrieve list of defined keys from the cells • CellCLI> LIST KEY

+asm 66e12adb996805358bf82258587f5050

PROD b67d5587fe728118af47c57ab8da650a

• Issue ALTER GRIDDISK to change the “availableTo”

field

• Remember that ALTER GRIDDISK overwrites the contents of

the availableTo field !

Database-Scoped to ASM-Scoped

1. Shutdown all databases and ASM

2. Remove all database keys, from cells CellCLI> ASSIGN KEY FOR PROD=''

3. Delete cellkey.ora from database directory(s)

4. Remove all databases from Grid-Disk availableTo CellCLI> ALTER GRIDDISK ALL availableTo='+asm'

5. Restart ASM and databases.

ASM-Scoped to Open Security

1. Shutdown all databases and ASM

2. Remove ASM keys, from cells CellCLI> ASSIGN KEY FOR +asm=''

3. Delete cluster cellkey.ora from hosts rm /etc/oracle/cell/network-config/cellkey.ora

4. Remove ASM from Grid-Disk availableTo CellCLI> ALTER GRIDDISK ALL availableTo=''

5. Restart ASM and databases.

Exadata Security Best Practices

• When configuring security, the flow is always

• Open security to

• ASM-scoped security to

• Database-scoped security.

• Make sure disk security setup is consistent

• All disks in an ASM group and all disks for a database

• When using RAC, cellkey.ora and permissions are vital

• Make sure they are the same on any node in the cluster.

• Don‟t mix ASM and DB scoped security in a system

• Use DCLI to automate security setup

Exadata Security Summary

• Prevent access to Grid-Disks using keys

• Key on DB nodes level for ASM scoped Security

• Key on DB Home directory for DB scoped Security

• All nodes in ASM cluster or all DB Instances use the same key

• Create alias for keys on Storage cell level

• Setup is per Griddisk on Storage Cell level

• Availableto: field needs both ASM and DB level access

• Shutdown all instances before making changes

• One wrong command can cause issues in the database

Q U E S T I O N S

A N S W E R S

On Exadata Security

Agenda

• Exadata Security

• Flash Cache and Log • Storage Indexes

• Parallel Execution

FLASH CACHE

• 5TB of Flash Memory

• Smart Flash Cache

• Million I/Os per Second

• Sub-Millisecond I/O time

• 50GB/sec Flash Scan

• 3x to 50x Compression on Flash

World’s First Flash Database Machine

Semiconductor Cache Hierarchy

Massive throughput and IOs

through innovative Cache

Hierarchy

• Database DRAM Cache • 400GB raw capacity

• Up to 4TB compressed user data

• 100 GB/sec

• Exadata Smart Flash Cache • 5TB raw capacity

• Up to 50TB compressed user data

• 50 GB/sec raw scan

• 1 million IO/sec

• Exadata disks • 100TB or 336TB raw

• Up to 500TB compressed user data

• 21 GB/sec scan

• 50K IO/sec

Sun Flash Accelerator F20 PCIe Card

• 96GB Storage Capacity

• 4 x 24GB Flash modules/doms

• 6GB reserved for failures

• x8 PCIe card

• Super Capacitor backup

• Optimized for Database caching

• Fully integrated into the Exadata V2

database machine.

• Measured end-to-end performance

• 3.6GB/sec/cell or 50GB/sec per DBM

• 75,000 IOPs/cell or 1 million IOPS per DBM

How can we use Flash Storage

• Exadata Smart Flash Cache • Store frequently used blocks (data and index) on Flash storage

• Exadata Smart Flash Logging • Use Flash as a buffer for your redolog writes

• Exadata Flash-Based Storage • Uses flash as permanent high speed storage

• Or ALL of the above • As we have 4 cards with 96GB of Flash memory

• We can split cards into separate sections with different functionality

Flash storage as Flash Cache

• What is Cached?

• Frequently accessed data and index blocks

• Control file reads and writes

• File header reads and writes

• DBA can also influence caching priorities

• What is Not Cached?

• I/O to mirror copies

• Backup related I/O

• Data Pump I/O

• Data file formatting

• Table scans don‟t monopolize the cache

Flash Storage as Flash Logging

• Redologs are sequential writes

• Disk DRAM cache usually as fast (or faster) than Flash writes

• Provided Disk cache gets flushed fast enough

• Can be bottleneck on high write workloads

• Redolog will write to Flash Cache and DRAM cache

• At the same time

• First one to finish will signal „write complete‟ to the DB

• Starting 11.2.2.4.0 Cell Patchset / 11.2.0.2 BP11 DB version

• Can be turned on or off via the IO Resource Manager

Flash logging example

Flash Storage as Datafile storage

• Using Flash as DB storage reduces Flash space

• 2x when using Normal Redundancy

• 3x when using High Redundancy

• Only use for heave scattered reads and writes

• Sequential writes are usually faster using

• Lots of disk and disk cache

• Dedicated disks so disk-arm does not have to move

• Can be used in a partitioning or ILM scheme

• Most (almost all) customers do not use this feature

Flash Best Practices

• Use Flash only for production

• Unless you have enough (speed and space) for everything

• Use Flash logging for redo-log bottlenecks

• Use Flash Cache and Logging instead of File storage

• Use all available Flash space

• Most performance benefit will be from reading and redologs

• Only change after AWR reports issues with small writes

• Use as much for Cache and Logging as possible

• If you need file storage, only use the part of Flash you need

• Keep the other parts for Cache and Logging

Exadata Smart Flash Cache

Commands

• Creating Flash Cache by default • CREATE CELL flashcache

• Creating Flash Cache explicitly • CREATE FLASHCACHE ALL size=100g

• Users can prioritize caching of tables • ALTER TABLE table_name STORAGE

(CELL_FLASH_CACHE KEEP)

• Without the setting, individual tables and indexes are cached

• Display Flash Cache Configuration • LIST FLASHCACHE DETAIL

Control access to flash cache

• Prevent databases to access flash cache

• Low priority databases

• Test databases

• Etc

• New attribute introduced for IORM: FlashCache

Cellcli> ALTER IORMPLAN

dbplan=((name=prod, flashCache=on),

(name=dev, flashCache=on),

(name=test, flashCache=off),

(name=other, flashCache=off))

Exadata Flash Summary

• 5+ Terabytes of Flash available in Exadata Machine

• About 400Mb per Exadata Storage Cell

• No controller bottlenecks; cards instead of disks

• 3.6GB/sec/cell or 50GB/sec per DBM

• 75,000 IOPs/cell or 1 million IOPS per DBM

• Use it as..

• Exadata Flash Cache

• Exadata Flash Logging

• Exadata Flash file storage

• Needs Normal of High Redundancy

• Only use it if AWR report shows issues with scattered writes

Q U E S T I O N S

A N S W E R S

On Exadata Flash storage

Agenda

• Exadata Security

• Flash Cache and Log

• Storage Indexes • Parallel Execution

STORAGE INDEXES

Biggest performance Bottleneck

• Disks are a big bottleneck for a database

• Its all about getting to data as fast as possible

• Flash disks are much faster than normal disks

• Normal databases need to read data

• To determine if the “where” clause applies to the data

• More data is read than needed

• Indexes help a lot

• But not-reading unneeded data would be faster !

Introducing Exadata Storage Indexes

• Memory based index of contents of blocks

• Index automatically build using memory of the Exadata cells

• Information about a block only stored in one place

• For each column in frequently accessed blocks

• Minimum and Maximum value is stored

• Occurrence of NULL values is stored

• Values automatically stored after first read of the block

• Values in RAM

• Take only small amount of memory

• Destroyed when power is removed (reboot, crash etc)

A B C D

3

1

5

Min B = 1

Max B = 5

Min B = 3

Max B = 8

A B C D

5

8

3

Min B = 6

Max B = 9

A B C D

6

7

9

How do Storage Indexes work An example..

• Only block 1 matches

• Other blocks do not need to be retrieved

• Saves huge amounts of IO per second

• Only works with Smart Scans / Direct Reads

Block 1 Block 2 Block 3

select * from table where B < 2

Storage Index with Partitions Example

• Queries on Ship_Date do not benefit from Order_Date partitioning

• However Ship_date and Order# are highly correlated with Order_Date

• e.g. Ship dates are usually near Order_Dates and are never less

• Storage index provides partition pruning like performance for queries on

Ship_Date and Order#

• Takes advantage of ordering created by partitioning or sorted loading

Order# Order_Date

Partitioning Column

Ship_Date Item

1 2007 2007

2 2008 2008

3 2009 2009

Orders Table

Measuring performance

• Use V$SYSSTAT for information • cell physical IO bytes saved by storage index

Storage Indexes - Summary

• In memory storage of block characteristics

• Highest and Lowest value of each column in a block

• Build when the block is accessed the first time

• Stays in memory

• Destroyed when cellsrv is down (power down, reboot etc)

• No communication between servers

• Low memory footprint

• Cannot turn it on or off

• No extra license fee

Q U E S T I O N S

A N S W E R S

On Storage Indexes

Agenda

• Exadata Security

• Flash Cache and Log

• Storage Indexes

• Parallel Execution

PARALLEL EXECUTION

Automated Degree of Parallelism An 11gR2 generic feature

• In the past, tuning parallelism was a manual process • one degree of parallelism does not fit all queries

• too much parallelism could flood the entire system

• Automated Degree of Parallelism automatically decides • If a statement will execute in parallel or not (serial execution

would take longer than a set threshold – 30 secs)

• What degree of parallelism the statement will use

• Optimizer derives the DoP from the statement based on resource requirements • Uses the cost of all scan operations

• Balanced against a max limit of parallelism

SQL

statement

Statement is hard parsed

And optimizer determines

the execution plan

Statement

executes serially

Statement

executes in parallel

Optimizer determines

ideal DOP

If estimated time

greater than threshold

Actual DOP = MIN(default DOP, ideal DOP) If estimated time less

than threshold PARALLEL_MIN_TIME_THRESHOLD

Automated Degree of Parallelism How it works

Parallel Statement Queuing

• Automatic Degree of Parallelism means

• More statements will run in parallel

• Potential system thrashing due to too many processes

• Parallel Statement Queuing automatically decides if a

statement can execute immediately or not

• When a parallel statement starts it checks if enough

parallel servers (aka Parallel Query Slaves) are

available

• If not, queue the statement

• Once enough parallel servers become available, statement is

de-queued and executed

SQL

statements

Statement is parsed

and Oracle automatically

determines DOP

If enough parallel

servers available

execute immediately

If not enough parallel

servers available queue

128 16 32 64

8

FIFO Queue

When the required

number of parallel servers

become available the first

stmt on the queue is

dequeued and executed

128

16 32 64

Parallel Statement Queuing How it works

Awaiting screen

shot from EM

Clock symbol indicated a queued statement

Click on the SQL ID for more info

Parallel Statement Queuing Monitoring in Enterprise Manager

Wait event indicates stmt is at the head of the queue

Parallel Statement Queuing Monitoring in Enterprise Manager

Real Application

Clusters

In-Memory Parallel Execution

• Data warehouse environments

can have large amounts of

memory that is not always used

• An algorithm places fragments

of an object (partitions) in

memory on different nodes

• Compression gets more data in

memory

• Parallel servers (aka PQ

Slaves) are then executed on

the corresponding nodes

• Requires Automatic Degree of

Parallelism

SQL

statement

Determine the size of the

table being looked at

Read into the buffer

cache on any node

Table is extremely small

Always use direct read

from disk

Table is a good candidate

for In-Memory Parallel

Execution

Table is

extremely Large

Fragments of Table are

read into each node’s

buffer cache

Only parallel server on

the same RAC node

will access each

fragment

In-Memory Parallel Execution How it works

In-Memory Parallel Execution

• A single database machine has over

400GB of memory usable for caching

• Database release 11.2 introduces parallel

query processing on memory cached data • Harnesses memory capacity of entire database cluster

for queries

• Foundation for world record 1TB TPC-H

• Exadata Hybrid Columnar Compression

enables multi-terabyte tables or partitions

to be cached in memory

315,842

1,018,321

1,166,976

ParAccel Exasol Oracle & HP

Exadata

QphH: 1 TB TPC-H

Faster than specialized in-memory warehouse databases

Memory has 100x more bandwidth than Disk

Source: Transaction Processing Council, as of 9/14/2009:

Oracle on HP Bladesystem c-Class 128P RAC, 1,166,976 QphH@1000GB, $5.42/QphH@1000GB, available 12/1/09.

Exasol on PRIMERGY RX300 S4, 1,018,321 QphH@1000GB, $1.18/QphH@1000GB, available 08/01/08.

ParAccel on SunFire X4100 315,842 QphH@1000GB, $4.57 /QphH@1000GB, available 10/29/07.

Paralellism - Summary

• Parallelism is important

• Speed up queries, uses all resources available

• No need for Parallelism tuning

• Needs init.ora parameter

• Optimizer decides on parallelism based on statistics

• No need for Parallelism limitation

• Statements will wait for free slaves

• Prevents overload on the system

• Queries are still faster than single threaded execution

• Parallelism can cause in-memory behaviour

• Qualified tables can be stored in memory automatically

Q U E S T I O N S

A N S W E R S

On Storage Indexes

Q U E S T I O N S

A N S W E R S

Now you do it !