oracle database 18c - s3-us-west-2.amazonaws.com · oracle sharding 28 linear scalability, fault...

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |

Oracle Database 18c

Maria ColganMaster Product ManagerOracle Database Server TechnologiesMarch 2018


Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Oracle Autonomous Database

3

AutonomousDatabase

Automated Data Center and

Database Operations

Expanded Infrastructure Automation

Expanded Database

Automation

Oracle Cloud


Oracle Database 18c

• First yearly release of the Oracle Database– Previously referred to as Oracle Database 12c Release 2

12.2.0.2

• Oracle Database 18c is aimed at providing incremental changes and a focus on stability

• Now Available on Oracle Cloud and Engineered Systems with on-premises following some time after

• “Release Schedule of Current Database Releases” MOS Doc 742060.1

4

https://mosemp.us.oracle.com/epmos/faces/DocumentDisplay?id=742060.1


Oracle Database 12c Release 2

5

Marquee Features

Massive Cost Savings and Cloud Agility with Multitenant

Plus more features for better Performance, Availability, Security, Analytics, and Application Development

Massive Web Applications with Sharding

Massive Performance with Database In-Memory


Oracle Database 12c “Under the Radar” Features

6

Onl

ine

Tabl

e M

ove JSON

Real Time Materialized Views

Index Compression

Auto Generated SequencesLong IdentifiersOnline Tablespace Encryption

Property Graph

SQL Plan Management EnhancementsTo

p N

Que

ries

Invisible ColumnsLonger Varchars

Security Assessment ToolLive SQL

Index Usage Stats

Application Continuity

Auto List Partitioning


What to expect fromOracle Database 18c

7


Performance

8


Performance

• In-Memory Column Store

• Engineered Systems

9

Today Oracle Database 18c

• Memory Optimized Row Store• In-Memory Column Store Improvements

• Performance improvements• Automatic Population


RAC optimizations for Exadata

• Undo Block RDMA-read– In some workloads, more than half of remote reads are

for Undo Blocks to satisfy read consistency– Undo Block RDMA-read uses RDMA to directly and

rapidly access UNDO blocks in remote instances

• Commit Cache– The Commit Cache maintains an in-memory table on

each instance which records the commit time of transactions

– Remote LMS directly reads the commit cache and sends back commit times for requested transactions.• Replaces having to send entire 8K transaction table block

10

New RDMA algorithm decreases messages, latency, and CPU time

Instance 1 Instance 2 Instance 3

UNDOUNDO

RDMA RDMA

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11

Memory Optimized Row Store for OLTP Workloads

• Ultra-Fast key-based lookup: – Uses new lock-free In-memory hash index

– Declare table MEMOPTIMIZE FOR READ

• New low latency client protocol with direct access to data engine

• Performance benefits:– Up to 4x throughput increase for Primary

Key based lookups

– 50% lower response times

Look up ofRay’s ID(4004)

Example : Read Rays Balance


In-Memory 18c Key Features

• Automatic In-Memory Management

• In-Memory For External Tables

• In-Memory for Extreme Capacity NVRAM Memory

• Many performance improvements:– In-Memory Dynamic Scans

– In-Memory Optimized Arithmetic

–Mixed workload performance improvements

12


• In-Memory tables and partitions

automatically ranked using access

Heat Map (introduced in 12c)

• Hot data automatically chosen for

In-Memory population

• Colder In-Memory data

automatically evicted

• Useful for managed cloud services

since no user intervention required

13

Automatic In-Memory Management

Greater Automation and Reduced Administration for Cloud Services

Sales_Q3

Sales_Q2

Sales_Q4

In-Memory Column Store

Sales _Q1


In-Memory Dynamic ScansBetter Columnar Scan Performance

14

• Fast parallelization of simple scans using light-weight threads• Number of active threads

controlled by Resource Manager• Fully leverage CPU resources to

maximize columnar scan performance• Up to 2X gains

SALES

IMCU 1

IMCU 2

IMCU 3

IMCU 4

In-Memory Column Store

REGION

SELECT SUM(total) FROM sales WHERE region = ‘ca’;

REGION REGION

REGION


eXtended Memory (XMEM) SupportUse Non-Volatile Memory to Massively Extend Memory Capacity

• In-Memory algorithms extended to use NVRAM as higher capacity but slower memory

• Performance-critical tables placed in DRAM, capacity-critical tables stored in NVRAM.

• XMEM allows a 3X bigger column store

15

DRAMColumn Store

XMEMColumn Store


• An Oracle Number is a decimal number with up to 38 significant digits• Traditionally all calculations on Oracle

Numbers were done in software for accuracy• Column-Store maintains native binary

representation of NUMBER columns• SIMD Vector Processing applicable on

native binary number representation• Aggregation and Arithmetic operators

can improve up to 40X16

In-Memory Optimized ArithmeticBlazing fast numeric operation

C1 02 35 1FC1 05 06C1 0BC1 2E 17 15

à 1.523à 4.05à 10à 45.222

Oracle Number (SUM) SUM(A)

Slow Row-by-Row Oracle Number Processing

FAST SIMD Vector Processing of Binary Numbers

SUM(A)

CPU

Vect

or R

egist

er152340501000045222


Database In-Memory Performance ImprovementsMultiple Areas of Performance Improvements

17

• Enhanced Mixed Workload performance– Up to 2X Performance Improvement

• Improved aggregation performance– Aggregation push-down to Exadata

• Improved JSON Processing

• Faster expression evaluation

• Improved Fast-Start performance

• Improved Join Processing

• Improved Compression at Query High

• TPCH Performance – 2X Faster


Multitenant

18


Multitenant

• Container managed database virtualization

• Manage Many as one• Patching, Backup, Security, Online Cloning, Online

Relocation

• Software as Service• Shared metadata, Data location transparency

19


• Refreshable PDB Switchover• Transportable Backups• Snapshot Carousel• Faster Upgrades


Refreshable PDB

• PDB Refresh– Incremental refresh of clone with latest

data

CRM

Dev / Test

Pricing Retail

Production

CRM

Snap Clone

Snap Clone

CRM Dev1 CRM Dev2

Developers

TIME

Changes made to database

Master Dev/Test version no longer in sync

Only changes copied and applied


Refreshable PDB Switchover

Server1

CDB1

CDB2

Server2

create pluggable database Greyfrom Grey@CDB2_Link refresh mode auto every 2 minutes;

create pluggable database Redfrom Red@CDB1_Linkrefresh mode auto every 2 minutes;


Refreshable PDB Switchover

Server1

CDB1

CDB2

Server2

1. alter pluggable database refreshmode auto every 2 minutes from Grey@dblink switchover;

2. alter pluggable database Greyopen read write;


Point-in-Time Recovery with Snapshot Carousel

• ”Flashback” to Thursday 10:17am:1. Restore Thursday midnight’s snapshot2. Roll forward to 10:17am from archivelog

– PDB will be “transactionally consistent” as of this time

Carousel stores archive logs along with corresponding snapshot clones

ThursdaySunday Monday Tuesday Wednesday Friday Saturday

Archive Logs


Point-in-Time Cloning with Snapshot Carousel• “Hot Clone” introduced with 12.2• Basic hot clone is transactionally consistent with source PDB

as of SCN at completion of statementalter pluggable database Hot_Clone open;

• Snapshot Carousel enables hot clone to a specific point in time (or SCN)– Restore most recent snapshot from carousel– Roll forward to required point in time by performing recovery from corresponding

archive log

24


Availability

25


Availability

• Comprehensive HA and disaster recovery functionality

• Real Application Clusters, Active Data Guard, Recovery Appliance

• Oracle Golden Gate for heterogeneous replication

• Scale out and fault isolation with Oracle Database Sharding

26


• Zero Impact Grid Infrastructure Patching

• Sharded RAC

• Logically partition data across instances in RAC

• Sharded access for shard-aware applications and transparency for non-sharded applications

• Sharding improvements

• User Defined Sharding

• Improved Cross Shard Query support


Zero Impact Grid Infrastructure Patching

• Zero Impact Patching enables patching of the Oracle Grid Infrastructure without interrupting database operations.

• Patches are applied out-of-place and in a rolling fashion with one node being patched at a time while the database instance(s) on that node remain up and running.

• Zero Impact Patching supports Oracle Real Application Clusters (RAC) databases on clusters with two or more nodes.

27

Never take down a database instance to patch Grid Infrastructure


Oracle Sharding

28

Linear Scalability, Fault Isolation, Global Data Distribution, Cloud Enabled

One giant database partitioned into many small databases (shards)

Customers Americas

Customers

Customers Europe

Customers Asia

• RAC and Data Guard meet needs of over 99% of applications while preserving application transparency

• Some Global-Scale OLTP applications prefer to shard massive databases into a farm of smaller databases

– Avoid scalability or availability edge cases of a single largesystem image database

– Willing to customize data model and applications to enable transactions to be automatically routed to the right shard

• Native SQL for sharding tables across up to 1000 Shards� Routing of SQL based on shard key, and cross shard queries

� Online addition and reorganization of shards

� Linear scalability of data, workload, users with isolation


Oracle Sharding Automated DistributionEnhanced SQL syntax for Sharding

…

CREATE SHARDED TABLE Customers( CustId VARCHAR2(60) NOT NULL,

FirstName VARCHAR2(60), LastName VARCHAR2(60),…

PRIMARY KEY(CustId),)PARTITION BY CONSISTENT HASH (CustId)…

• SQL syntax for creating sharded tables

• Not proprietary APIs as with NoSQL

• Creation of a sharded table automatically

partitions data across shards

• Transparent resharding as data grows

• Choice of sharding methods:

• System managed - consistent hash

• User defined - range, list

• Composite - range-hash, list-hash

• Common reference data (e.g. Price List) is

automatically duplicated on all shards

• Supports shard placement in specific

geographies to satisfy government data privacy

…

…


Sharded Schema

Customers Orders Line Items

Products

Sharded

Duplicated


New Sharding Method: User-defined Sharding

• Partition shards by RANGE or LIST

– List or Range of sharding key values are assigned to each chunk by the user

• Full control on location of data provides:

• Regulatory compliance• Data remains in country of origin

• Hybrid cloud and cloud bursting• Some shards on premises; other shards in the cloud

• Efficient range queries

• Precisely controlled data distribution

32

Explicit mapping of data to individual shards for performance & regulationsSharded Database

User-defined shards on hybrid cloud


• REST API to route requests to the mid-tier associated with a sharding key

– Called by customers’ routing layer

• Affinity improves performance

– Improves caching effectiveness of mid-tier state, reduces # of connections, easy to scale mid-tiers as we scale DB tier

– Greatly reduces network latency for shards distributed between on-prem and public cloud, and geographically distributed shards

Affinity for Mid-tiers and Shards: “Swim Lanes”

…

Router Routing API

…

33


Sharded RAC

• Affinitizes shards to RAC instances

– Requests that specify sharding key will be routed to the RAC instance that logically holds the shard

– Affinity gives better cache utilization and reduced block pings across instances

• Requests that don’t specify sharding key still work transparently

• Gives Sharded Database performance with minimal application changes

– Just add sharding key to the most performance intensive operations

34

Higher performance for shard-aware RAC applications

Oracle RAC Database

Instance 1Partition P1




Data Warehousing and Big Data

35


Data Warehousing and Big Data

• The most advanced analytics engine available today• Partitioning, Compression, SQL, Analytical Views, Analytical

SQL, Data Mining

• Easily analyze data held in Hadoop with Big Data SQL

• Big Data Appliance

36


• In-Memory for external tables• Automatic propagation of nologged data to

standby• More Machine Learning algorithms• Polymorphic Table Functions• Alter Table Merge Partition Online• Approximate Query Improvements


• Oracle Database 11g Release 1– Create index – Add column with default value – Add constraint

• Oracle Database 12c Release 1– Drop Index – Drop Constraint– Alter table set unused column– Alter table move partition

• Oracle Database 12c Release 2– Alter table move for heap tables– Alter table split partition– Alter table modify non-partitioned

table to partitioned table

• New in 18c– Alter table merge partition– Alter table modify partitioned table to

partitioned table

37

Online Table Maintenance Operations


Top-N approximate aggregationInteractive response times against terabytes of data

• Approximate results for common top-N queries– How many approximate page views did the top five blog posts get last week?

– What were the top 50 customers in each region and their approximate spending?

• Order of magnitude faster processing with high accuracy (error rate < 0.5%)

• New approximate functions APPROX_COUNT(), APPROX_SUM(), APPROX_RANK()

39

SELECT blog_post, APPROX_COUNT(*)FROM weblogGROUP BY blog_postFETCH FIRST 5 ROWS ONLY;

SELECT region, customer_name, APPROX_RANK(PARTITION BY region

ORDER BY APPROX_SUM(sales) DESC) apx_rank,APPROX_SUM(sales) apx_sales

FROM sales_transactionsGROUP BY region, customer_nameHAVING APPROX_RANK(..) <=50;

Top 5 blogs with approximate hits Top 50 customers per region with approximate spending


Polymorphic Tables: Self-Describing, Fully Dynamic SQL

• Part of ANSI 2016

• Encapsulate sophisticated algorithms– Hides implementation of algorithms

– Leverage powerful, dynamic capabilities of SQL

– Pass in any table-columns for processing

– Returns SQL rowset (table, JSON, XML doc etc)

• E.g. return credit score and associated risk level

40

CREDIT RISK ALGORITHM

SQLQUERY

POLYMORPHIC TABLE FUNCTION

INPUTS:

TABLE JSONXML ANALYTIC VIEW

SELECTstate_id, . . ., AVG(credit_score), risk

FROM CREDIT_RISK(tab => scott.customers,cols => columns(dob, zip,loan_default),outs => columns(credit_score, risk_level))

WHERE risk_level = ‘High’GROUP BY state_id;

SCOTT.CREDIT_RISK

HHHH

STATE_ID RISKA_SCOREPOP LOANS A_LOAN


Inline external tablesTransparently access external data

• External table definition provided at runtime– Similar to inline view

• No need to pre-create external tables that are used one time only– Increased developer productivity

41

INSERT INTO salesSELECT sales_xt.*FROM EXTERNAL(

(prod_id number, … )TYPE ORACLE_LOADER …LOCATION ’new_sales_kw13')REJECT LIMIT UNLIMITED );

CREATE TABLE sales_xt(prod_id number, … )TYPE ORACLE_LOADER …LOCATION ’new_sales_kw13')REJECT LIMIT UNLIMITED );

INSERT INTO sales SELECT * FROM sales_xt;

DROP TABLE sales_xt;


• External Tables allow transparent access to data outside the DB• In-Memory For External Tables

builds in-memory column cache of data outside the DB for ultra-fast analytics on external data • All In-Memory Optimizations apply– Vector processing, JSON expressions

extend transparently to external data

• Up to 100X faster

In-memoryExternal

Tables

RDBMSIn-memory

DatabaseTables

External Data

DB TABLES

In-Memory For External TablesFast Analytics on External Data

Object Storage

Files


Machine Learning and Advanced Analytics in 18c

• Machine Learning Algorithms available since 9i Release 2 (2002)• New algorithms in 18 – Random Forests for Classification– Neural Networks for both classification and regression– Explicit Semantic Analysis ML algorithm extended to support classification– Time Series via Exponential Smoothing– CUR decomposition-based algorithm for attribute and row importance

• New ability to export ML models to C and Java for applications deployment


Property Graph Improvements

• PGQL – Property Graph Query Language

– SQL-like declarative language to queryin-memory and in-database Property Graph

– Supports graph pattern matching and recursivepath queries

– Proposing as ISO standard; Language is also licensed under Apache https://github.com/oracle/pgql-lang

• In-memory virtual columns for RDF Graph

– Up to 100X faster queries performance

• RDF Graph networks now support list-hash composite partitioning

– 5 to 10 times query performance improvement

44

PGQL

SQL WITH temp(device_id, device_name) AS (-- Anchor member:SELECT device_id, nameFROM DevicesWHERE name = 'Regulator, HVMV_Sub_RegB‘

UNION ALL-- Recursive member:SELECT Devices.device_id, Devices.nameFROM temp, Devices, Connections conn1,

Connections conn2, ConnectorsWHERE temp.device_id = conn1.to_device_idAND conn1.from_connector_id =

Connectors.connector_idAND Connectors.connector_id =

conn2.from_connector_idAND conn2.to_device_id = Devices.device_idAND temp.device_id != Devices.device_id)

CYCLE device_id SET cycle TO 1 DEFAULT 0SELECT DISTINCT device_nameFROM tempWHERE cycle = 0AND device_name = 'Regulator, HVMV_Sub_RegB'

PATH connects_to := (from) <- (connector) -> (to)SELECT y.nameWHERE (x:Device) -/:connects_to*/-> (y:Device),

x.name = 'Regulator, HVMV_Sub_RegB'),x != y

https://github.com/oracle/pgql-lang


• Spatial Index Performance– Enhancements to CBTree index to use the data layer directly for Spatial index access.

– 3x faster query performance for large point data sets.

• Sharding support for Spatial tables

• Database In-Memory support for Point data

• ORDS support for Spatial operations

• Support distributed transactions on tables with spatial indexes

• Dynamic tile layers for spatial map visualization

45

Spatial Improvements


Security

46


Security

• Security In-Depth• Access Controls, Encryption, Redaction, Masking, Auditing,

SQL Firewalls

• Key Vault

• Audit Vault Database Firewall

• Security Assessment Tool

47

Today

Active Directory

• Integration with Active Directory• Authorization to database through Active Directory

user/group mappings to database schema users and roles

• Per PDB Key storage

• Password-less schema creation• No default passwords

Oracle Database 18c


Development

49


• Application Express 5.1

• SQL Developer & SQLCl

• Comprehensive Language support• PL/SQL, SQL , Python, Node.js, PHP, Java, C, .NET, REST

• JSON• SODA API, SODA REST API

• SQL Support

• Improved JSON Support

• Rolling patches for OJVM

• Private Temporary Tables

50

Development



JSON Enhancements

• Generate large JSON documents from relational data– JSON generation extended to supports LOB’s

• New SODA (Simple Oracle Document Access) drivers– OCI and PL/SQL now added, in additional to JSON and REST– Simple, non-relational (‘nosql-like’) API for accessing JSON data

• New TREAT (<expression> AS JSON) operator– Dynamically declare operands to be handled as JSON data, enabling more seamless JSON

optimizations • Extended key length for JSON search indexes– Raise the key length from 64 bytes to 255 bytes; enables faster search queries for JSON

objects containing long key names.

51

Simpler development of JSON-centric applications using Oracle Database

Oracle Confidential – Internal

oracle database 18c - s3-us-west-2.amazonaws.com · oracle sharding 28 linear scalability, fault...

Documents