Modernize your Data Platform with SQL TechnologiesBob WardPrincipal ArchitectData Group, Tiger Team

bobward@microsoft.com@bobwardms, #bobsql

Deck on slideshare.net

Demos on github

• Disparate systems and processes

• Multiple tools and skillsets

• High cost of ownership

• Siloed insights on disconnected data

• Costly data movement and delay

Challenges of the Modern Data PlatformFragmented architectures are inefficient

Analytics

Visualization

Relational in the cloud

Analytics

Visualization

Hadoop

{ }NoSQL Data

Lake

Analytics

Visualization

Relational DBAppliance

Analytics

Visualization

Hadoop Appliance

Clou

dOn

-pre

mise

s

Relational Non-relational

Azure SQL DatabaseAzure SQL Data WarehouseSQL Server 2016 on VM

Analytics Platform System

Azure Data Lake

SQL Server 2016Analytics Platform System

SQL

Relational Non-relational

On-p

rem

ises

Clou

dMicrosoft Data PlatformTransforming data into intelligent action

• Manage all your data in a mission critical, scalable, secure way

• Gain deep insights across all of your data without data movement

• Utilize existing skills and investments

• Consistent on-premises, cloud and hybrid experiences

Federated queryBusiness intelligenceMachine learning & advanced analytics

Insights

Microsoft SQL Server – On-premises & cloud

SQL Server SQL Server in Azure VM

Azure SQL Database

Dev ops tools Identity VirtualizationManagement tools

Azure SQL Data WarehouseAnalytics Platform System

Common tools for development and managementCommon T-SQL surface area

Simple cloud migration

Consistent experience is everything

SQL Server 2016

Continuous Innovation

End-to-end mobile BI

Advanced Analytics

Enterprise-grade DW

Mission critical OLTP

The bestSQL Server release in history

SQL Server 2016

SpeedAgility

ProvenFeedback

SQL DB SQL DB SQL DB SQL DB

New innovations

Across customer base

Cloud-First Approach + DevOps

Continuous enhancemen

ts

Hybrid cloud

1/12th the cost of Oracle

National Institute of Standards and Technology Comprehensive Vulnerability Database update 12/2016.

Best price/performance

Everything built-in

Highest performing data warehouse

Most secure database

Best TCO Most consistent experience

On-premises to cloud #1 performance

TPC-H

Oracle

is #5#2

SQL Server

#1

SQL Server

#3

SQL Server

The Power of SQL ServerEverything built-in

June 2016SQL Server 2016

TPC-E

6 years in a rowleast vulnerable

2010 2011 2012 2013 2014 2015 20160

10

20

30

40

50

60

70

80

0 1 40 0 3 0

3429

2215

5

2216

6

43

20

69

18

49

74

3

SQL Server

Oracle

MySQL2

SAP HANA

1/12

Only data solution to encrypt your data at rest and in motion

Connect your relational data to big data with PolyBase

Real-time operational analytics without impacting performance

Up to 30x faster transactions, 100x faster queries with InMemory

Unparalleled choice for developer tools and languages

1 T-SQLJavaC/C++C#/VB.NET

PHPNode.jsPythonRuby

For all your applicationsInnovations across all editions

Available nowSQL Server 2016 SP1

On the platform of your choiceSQL Server v.Next

Targeting CY2017SQL Server v.Next GA*

*SQL Server v.Next Public Preview available now on Linux, Windows, and Docker.

Here is How SQL Server 2016 Just Runs FasterCore Engine ScalabilityAutomatic Soft NUMADynamic Memory ObjectsSOS_RWLockFair and Balanced SchedulingParallel INSERT..SELECTParallel Redo

DBCCDBCC ScalabilityDBCC Extended Checks

TempDBGoodbye Trace FlagsSetup and Automatic Configuration of FilesOptimistic Latching

I/OInstant File Initialization is No Longer HiddenMultiple Log WritersIndirect Checkpoint Default Just Makes SenseLog I/O at the Speed of Memory

SpatialNative ImplementationsTVP and Index Improvements

ColumnstoreBatch Mode and Window Functions

Always On Availability GroupsTurbochargedBetter Compression and Encryption

and there is more

Persisted Log Buffer

The evolution of storage

HDD SSD (ms)PCI NVMe SSD (μs)

Tired of WRITELOG waits?Along comes NVDIMM(ns)

Windows Server 2016 supports block storage (standard I/O path)A new interface for DirectAccess (DAX)

2x speeds over PCI NVMe

Persistent Memory

(PM)

Watch these videos

Channel 9 on SQL and PMM

NVDIMM on Win 2016 from \\build

Format your NTFS volume with /dax on

Windows Server 2016

Create a 2nd tlog file on this new volume on SQL

Server 2016 SP1

Tail of the log is now a

“memcpy” so commit is fast

WRITELOG waits = 0 ms Gory Details here

Mission critical platform Deeper Insights Hyperscale

In-Memory OLTPGreater T-SQL surface area, terabytes of memory supported, and higher number of parallel CPUs

Query Store A flight data recorder for your database

Always EncryptedSensitive data remains encrypted at all times, with ability to query

Row-Level SecurityFine-grained access control for table rows

Enhanced Always OnDistributed Availability Groups, Basic Availability Groups, Automatic Failover

The richest set of diagnostics in the industry.

Read more here

High-Performance Execution Engines

ColumnStore + Batch Mode Execution• Columnar compression to save

space in memory and on disk• New execution engine that

optimizes instructions/row and minimizes CPU data bus I/O stalls

• Often gives speedups of 10x• Allows you to store, manage,

and gain value out of significantly more data

In-Memory OLTP Engine Model• Lock-free algorithms• Optimistic concurrency model• Full transactional support• Compiled procedures• Can give significant speedups

(2x-50x) on high-end OLTP applications

Capturing technical breakthroughs in two areas

These enable you to write SQL and leverage ever-increasing performance for your

applications

Mission Critical

Platform

Demo

Absorb shocks with In-Memory OLTP

Secure Code

•Secure development lifecycle•Least vulnerable last 6 years

Most Secure DatabaseSecure engine

Least vulnerable last 6 years

2006 2007 2008 2009 2010 2011 2012 2013 2014 20150

1020304050607080

0 211 8

0 1 4 0 0 3

69

53 53 55

3429

2215

5

22

5

28 25 2921

613

3 06

16 169 8 6

43

20

69

18

49

3

SQL Server Oracle DB2 MySQL SAP HANA

* National Institute of Standards and Technology Comprehensive Vulnerability Database update 10/2015

while being most utilized

Mission Critical

Platform

Most Secure DatabaseSurrounded by layers of protection

Secure Code

•Secure development lifecycle•Least vulnerable last 6 years

•SQL Threat Detection •SQL Server Auditing

•Row-level Security•Dynamic Data Masking

•Always Encrypted•Transparent Data Encryption•Encryption-in-flight

Control Access

•SQL Authentication•Windows Authentication• Azure Active Directory Auth.

Monitor activity

Control access

Protect data

Cloud-only

Come to the instructor lead security lab at 4:45

pm today!

Mission Critical

Platform

trust boundary

"SELECT Name FROM Customers WHERE SSN = @SSN","111-22-3333"

Always EncryptedProtect data at rest and in motion, on-premises and in the cloud

NameWayne Jefferson

ADO .NET Name0x19ca706fbd9a

Result SetResult Set

Client

Name SSN Country0x19ca706fbd9a

0x7ff654ae6d USA

SQL Server or SQL Database"SELECT Name FROM Customers WHERE SSN = @SSN",0x7ff654ae6d

ciphertext

Encrypted sensitive data and corresponding keys are never seen in plaintext in SQL Server

dbo.Customers

ciphertext

Security

Mission Critical

Platform

Optimizing query performance and costsQuery Store = flight data recorder for databases

Built-in query monitoring• Negligible perf & storage overhead• Survives failover, memory

pressure, …• Data ready when that incident

happens

T-SQL interface and SSMS UX

Top insights into your workload

• Query and resource stats • Query plans

Developer

App User

Azure

application On-premise

Developer needs to find and fix the underlying problem, ASAP

Customer reports the issue(app is slow/unresponsive)

Mission Critical

Platform

Demo

Use Query Store to Fix your Critical Performance Problem

Improved Always On Availability Groups• 3 Replicas for Automatic

Failover• Database Failure Detection

Higher Availability

• Support for Column-store Indexes

• Load Balancing of Read Workloads

More Powerful Readable

Secondaries

• 4X Synchronization ThroughputScalability

• Different Windows Clusters• Different Windows DomainsFlexibility

• Basic Availability Groups in Standard EditionLicensing

Always On Turbocharged

Mission Critical

Platform

Mission critical platform Deeper Insights Hyperscale

Real-time operational analyticsRunning of analytical workloads concurrently with OLTP workloads

R IntegrationInsights from data across SQL Server and Hadoop with the simplicity of T-SQL

PolyBaseInsights from data across SQL Server and Hadoop with the simplicity of T-SQL

Mobile BIMobile reports on SSRS with Mobile Report Publisher

Temporal Tables, JSONTrack change history, integrate with modern service easily

0100101010110

SQL Server OLTP

SQL Serverdata

warehouse

ETL

In-memory ColumnStore

In-memory OLTP

Real-time frauddetection

Operational analyticsSingle system for OLTP and real-time analytics

CapabilityIn-memory Columnstore index over in-memory/disk based OLTP tables

Benefits You gain analytics on fresh data

& 30x faster transactions & 100x faster analytics queries

In-memory for more applications

Fraud detected

2-24hrs

Deeper Insights

Data ScientistInteract directly with data

Built-in to SQL Server

Data Developer/DBAManage data and analytics together

Built-in advanced analyticsRunning R algorithms at massive data scale

Example Solutions• Sales forecasting• Warehouse

efficiency• Predictive

maintenance

Relational Data

Analytic Library

T-SQL Interface

Extensibility

?RR

Integration

010010

100100

010101

Microsoft Azure Marketplace

New R scripts01001

010010

001010

1

010010

100100

010101

010010

100100

010101

010010

100100

010101

010010

100100

010101

• Credit risk protection

Deeper Insights

PolyBaseQuery relational and non-relational data with T-SQL

T-SQL query

CapabilityT-SQL query across SQL Server and Hadoop

Benefits New insights across “data

lake” Leverage existing skillsets

and BI tools Faster time to insights,

simple ETL

SQL Server

Hadoop

Quote:****************************************************************************************************************

$658.39

Jim GrayName

11/13/58

DOBWAState

Ann Smith

04/29/76

ME

Deeper Insights

Support for Temporal DataTrack historical data

Deeper Insights

• Data changes over time (!)

• Databases naturally provide a current view

• But an historical perspective is often critical

Time travel

Data audit

Predictive analytics

Repair record-level corruptions

Demo

Go back in time with temporal tables

Mission critical platform Deeper Insights Hyperscale

Stretch DatabaseStretch operational tables in a secure manner into Azure for cost-effective historic data availability.

Enhanced backup to AzureFaster restore times, 50% reduction in storage. Larger DBs

Order history    Name SSN DateJane Doe cm61ba906f

d2/28/200

5Jim Gray ox7ff654ae6

d3/18/200

5John Smith i2y36cg776r

g4/10/200

5Bill Brown nx290pldo9

0l4/27/200

5Sue Daniels ypo85ba616

rj5/12/200

5Sarah Jones bns51ra806f

d5/22/200

5Jake Marks mci12hh906

fj6/07/200

5Eric Mears utb76b916gi 6/18/201

4Rachel Hogan px61hi9306f

j 7/1/2014

Sam Johnson ol43bi506gd 7/12/2014

David Simon tx83hal916fi 7/29/2014

Michelle Burns nb95re926gi 8/10/2014

Reed Dean vc61ira536fe

8/23/2014

Order history    Name SSN DateJane Doe cm61ba906f

d2/28/200

5Jim Gray ox7ff654ae6

d3/18/200

5John Smith i2y36cg776r

g4/10/200

5Bill Brown nx290pldo9

0l4/27/200

5

Customer data

Product data

Order History

Stretch to cloud

Stretch SQL Server into AzureStretch cold data to Azure with remote query processing

CapabilityStretch cold data from on-premises SQL Server tables to Azure with remote query processing

Benefits Cost effective historical data Entire table is online and remains

queryable from on-premises apps

Transparent to applications Supports Always Encrypted &

Row Level SecuritySQL Server App

Query

Microsoft Azure

Jim Gray ox7ff654ae6d

3/18/2005

Hyperscale Cloud

SQL Server in the cloud

Azure SQL Virtual Machines

AzureSQL Database

DB

AzureSQL Data Warehouse

DW

Fully managed Database Service …So you can focus on your Business

• Built for SaaS and Enterprise applications• Predictable performance & Pricing • Elastic database pool for unpredictable SaaS

workloads• 99.99% availability built-in• Geo-replication and restore services for data

protection• Secure and compliant for your sensitive data• Fully compatible with SQL Server 2016

databases

Predictable workloadsSingle databases or partitioned data across multiple

databases; scale between service tiers and performance levels as capacity needs fluctuate.

Scal

e da

taba

ses

up

as n

eede

d

Scale out/in the pool

reads/writes

reads/writes

read

s/writ

es

reads/

writes

…

Single database or partitioned databases

Customer1

Customer2

Customer3 Customer

#N…

Unpredictable workloadsFor large numbers of databases with unpredictable performance demands; pool resources to be shared between these databases.

Elastic Database Pool

Data

base

s co

nsum

e re

sour

ces

as n

eede

d

Managing large numbers of Databases

SQL Database Service Tiers (single DB model)

*The 99.99% availability SLA does not apply to the existing Web and Business editions, which will continue to be supported at 99.9% availability.

Built For

Available SLA

Database Max Size

Business Continuity

Security

Performance Objectives

Database Transaction Units (DTUs)

Available Tiers ($/Month) and GA

Price

Point-in-time Restore (“oops” Recovery)

BASIC PREMIUMSTANDARD

P1S0

Light transactional workloads

Medium transactional workloads Heavy Transactional Workloads

99.99%*

2 GB 250 GB 500 – 1,000 GB (more soon)

Any point within 7 days Any point within 14 days Any point within 35 days

Geo-restore Geo-replication, standby secondary

Active geo-replication, up to four readable secondary backups

Auditing, row-level security, dynamic data masking, encryption

Transactions per hour Transactions per minute Transactions per second

5

$4.99

S1 S2 S3 P2 P4 P6 P11

10 20 50 100

$15 $30 $75 $150

125 250 500 1,000 1,750

$465

$930

$1,860

$3,720

$7,001

P15 …

4,000

$16,003

Azure SQL Data Warehouse• A relational data-warehouse-as-a-service, fully managed by Microsoft. • Industries first elastic cloud data warehouse with proven SQL Server capabilities.• Support your smallest to your largest data storage needs.

Elastic scale & performance

Scales to petabytes of dataMassively Parallel ProcessingInstant-on compute scales in secondsQuery Relational / Non-Relational

Saas

Azure

PublicCloud

Office 365Office 365

Get started in minutesIntegrated with Azure ML, PowerBI & ADF

Powered by the Cloud

Market Leading Price & Performance

Simple billing compute & storagePay for what you need, when you need it with dynamic pause

AzureAzure

Migrate to the Modern Data Platform

Migration to Meet your Needs

Upgrade MigrationOther

DatabaseSource

Target

Windows Azure Windows Azure

Tool(s) DMA DMA SSMA

Legacy

Modern

ComparisonReports

A’SQL 2008 Instance

Extensive tracing

StatisticalAnalysis

Analyze Results

BSQL 2016 Instance

Extensive tracing

capture prod workload

DEA

replay

BMW Migrations by Industry

Financial Services 139

Discrete Manufacturing 122

Retail & Consumer Goods 113

Government 105

Health 94

Professional Services 93

Process Manufacturing & Resources

59

Hospitality & Travel 44

Telecommunications 36

Power & Utilities 31

Other Industries 119

TOTAL 955

Azure SQL DatabaseAzure SQL Data WarehouseSQL Server 2016 on VM

Analytics Platform System

Azure Data Lake

SQL Server 2016Analytics Platform System

SQL

Relational Non-relational

On-p

rem

ises

Clou

dMicrosoft Data PlatformTransforming data into intelligent action

• Manage all your data in a mission critical, scalable, secure way

• Gain deep insights across all of your data without data movement

• Utilize existing skills and investments

• Consistent on-premises, cloud and hybrid experiences

Federated queryBusiness intelligenceMachine learning & advanced analytics

Insights

• SQL Server 2016 Home Page• Free E-Book on SQL Server 2016 • SQL Server Blog• MSSQL Tiger Team Blog• SQL Server 2016: It Just Runs Faster

Blog Series

Resources

bobward@microsoft.comslideshare.net, github

@bobwardms, #bobsql

© 2016 Microsoft Corporation. All rights reserved.

Bonus Material

Industry ObservationsWorkload-Specific, Optimized Engines Major Performance Enhancements vs. Traditional B+-Tree-based on-disk technologies Leveraging Knowledge of Workload to Improve Performance by Orders of Magnitude In-Memory OLTP and In-Memory + ColumnStore address these within SQL

Rise of Clouds (Public and Private) Have a VM or database in minutes Shut it off when you don’t want it anymore to stop paying for it Simplified Management Experiences, PaaS/IaaS

Accelerated Value Delivery To Customers Customers want value more quickly - Engineering model evolving to go faster Increase Feedback During Development Cycles to Improve Features Make it faster and easier for developers and DBAs to solve problems through T-SQL

In-Memory Transactional Processing

SQL Server Integration

• Same manageability, administration & development experience

• Integrated queries & transactions

• Integrated HA and backup/restore

In-Memory Data Access

• Indexes (hash and range) exist only in memory

• Records exist only in memory (pointers from indexes)

• No buffer pool, B-trees

• Stream-based storage

T-SQL Compiled to Machine Code• T-SQL compiled to

machine code via C code generator

• Invoking a procedure is just a DLL entry-point

• Aggressive optimizations @ compile-time

Declining memory price

Many-core processors

Stalling CPU clock rate Lower TCO

Hardware trends Business

Integrated experience

High performance

data operations

Efficient business-logic

processingCust

ome

r Be

nefit

sAr

chit

ectu

ral P

illar

sD

rive

rs

High Concurrency

• Core engine uses lock-free algorithms

• No lock manager, latches or spinlocks

• Multi-version optimistic concurrency control with full ACID support

Frictionless scale-up

45

Column-Store Indexes

Improved compression:Data from same domain

compress better

Reduced I/O:Fetch only columns needed

…

Row-store indexes Column-store indexes

Ideal for OLTP Efficient operation on small set of rows

C5C1 C2 C3 C4

Improved Performance:More data fits in memoryOptimized for CPU utilization

Ideal for DW Workload

rowgroup

segment

Benefits of Column-Store Indexes

Safe Upgrades to SQL 2016 with Query StoreSafe Upgrade Workflow:

1. Upgrade to SQL 2016, but do not enable new features yet (keep the old DB Compatibility Level)

2. Turn Query Store ON, record your workload in Query Store

3. Enable new Compatibility Level (130)

4. Use Query Store to find any and fix performance regressions

Temporal TablesTracking history is important but challengingCommon to healthcare, pharmaceutical, financial services, social networks…

SQL Server makes it super easyTrack data changes with zero code in your appZoom-in data analysis on any point in time

Time TravelData Audit

Row-level error correction

• Interchange data with apps and services

• Exploit agility of NoSQL to easily extend your app

Agile development with JSON

SQL ServerWeb app, service

CREATE TABLE Departments ( DeptId int NOT NULL PRIMARY KEY CLUSTERED , DeptName varchar(50) NOT NULL , ManagerId int NULL) ;

How Temporal Tables WorkEnd-to-end m

obile BI

CREATE TABLE Departments ( DeptId int NOT NULL PRIMARY KEY CLUSTERED , DeptName varchar(50) NOT NULL , ManagerId int NULL , SysStartTime datetime2 GENERATED ALWAYS AS ROW START NOT NULL , SysEndTime datetime2 GENERATED ALWAYS AS ROW END NOT NULL , PERIOD FOR SYSTEM_TIME (SysStartTime, SysEndTime) ) WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = DepartmentHistory)) ;

How Temporal Tables WorkEnd-to-end m

obile BI

How Temporal Tables Work

Insert

prior version Start EndDeptId DeptName ManagerId

DepartmentsHistory

End-to-end mobile

BI

Update / Delete

Departments (temporal)

DepartmentsStart EndDeptId DeptName ManagerId

How Temporal Tables Work

Start EndDeptId DeptName ManagerId

DepartmentsHistory

End-to-end mobile

BI

Normal querySELECT * FROM Departments

GO

Departments (temporal)

DepartmentsStart EndDeptId DeptName ManagerId

How Temporal Tables Work

Start EndDeptId DeptName ManagerId

Departments DepartmentsHistory

End-to-end mobile

BI

Temporal query SELECT * FROM Departments

FOR SYSTEM_TIME AS OF '2015.01.01'

GO

Departments (temporal)

Start EndDeptId DeptName ManagerId

Paginated reportsDesign beautiful documents using updated tools and new features

Mobile reportsCreate responsive, interactive reports optimized for mobile devices

Modern web portal Consume both types of reports in one web portal using modern browsers

SQL Server Reporting ServicesModern enterprise-ready on-premises reporting platform

SQL Server Analysis Services 2016Enterprise scale models with in-memory technology built-in

Use SQL Server Analysis Services asa semantic model without storing dataHigh performance DirectQuery for Tabular

Make it easier to create powerful models Tabular modeling capabilities and performance

Improve manageability and securityProvide additional tools for administrators

Strengthen multidimensionalPerformance and other updates

Easily create models

IT pro

Increase adoption

Business analyst

Share insights faster

Faster time to insight

BI consumers

Move your SQL data tier to CLOUD

Why move? → TCO, Agility, Elasticity, SimplicityElastic resources, on-demand, pay-as-you-goDramatically less management burden on youRich ecosystem of services, to build rich appsMigration assistants to help you move

OLTP

Industry leading TCO

$648K+ $120

per user for Power BI

SQL Server 2016

ADVANCED ANALYTICS

BUSINESS INTELLIGENCE

OLTP

DATA WAREHOUSING

3.4x

20x per user

$2.2M+

$2,230 per user for BI

Note: For OLTP and DW scenario, the price comparison is based on a server with 2 proc, 8 cores each

Built-in with SQL Server vs. expensive add-ons with Oracle

Complete mobile BI

In-memory

End-to-end security

Advanced Analytics

built-in

built-in

built-in

built-in

DATA WAREHOUSING

BUSINESS INTELLIGENCE

ADVANCED ANALYTICS

Easy with SQL Server migration assistants

SSMA for Oracle

SSMA for Sybase

SSMA for DB2

SSMA for MySQL

SSMA for Access

Confidential under NDA