Lynn LangitSr. Developer EvangelistMicrosoft CorporationBlogs.msdn.com/SoCalDevGal

Migrating Applications to SQL Azure

v1.1

SESSION CODE: COS311

2

Migrating Applications to SQL Azure

OverviewSQL Azure

App Migration Data Access Tier Considerations

Database MigrationMigrating to SQL Azure from MySqlMoving Schema and Data into SQL AzureTransferring Large Data into SQL Azure

Future

3

Developer ExperienceUse existing skills and tools

The Windows Azure Platform

platformAppFabric

Compute Storage Management Relational data Management Connectivity Access control Reporting & BI

Billing & Payments

Flexible APIs

Information Marketplace

4

SQL Azure DatabaseThe first and only true relational database as a service

Elastic Scale Developer Agility

Self-managed

• Database utility; pay as you grow

• Flexible load balancing• Business-ready SLAs• Enable multi-tenant

solutions• World-wide presence

• Easy provisioning and deployment

• Auto high-availability and fault tolerance

• Self-maintaining infrastructure; self-healing

• No need for server or VM administration

• Build cloud-based database solutions on consistent relational model

• Leverage existing skills through existing ecosystem of developer and management tools

• Explore new data application patterns

Application Topologies

Application/ Browser

App Code

(ASP.NET)

App Code(ASP.NET)

TS

QL T

DS

SQL Azure

WindowsAzure

Code Near

App code/ Tools

SQL Azure

Code Far

Hybrid

SQ

L A

zure

/ D

ata

Syn

c

SQL Azure

SQL Server App code/ Tools

App Code

(ASP.NET)

App Code(ASP.NET)

T-S

QL /

TD

S

TS

QL T

DS

WindowsAzure

6

When Migrating Applications

Data Access APIs SupportSecurity ModelConnections & SizingAdmin and Programmability Surface SupportCollation Behavior

7

Data Access APIsSupported APIs

Connection StringADO.Net - Encrypt=True and add @servername to User IDODBC - Encrypt=yes and add @servername to Uid

Supported UnsupportedADO.Net .Net 3.5 SP1 and 4.0 ODBC - SNAC 10Entity Framework .Net 3.5 SP1 and 4.0SQL Server 2008 Driver for PHP v1.1

OleDB

8

Data Access APIs cont.

Protocols

Authentication Mode

Supported UnsupportedTCP/IP over port 1433 Dynamic ports

Named pipesShared memory

Supported UnsupportedSQL Auth Windows Auth

9

Connectivity & SizingRobust Connection Management through Retry Logic

More On this Topic - COS305: Microsoft SQL Azure Development Best PracticesFailover

HW Failures or Upgrades

Resource Management OperationsThrottling and Load Balancing

ALTER DATABASE (T-SQL)Changes to Database Name Changes to MAXSIZE/EDITION

ALTER DATABASE [TestDB] MODIFY (MAXSIZE=50GB)GO

10

Security ManagementAuthentication - SQL Server (only)Manage Logins

CREATE LOGIN [testlogin] WITH PASSWORD='password‘

Manage UserCREATE USER testuser FROM LOGIN testlogin;

Manage Server RolesEXEC sp_addrolemember ‘dbmanager', 'testuser'; [securityadmin role]EXEC sp_addrolemember ‘loginmanager', 'testuser'; [dbcreator]

Granting Database Access to a LoginCreate Login and Database in MasterCreate User and Grant Role in User Database

Viewssys.sql_logins. sys.databases, sys.database_principal

11

Firewall ManagementPortal AccessProgrammatic Access

System Views and Stored Procedures - hereView: sys.firewall_rulesStored Procedure: sp_set_firewall_ruleStored Procedure: sp_delete_firewall_rule

API Implementation Example- here

12

CollationsDefault Collation

Set to SQL_LATIN1_GENERAL_CP1_CI_ASApplies to…

Metadata, TempDb and Default for User Data Object Names (tables, sprocs etc), String variables and Columns when no collation is specified.

Use column and/or expression level collation when neededCREATE TABLE [dbo].[ErrorLog](

[ErrorLogID] [int] IDENTITY(1,1) NOT NULL,[ErrorTime] [datetime] NOT NULL,[UserName] [nvarchar(4000)] COLLATE Turkish_100_CI_AI NOT NULL,[ErrorNumber] [int] NOT NULL,[ErrorMessage] [nvarchar](4000) COLLATE Japanese_CI_AS NOT NULL,

CONSTRAINT [PK_ErrorLog_ErrorLogID] PRIMARY KEY CLUSTERED ([ErrorLogID] ASC) )

declare @str nvarchar(128) COLLATE Greek_CS_AI

13

Feature ParityAdmin Surface – no physical administration

You have a master database You do NOT have server level constructs (sp_configure, endpoints, DBCC commands, server level DMVs and System Views…)

Programmability SurfaceMost RDMS features availablePartially available - USE, XML processing, deprecated T-SQL etc. – list hereSome Features are NOT available today

Full-text Search, Remote Data Access and Linked Servers, Distributed Transactions, Change Tracking, Service Broker etc.

List is available here

14

Migrating ApplicationsApplication Data Access Tier Just Works Need to Investigate

Data Access APIs Support ADO.Net, ODBC, EF and PHP

OleDB

Security Model SQL Auth Windows Auth

Connection Management Apps with transaction retry logic, 3-tiered, stateless or partially statefull apps

Legacy Apps2 tiered apps with online connections

Admin and Programmability Surface Support

Tables, Views, Sprocs, Functions, Triggers, built-in data types etc

Server level options, FTS, Service Broker, Distributed Transactions

Collation Behavior Apps explicit with collation options

Non-English apps depending on db/server collation defaults

15

Database Migration

Moving Schema and / or DataSource Database types

SQL ServerMySQLAccessOther RDMS

16

From SQL Server to SQL Azure

Tool ChoicesSSMS for R2 > Generate Script WizardSSMS for R2 > Import / Export WizardCopy Database commandSSIS for R2SQL Azure Migration Wizard3rd Party

17

SSMS– Generate Script WizardMigration of schema and/or data w/ fine grain control

Pros - Native support for SQL Azure Schema options: “Engine Type = SQL Azure”

Ensures correct options and settings are applied for the TSQL script generation.Requires explicit action on unsupported objects.

Cons - Verbose – INSERT Statements instead of raw data

Data is scripted with a fixed 100 row batch size.Edit “GO” statements between small batches

Round-trip EfficiencyUse “SET NOCOUNT ON”

18

Generate Script Wizard - DEMO

Migrating Databases to SQL Azure

19

SQL Azure Migration Wizard

http://sqlazuremw.codeplex.com

20

Migration Assist for MySql / Access (CTP)

Scenario - Auto porting of schema, database code and data from MySql and Access to SQL Azure

SQL Server Migration Assistant for MySql and Access

Supports MySQL 4.1 and upSupport Access v 97 and upSQL Server versions supported (all editions)

SQL Azure, SQL Server 2005, SQL Server 2008 and 2008 R2

21

MySQL to SQL Azure - DEMO

SQL Server Migration Assistant Version for MySQLVersion for Access

22

MySQL Migration Tool

23

DAC (Data-tier Application) PackageDACPac = Single unit for authoring, deploying, and managing data-tier objects (schema)

Development Lifecycle (VS 2010)Creating / Editing DACs

Schema and DB Code DevelopmentCode Analyses, Deployment Policy SettingsSchema Comparison and more…

Management Lifecycle (SSMS 2008 R2)Extracting DACs (as files)Managing DACs (deploy or upgrade)Registering existing database as DACs

If ‘regular’ into MSDBIf ‘Azure’ into masterIf using UCP into UCP database

24

DACs - DEMO

Migrating Databases to SQL Azure

25

Transferring Large Data to SQL Azure

Best Practices for efficient Data movement to SQL Azure

Use the right tools is key BCPSSIS

Network performance considerationsLatency vs Bandwidth

26

BCP & Bulk Copy APIs for Large Data Transfer

Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checking

-N Native mode so no data type conversion needed.

-c Character mode for interoperability-b batchsize will let you control transaction size-h”TABLOCK, ORDER(…)” optimize writes

27

Import/Export Wizard – for data transfer

Scenario - Simplified wizard for migrating schema and data through SSISPros

Great performance out of the boxAllows ‘save as package’ for full control

28

SSIS Data Flow Task (Large Data Xfer)Diverse Source and destinations - To/From Flat Files, ADO.Net, OleDBFully programmable flow

Loops, Sorts, Conditional operators, XML/WebServices Processing etcVS Debugging support with data viewers, watches and conditional breakpoints

Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checkingBatch Size: Adjust the transaction sizeBuffer and Blob temp storage area; spread over to multiple drives for large data transfersParallelization – based on execution trees, task will auto parallelize

29

SSIS package - DEMO

Use supported features only, such as bcp, but not BULK INSERT, etc…

30

Parallelization & Network PerfLatency & Bandwidth for high performance import & export

Latency is about laws of physicsLong distances means high latencyParallelization can help mitigate some latency

Bandwidth is key! Once you reach the bandwidth limit parallelization won’t help

See also COS13-INT: Database Performance in a Multi-tenant Environment

31

Datacenter Locations

32

BCP Performance – test set up example

CREATE TABLE [dbo].[table1]( [c01] [int] NOT NULL, [c02] [int] NOT NULL, [c03] [smallint] NOT NULL, [c04] [int] NOT NULL, [c05] [smallint] NOT NULL, [c06] [smallint] NOT NULL, [c07] [char](50) NOT NULL, [c08] [char](24) NOT NULL, … [c17] [char](24) NOT NULL, PRIMARY KEY CLUSTERED ([c01] ASC,[c02] ASC))

CREATE NONCLUSTERED INDEX [index1] ON [dbo].[table1] ([c07] ASC)

Data Distribution:Row Size: 600BytesRow Count: 100KPages: ~7700Total Size: ~60MBEven Distribution on PkeyConstraints and Indexes

Compound Pkey (int+int) None-clustered key on char(50)No triggers or relationships

33

BCP Upload Perf - Rows/Sec

Low Bandwidth, Medium Latency - 114ms

Low Bandwidth, Very High Latency - 289ms

High Bandwidth, High Latency - 169ms

High Bandwidth, Medium Latency - 54ms

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

Parallel Upload Batch Size 1K (sum rows/sec)

Native Sorted - Batch 10K (rows/sec)

Native - Batch 10K (rows/sec)

Native - Batch 100 (rows/sec)

34

BCP Download Perf – Rows/sec

Medium Bandwidth, Medium Latency - 114ms

Medium Bandwidth, Very High Latency - 289ms

High Bandwidth, High Latency - 169ms

High Bandwidth, Medium Latency - 54ms

0 2000 4000 6000 8000 10000 12000 14000 16000

4x Native Sorted - Batch Size 1K (sum rows/sec)

1x Native Sorted - Batch 10K (rows/sec)

35

Guidance on Data Transfer to SQL Azure

Split data transfer into multiple concurrent streamsTune BCP batch size and network packet size option Add non clustered indexes after loading data to SQL Azure.

Two additional indexes created before loading the data increased the final database size by ~50% and increased the time to load the same data by ~170%.

If, while building large indexes, you see a throttling-related error message, retry using the online option.

36

Monitoring Tasks• Space Used

• Monitor space used via ‘sys.dm_db_partition_stats’

• Trap ‘out of space exception’ – SQL Error 40544

• Synchronized Copies• Database copy• SQL Azure Synchronization

Provider•Other monitoring tasks –

use supplied DMVs (list in notes for this slide

37

Database Migration Tools

Tools SSMA Generate Script Wizard

DACs SSIS BCP

MySql and Access to SQL Azure

√ (Schema and Data)

√ (Data Only)

Move SQL Schema √ √ √

Move Data √ √ √

Move Large Data √ √

38

Future for Migration More information on all upcoming features at

DAT209: What’s New in Microsoft SQL AzureMigrating Applications

ConnectivityOleDb Support

SQL Server ParityDB Collation Support

OData Support – SQL Azure LabsFurther reach for SQL Azure data

39

Related ContentBreakout Sessions

AZP301 – A Lap Around the Azure PlatformAZP302 – Using Visual Studio 2010 to Build Applications that run on AzureDAT207 – What’s New in Microsoft SQL AzureAZP303 – Migrating Applications to Microsoft SQL Azure

40

Track Resources

Team Blog – http://blogs.msdn.com/sqlazure

MSDN Dev Center – http://msdn.microsoft.com/azure/sqlazure

Follow us on Twitter - @sqlazure

Windows Azure Platform Training Kit - http://bit.ly/dgvMv0

SQL Azure Labs – http://www.sqlazurelabs.com

41

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Complete an evaluation via CommNet and Tag to win amazing prizes!

43

TeachingKidsProgramming.org

• Do a Recipe Teach a kid• Free Courseware (recipes)• Ages 10 ++• Microsoft SmallBasic

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any

information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.