SQL Server Optimization for DevelopersANIL DESAI (ANIL@ANILDESAI.NET | HTTP://ANILDESAI.NET)AUSTIN .NET USER’S GROUP, 04/14/2014

Presentation Overview

Database Performance Goals and Challenges

Monitoring and Optimizing Performance Understanding indexes SQL Profiler and Database Engine Tuning Advisor

Tuning Database Queries Understanding the Query Optimizer and Execution Plans Seeing the effects of indexes

Application Design Best Practices

Best Practices: Optimize for real-

world workloads

Monitor/review performance regularly

Focus on specific issues

Performance Monitoring Process

Establish a baseline

Identify bottlenec

ks

Make one change at

a time

Measure performa

nce

Repeat (if

desired)

Monitoring and Troubleshooting Scenarios

Server-Level Issues

• Users are reporting slow performance• Intermittent transaction timeouts• “The server/application seem sluggish”

Application-Specific Issues

• “Application A is running more slowly than usual”• “The End-of-Month report is taking too long to

run.”

Other Issues• Ad-hoc reports are running slowly• CPU, memory, disk, or network alerts are being

generated

Which queries are taking the longest

to run?Which queries are

using the most system

resources?

Why does database/web site access slow down during particular times

or during certain patterns of activity?

Is indexing configured optimally for my live

(production) database workloads?

How will my development updates

affect production performance?

Which are the most cost-effective upgrades I can

make to improve performance?

How can I optimize a

specific query?

Common Datbaase Questions

Database Performance Tools

System/OS

Performance Monitor

Resource Monitor

Task Manager

SQL Server

SQL Server Management Studio

SQL Profiler / SQL Trace

Database Engine Tuning Advisor

Dynamic Management Views

(DMVs)

Query-Level

Database Engine Tuning

Advisor

Query Execution Plans

Database Design Issues

Transaction processing (OLTP) Favors normalized schema Many tables, each with fewer columns Optimized for write (transactional) activity

Reporting and Analysis Centralized, consistent storage of required data Favored by denormalized schema Fewer tables with many columns in each Data is aggregated from multiple sources into a data mart or data

warehouse May store aggregates in warehouse

Understanding Indexes

Index types Clustered Index Non-Clustered Indexes Columnstore indexes

Indexing strategies Goal is ideal index coverage Index maintenance can slow-down write operations (Insert, Update, Delete)

Referential Integrity Primary Key (default = clustered index) Foreign Key references Constraints

Statistics (manual vs. automatic)

General Index Tuning Best Practices

Make tuning a part of your development process Dev: Use synthetic workloads and test cases Test: Use real-world databases, whenever possible Production: Capture real usage statistics for analysis

Collect a representative workload, whenever possible Consider all applications and workloads when tuning a database Use naming conventions for indexes and related objects Use query hints sparingly (NOLOCK)

Purpose / Features: GUI for managing SQL Trace Monitor important events Capture performance data / resource usage Replaying of workloads / transactions Identifying performance bottlenecks Correlation of data with System Monitor Workloads for Database Tuning Advisor

Examples: Generate a list of the 100 slowest queries Monitor all failed logins (Security)

Using SQL Profiler

Database Engine Tuning Advisor

Automatic workload analysis for Physical Design Structures (PDS) Data Source:

File (Profiler Trace or .SQL files)

Table Plan Cache

Tuning Options Keep existing PDS Advanced Features:

Partitioning, indexed views, etc.

Demo: Optimizing Indexes

Generate sample queries / tables View query execution plans View the effects of indexes on common queries

Capture Performance Data with SQL Profiler SQL Profiler traces, events, and filters Using SQL Load Generator to generate database load Capturing and storing

Analyzing and optimizing with Database Tuning Advisor Analyzing index usage reports Saving and applying index recommendations

Tuning Individual Queries

Query Analyzer Features Execution Plan (estimated and actual) Include Client Statistics (multiple trials) Analyze in Database Engine Tuning Advisor (single query) Trace query in SQL Profiler (single query)

Keep query logic close to the database Filter returned data at the database layer Minimize the size of result sets Minimize round-trips to the server

Use standard (inner) joins, where possible Consider strategic denormalization for core sets of data

Query Optimizer Details

Goal: Find the most efficient method to return the data Come up with a plan quickly Minimize CPU, memory, and I/O requirements Use statistics and index details to improve plans

Query plan caching Relational engine vs. storage engine Execution Plan output

Save as .sqlplan file for later analysis Output in graphical, text, and XML formats Can store and export plans using SQL Profiler (ShowPlan XML event)

Can use query hints

Understanding Execution Plans

Optimizing individual queries Rewrite query logic Use other objects (views, stored, procedures, etc.) Strategic demoralization

Data Retrieval: Table scan, index seek/scan Index Usage

Covering indexes Join conditions

Execution Plan Example

Execution Plans in Windows Azure

Client Statistics Example

Application Design Best Practices

Create an abstraction layer between business and database objects ADO.NET Microsoft Enterprise Library NHibernate Entity Framework

Use caching wherever possible Server-side (web services) Application-level (middle tier) Client-side (desktop or mobile apps)

Minimize transaction times

Dev Best Practices: Application and Data Architecture

•Standards-based communications•Loosely-coupled tiers and componentsArchitecture

•Test harness•Unit tests that use data•Query performance monitoring

Development / Testing

•Windows PerfMon Counters•Instrumentation and Logging•Alerts / pro-active corrections•Auto-scaling

Performance and Monitoring

•Error and event handling•Transaction retry (random interval); Deadlock management•User notifications (responsive UI / cancel and retry options)Reliability

Windows Azure and Cloud Databases

Practical cloud benefits Data redundancy and geographic distribution Lower management overhead

Potential issues Keeping data close to applications and services Data synchronization Network performance issues Data security, legal issues, and regulatory compliance

Determine where/how to use cloud-based services SaaS vs. PaaS vs. IaaS

Azure Database Services

SQL Azure Database Cost-effective, managed database instances Can be managed with standard tools (Visual Studio and SSMS) Some limitations (CLR, Mirroring, Partitioning, Replication, Extended SP’s)

Other Services Azure Virtual Machines (SQL Server templates) Azure Web Sites (with gallery templates) Azure HDInsight, Cache Service Azure Backup and Recovery Manager SQL Azure Reporting Network, Active Directory, Service Bus, etc.

Managing SQL Azure Instances

ORM Considerations

General issues Development efficiency vs. hardware/software efficiency Latency, query inefficiency (outer joins), platform-specific optimizations Frequency and number of server round-trips

ORM-generated queries can be inefficient Difficult to tune or modify individual queries

Potential Solutions Make sure entity relationships are correct Can use views or stored procedures to improve performance in some

cases Bypass the ORM for some types of operations

New Features in SQL Server 2014

Memory-optimized tables (In-Memory OLTP) Buffer Pool Extension (for SSD usage) Delayed durability

Async log writes can result in data loss Enable at database-level; use with BEGIN ATOMIC … COMMIT

Resource Governor storage I/O limits Updateable Clustered ColumnStore indexes

Primarily for data warehousing; supports data index compression Azure storage for SQL Server data/log files Backup to Azure; Backup encryption

Dev Best Practices: Managing Data

Large UPDATE or DELETE operations: Use loops to minimize locking and transaction log growth

Large INSERT operations Disable indexes and triggers (if present) Use BULK INSERT, bcp, SSIS, or DTS Change transaction isolation level (if appropriate) Change recovery model

Use SQL to generate SQL Example: INSERT statements

Schedule or delay non-critical operations

Dev Best Practices: Schema Changes

Generate Scripts Script specific objects using SQL Server Management Studio Script the entire database using Generate Scripts

Can include schema and/or data Schema changes

Use ALTER commands when possible Drop and recreate objects, as needed Make all scripts re-runnable Check before and after state of all objects

Dev Best Practices: Performance Testing

Build performance testing/optimization into the dev process Develop load tests or test “harnesses” Using synthetic load generation tools Use representative test data

Consider caching effects: Index maintenance (fragmentation) DBCC DropCleanBuffers DBCC FreeProcCache

Advanced Performance Approaches

Database Federations Vertical and horizontal data partitioning

Cross-Server queries Use Linked Servers to query across databases Potential performance issues

Data compression (row- or page-level) Resource governor SQL Server Analysis Services (SSAS)

Pre-aggregation for performance Dependent on a denormalized schema (optimized for reporting)

Links and References

Presenter: http://AnilDesai.net | Anil@AnilDesai.net Presentation slides and sample code

Microsoft TechNet Virtual Labs Sample Databases

AdventureWorks Sample Databases (CodePlex) Microsoft Contoso BI Demo Dataset

Database-related tools SQL Load Generator by David Darden (CodePlex) Glimpse Red Gate Software Spotlight

Summary and Conclusion