shopping around for query optimizations? try the query store!
TRANSCRIPT
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
SQL Server 2016 The Query StoreAuthor: Sean WerickCompany: Pragmatic Works
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Who is Pragmatic Works? o 100% Microsoft SQL Server Focused o Team of Microsoft MVPs, Authors, Presenterso Concentrated on Optimizing your Data Lifecycle
What is Pragmatic Works? o Software Solutions o Training Offeringso Professional Services
Why Pragmatic Works? o Microsoft Gold Partner – Data Platform & Business Intelligence o Microsoft’s only “National Solution Provider” for Business Intelligence o Microsoft Analytics Platform System Implementation Partner of the Yearo INC Magazine’s Fastest 5000 Growing Companies o Leader in offerings centered around Microsoft’s emerging technologies
About Pragmatic Works
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
• Principal Consultant• Community Speaker• Blogs: www.seanwerick.com• Email: [email protected]
About Sean
SQL Server 2016New Innovations
VISUALIZE + DECIDE
MobileReports
Natural languagequeryDashboardsApplications
Streaming
CAPTURE + MANAGE
RelationalInternal & external
Non-relational NoSQL
TRANSFORM + ANALYZE
OrchestrationMachine learningModeling
Information management
Complex event processingDat
a
The Microsoft data platform
Cloud firstSpeedAgilityProvenFeedback
All of this results in a better on-premises SQL Server
AnnouncingSQL Server 2016Deeper insights
across data
Hyperscale cloud
Do more. Achieve more.Hyperscale cloud
Deeper insights across data
0100101010110
SQL Server OLTP
SQL Serverdata
warehouse
ETL
In-memory ColumnStore
In-memory OLTP
Real-time frauddetection
In-memory enhancementsOperational analytics & enhanced performance
CapabilityIn-memory Columnar index over in-memory/disk based OLTP tablesEnhanced OLTP T-SQL surface areaScale to higher compute and memory
Benefits Unlike competition, you gain
operational analytics & 30x faster transactions & 100x queries
In-memory for more of your applications
Fraud detected
2-24hrs
dbo.Patients
Jane DoeName
243-24-9812
SSNUSACountry
Jim Gray 198-33-0987
USAJohn Smith 123-82-
1095USA
dbo.Patients
Jane DoeName
1x7fg655se2e
SSNUSA
Jim Gray 0x7ff654ae6d
USAJohn Smith 0y8fj754ea
2cUSA
Country
Result Set
Jim GrayName
Jane DoeName
1x7fg655se2e
SSNUSACountry
Jim Gray 0x7ff654ae6d
USAJohn Smith 0y8fj754ea
2cUSA
dbo.Patients
SQL Server
ciphertext
Query
Always EncryptedHelp protect data at rest and in motion, on-premises & cloud TrustedApps
SELECT Name FROM Patients WHERE SSN=@SSN@SSN='198-33-0987'
Result Set
Jim GrayName
SELECT Name FROM Patients WHERE SSN=@SSN
@SSN=0x7ff654ae6d
Column Encryption
Key
EnhancedADO.NET
Library
ColumnMasterKey
Client side
Mission critical performancePerformance
Operational analytics
In-memory OLTP for more applications
Query data store
Native JSON Temporal database support
SecurityAlways Encrypted
Row Level Security
Dynamic Data Masking
AvailabilityEnhanced AlwaysOn• 3 synchronous replicas for
auto failover across domains
• Round robin load balancing of replicas
• DTC for transactional integrity across database instances with AlwaysOn
Enhanced online operations
ScalabilityEnhanced database caching
Support for Windows Server 2016• 12 TB memory support
PolyBaseQuery relational and non-relational data with T-SQL
T-SQL query
CapabilityT-SQL for querying relational and non-relational data across SQL Server and Hadoop
Benefits New business insights across
your data lake Leverage existing skillsets
and BI tools Faster time to insights and
simplified ETL process
SQL Server
Hadoop
Quote:****************************************************************************************************************
$658.39
Jim GrayName
11/13/58
DOBWAState
Ann Smith
04/29/76
ME
Data ScientistInteract directly with data
Built-in to SQL Server
Data Developer/DBAManage data and analytics together
Built-in advanced analyticsIn-database analytics at massive scale
Example Solutions• Sales forecasting• Warehouse
efficiency• Predictive
maintenance
Relational Data
Analytic Library
T-SQL Interface
Extensibility
?RR
Integration
010010
100100
010101
Microsoft Azure Marketplace
New R scripts01001
010010
001010
1
010010
100100
010101
010010
100100
010101
010010
100100
010101
010010
100100
010101
• Credit risk protection
SQL Server Report Builder
Vision for SQL Server BI and Power BI SQL Server 2016+
PIN REPORTS
SQL Server BI “Reporting Services”• Power BI Desktop• Mobile reports• Paginated reports
Power BI• Power BI Desktop• Microsoft Excel• Mobile reports• Paginated reports
Power BI Web Portal
Power BI Desktop
POWER BI REPORTS
CONSUMEMANAGECREATECLOUDANALYTICAL REPORTS
Microsoft Excel
MOBILE REPORTS
“Datazen” Publisher**
PAGINATED REPORTS
Report Designer inSQL Server Data Tools
ON-PREMISES “REPORT SERVER”
OPTIONALLY EMBED
ONE SET OF MOBILE APPS
Windows App
Android App
iOS App
Report Server Web portal(all BI report types)
SharePoint We b
**Datazen Publisher will become part of Power BI Desktop
Deeper insights across dataAccess any data
PolyBase
Power Query for analytics and reporting
Enhanced SSIS• Designer support for previous
SSIS versions• Support for Power Query
Scale and manageEnterprise-grade Analysis Services
New single SSDT in Visual Studio 2015
Enhanced MDS • Excel add-in 15x faster• More granular security roles• Archival options for transaction logs• Reuse entities across models
Powerful insightsBuilt-in advanced analytics
Business insights through rich visualizations on mobile devices
Enhanced Reporting Services
Order history
Name SSN DateJane Doe cm61ba906
fd2/28/20
05Jim Gray ox7ff654ae
6d3/18/20
05John Smith i2y36cg776
rg4/10/20
05Bill Brown nx290pldo9
0l4/27/20
05Sue Daniels ypo85ba616
rj5/12/20
05Sarah Jones bns51ra806
fd5/22/20
05Jake Marks mci12hh906
fj6/07/20
05Eric Mears utb76b916g
i6/18/20
14Rachel Hogan px61hi9306
fj7/1/201
4Sam Johnson ol43bi506gd 7/12/20
14David Simon tx83hal916
fi7/29/20
14Michelle Burns nb95re926g
i8/10/20
14Reed Dean vc61ira536f
e8/23/20
14
Order history Name SSN DateJane Doe cm61ba906
fd2/28/20
05Jim Gray ox7ff654ae
6d3/18/20
05John Smith i2y36cg776r
g4/10/20
05Bill Brown nx290pldo9
0l4/27/20
05
Customer data
Product data
Order History
Stretch to cloud
Stretch SQL Server into AzureStretch warm and cold tables to Azure with remote query processing
CapabilityStretch cold database tables from on-premises SQL Server Databases to Azure with remote query processing
Benefits Cost effective historical data Entire table is online and
remains queryable from on-premises apps
Transparent to applications Supports Always Encrypted &
Row Level SecuritySQL Server App
Query
Microsoft Azure
Jim Gray ox7ff654ae6d
3/18/2005
Hyperscale cloudHybrid solutionsStretch tables into Azure
Power BI with on-premises data
Hybrid scenarios with SSIS
Enhanced backup to Azure
SimplicityEasy migration of on-premises SQL Server
Simplified Add Azure Replica Wizard
ConsistencyCommon development, management and identity tools
Consistent experience from on-premises to Azure
Business ScenariosRecommendations,
customer churn,forecasting, etc.
Perceptual IntelligenceFace, vision
Speech, text
Personal Digital Assistant
Cortana
Dashboards and Visualizations
Power BI
Machine Learning
and Analytics
Azure Machine Learning
Azure HDInsight (Hadoop)
Azure Stream Analytics
Cortana Analytics SuiteTransform data into intelligent action
DATA
Business apps
Custom apps
Sensors and devices
INTELLIGENCE ACTION
People
Automated Systems
Big Data Stores
Azure Data Lake
Azure SQL Data Warehouse
Information Management
Azure Data Factory
Azure Data Catalog
Azure Event Hub
Introducing Azure SQL Data Warehouse
Integration with Power BI, ADF, and Machine Learning services
Separate storage & compute
Elastic scale
Scale-out relational data warehouse
MPP massive sca
le-
out
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
What is the Query Store?A SQL Server 2016 (and SQL DB) Feature that can be Enabled
• Provides insight on query plan choice and performance
• Simplifies performance troubleshooting by quickly finding performance differences caused by query plan changes
• Automatically captures a history of queries, plans, and runtime statistics, and retains these for review
• Separates data by time windows to better illustrate database usage patterns and understand when query plan changes happened on the server
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Why Use the Query Store?Query Performance Insight Helps Satisfy 2 Main Goals
Performance Cost
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
WHAT IS THE REAL COST?
Why Use the Query Store?
All about the DTUs (PaaS)
A Database Transaction Unit (DTU) is a blended measure of the resources allocated at each performance level.
• Processor - % Processor Time• Logical Disk - Disk Reads/sec• Logical Disk - Disk Writes/sec• Database - Log Bytes Flushed/sec
• Review the list of top resource-consuming queries
• Select an individual query to view its details
• Click Settings to customize how data is displayed or to show a different time period
• Open Index Advisor* (PaaS) and check if any recommendations are available
Performance Cost
* Index Advisor - https://azure.microsoft.com/en-us/documentation/articles/sql-database-index-advisor/
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
How the Query Store “Works”
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Does It Work with In-Memory OLTP?
• SQL Server Query Store allows you to monitor the performance of natively compiled code for workloads running in-memory OLTP
• Compile and runtime statistics are collected and exposed the same way as for disk-based workloads
• When you migrate to in-memory OLTP you can continue using Query Store views in SQL Server Management Studio as well as custom scripts you have developed for disk-based workloads before migration
• This saves your investment in learning Query Store technology and makes it generally usable for troubleshooting all type of workloads
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
How to Use the Query StoreIn Just 3 Easy Steps, You TOO can be a DBA!!
Okay, okay. This may take a bit more effort and knowledge.
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
There are 2 Ways to Enable Query Store
By Using the Query Store Page in SSMS
Setting up the Query Store
In Object Explorer, right-click a database, and then click Properties
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
By Using the Query Store Page in SSMS
Setting up the Query Store
In the Database Properties dialog box, select the Query Store pageSelect related attributes• General
o Operation Mode (Read or Read/Write)• Monitoring
o Data Flush Interval (Manual Entry in Minutes)o Statistics Collection (5 mins, 10 mins, 15 mins, 30 mins, 1 hr,
1 day)• Max Size
o Query Store Capture Mode (All or Auto)o Size Based Cleanup Mode (Off or Auto)o Stale Query Threshold (Manual Entry in Days)
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
By Using T-SQL Statements
• Use the ALTER DATABASE statement ALTER DATABASE AdventureWorks2012 SET QUERY_STORE = ON;Ability to
Setting up the Query Store
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
GUI Option T-SQL OptionOperation Mode (Requested) OPERATION_MODE
Data Flush Interval (Minutes) DATA_FLUSH_INTERVAL_SECONDS
Statistics Collection Interval INTERVAL_LENGTH_MINUTES
Max Size (MB) MAX_STORAGE_SIZE_MB
Query Store Capture Mode QUERY_CAPTURE_MODE
Size Based Cleanup Mode SIZE_BASED_CLEANUP_MODE
Stale Query Threshold (Days) CLEANUP_POLICY (STALE_QUERY_THRESHOLD_DAYS)
MAX_PLANS_PER_QUERY
Query Store Options with T-SQLALTER DATABASE [db_name]SET QUERY_STORE ( [Query Store options] );
-- ExampleALTER DATABASE AdventureWorksSET QUERY_STORE ( MAX_STORAGE_SIZE_MB = 250 );
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
sys.query_store_plansys.query_store_runtime_stats
sys.query_context_settings
sys.query_store_runtime_stats_interval
sys.query_store_query
sys.query_store_query_text
plan_id
runtime_stats_interval_id
query_id
context_settings_id
query_text_id
DATA MODEL
sys.database_query_store_options
What Information Does It Provide?
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
What Information Does It Provide?
*Additional Detail - https://msdn.microsoft.com/en-us/library/dn818146.aspx
Query Store View Description
sys.database_query_store_options Query Store options for this database
sys.query_context_settings Semantics affecting context settings associated with a query (The same query text compiled under different settings may produce different results [depending on the underlying data])
sys.query_store_plan Each execution plan associated with a query
sys.query_store_query Information about the query and its associated overall aggregated runtime execution statistics
sys.query_store_query_text Transact-SQL text and the SQL handle of the query
sys.query_store_runtime_stats Information about runtime execution statistics information for the query
sys.query_store_runtime_stats_interval Information about the start and end time of each interval over which runtime execution statistics information for a query has been collected
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store Best Practices Information and performance are generally a give and take, but there ARE some
best practices
Every single setting in the Query Store has associated costs and benefits
• Treat Query Store performance tuning like an agile project with Sprints
o Sprint 1 – Keep Operation Mode in READ/WRITE mode for information collection
This reduces write time, and subsequent overall database performance, for data collection as no reporting is occurring
o Sprint 2 – Change Operation Mode to READ mode for analysis and actual performance tuning
This it increases report performance because no writes are occurring. This also results in no “dirty reads” because of the asynchronous writes
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store Best Practices
• Flush the data at specific business case intervals. For example, if performance of the DB seems to be poor around end of the month due financial reporting, consider using the Agile methodology to complete one iteration, then flush the data before the next iteration.
• Keep the Max Size small enough to keep the reporting relatively snappy, however, keep enough data to capture all of the business needs
• Keep an accurate Cleanup Policy as the business and reporting solution evolves
• Collect statistics as frequently as possible to see exactly which execution plans are being executed. If the statistics are not updated for some time, you may be looking at an old execution plan because the statistics have changed
Information and performance are generally a give and take, and it ultimately relies on each and every business and its processes
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store ReportingAzure SQL DB• Cloud offering (PaaS)
• Learn and adapt dynamically with your app
• Scale performance on the fly, without app downtime
• Build multitenant apps with customer isolation and efficiency
• Work within your preferred development environments
• Help protect and secure your app data
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store ReportingThe top queries view opens and the top CPU consuming queries are listed
• The top line shows overall DTU% for the database, while the bars show CPU% consumed by the selected queries during the selected interval
• The bottom grid represents aggregated information for the visible queries
o Average CPU per query during observable interval
o Total duration per query
o Total number of executions for a particular query
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store ReportingViewing Individual Query Details
Click any query in the list of top queries.
The details view opens and the queries CPU consumption is broken down over time.
Click around the chart for details.The top line is overall DTU%, and the bars are CPU% consumed by the selected query.
Review the data to see detailed metrics including duration, number of executions, and resource utilization percentage for each interval the query was running.
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store On Premise ReportingOn Premise Solution?
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store DashboardsSSMS
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Custom Views and SQLHOW?
sys.query_store_plansys.query_store_runtime_stats
sys.query_context_settings
sys.query_store_runtime_stats_interval
sys.query_store_query
sys.query_store_query_text
plan_id
runtime_stats_interval_id
query_id
context_settings_id
query_text_id
sys.database_query_store_options
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Query Store Admin with T-SQLStored Procedure Description
sp_query_store_flush_db Flushes the in-memory portion of the Query Store data to disk
sp_query_store_reset_exec_stats Clears the runtime stats for a specific query plan from the query store
sp_query_store_force_plan Enables forcing a particular plan for a particular query.
When a plan is forced for a particular query, every time SQL Server encounters the query, it tries to force the plan in the optimizer. If plan forcing fails, a XEvent is fired and the optimizer is instructed to optimize in the normal way.
sp_query_store_unforce_plan Enables unforcing a particular plan for a particular query
sp_query_store_remove_plan Removes a single plan from the query store
sp_query_store_remove_query Removes the query, as well as all associated plans and runtime stats from the query store
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Forcing Execution Plans
OLD WAY1. Run SET SHOWPLAN_XML ON to return the XML
Execution Plan, or alternatively, grab the XML Plan from the sys.dm_exec_text_query_plan DMF)
2. Create the Plan Guide using the sp_create_plan_guide Stored Procedure
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Forcing Execution Plans
NEW WAY1. Click the “Force Plan” button inside one of the
Query Store reports
2. Use the sp_query_store_force_plan SP.EXEC sp_query_store_force_plan 136, 146;
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Error StateRemember to verify intermittently that the Query Store is still collecting data
SELECT actual_state_desc, desired_state_desc, current_storage_size_mb, max_storage_size_mb, readonly_reason, interval_length_minutes, stale_query_threshold_days, size_based_cleanup_mode_desc, query_capture_mode_desc
FROM sys.database_query_store_options;
If there is a difference between the actual_state_desc and desired_state_desc, this indicates that a change of operations mode occurred automatically.
The most common change is for the Query Store to silently switch to read-only mode. In extremely rarely circumstances, Query Store can end up in the ERROR state because of internal errors.
When the actual state is read-only, use the readonly_reason column to determine the root cause. Typically, the Query Store transitioned to read-only mode because the size quota was exceeded. In that case the readonly_reason is set to 65536.
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Error StateHow to Fix
Reactive
Increase the maximum storage size by using the MAX_STORAGE_SIZE_MB option of ALTER DATABASE
Clean up Query Store data ALTER DATABASE [QueryStoreDB] SET QUERY_STORE CLEAR;
Change Operation Mode back to READ/WRITE ALTER DATABASE [QueryStoreDB] SET QUERY_STORE (OPERATION_MODE = READ_WRITE);
Proactive
Apply best practices, specifically related to Query Store size
Make sure that most recent data is retained, configure time-based policy to remove stale information regularly
Consider setting Query Capture Mode to Auto as it filters out queries that are usually less relevant for your workload
INTELLIGENT DATA SOLUTIONS WWW.PRAGMATICWORKS.COM
Questions and Answers