optimising queries - series 1 query optimiser architecture
TRANSCRIPT
![Page 1: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/1.jpg)
www.dageop.com
Optimising Queries
®
OQ-01 Query Optimiser Architecture
DR. SUBRAMANI PARAMASIVAM
(MANI)
![Page 2: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/2.jpg)
About me
Dr. SubraMANI ParamasivamPhD., MCT, MCSE, MCITP, MCP, MCTS, MCSACEO, Principal Consultant & Trainer @ DAGEOP (UK) Email: [email protected] Blog: http://dataap.org/blog
Follow Us
https://www.facebook.com/pages/YOUR-SQL-MAN-LTD/
http://www.youtube.com/user/YourSQLMAN
https://twitter.com/dageop
https://uk.linkedin.com/in/dageop
Proud Sponsor
• SQLBits• SQL Saturdays• MCT Summit• SQL Server Geeks
Summit• Data Awareness
Programme• Dageop’s Data Day
®
www.DataAP.org
SPEAKER
![Page 3: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/3.jpg)
Contents
• OQ-01 Query Optimiser Architecture• Phases
• Strategies
• Data access plans
• Auto-parameterisation
• Avoiding recompilation of queries
www.dageop.comOptimizing Queries
![Page 6: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/6.jpg)
Optimising Queries
• Optimising Query is very important task to maintain the server resources.
• Most of the time server is hit by performance issues.
• Clean the SQL Server Cache.
• Well written T-SQL Queries will help to optimise.
www.dageop.comOptimizing Queries
![Page 8: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/8.jpg)
Introduction to Query Optimiser Architecture
2 Major Components in SQL Server• SQL Server Database Engine
• Reading data between disk and memory
•Relational Engine (Query Processor)• Accepts queries, analyse & executes the plan.
www.dageop.comOptimizing Queries
![Page 9: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/9.jpg)
Introduction to Query Optimiser Architecture
• Analyse Execution Plans (Cannot consider every plan)
• Estimate the cost of each plan
• Select the plan with low cost
www.dageop.comOptimizing Queries
Plan1
Plan2
Plan3
Plan4
Plan5
Plan6
Plan7
Plan1 – 80%
Plan2 – 60%
Plan3 – 10%
Plan5 – 90%
Plan6 – 70%
Plan3 – 10%
Analyse Estimate Select
COST X 2Plan & itself
![Page 10: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/10.jpg)
BETTER UNDERSTANDING of Query Optimiser is a must
for both DBA & DEVELOPERS.
Main Challenges are
Cardinality & Cost Estimations
www.dageop.comOptimizing Queries
![Page 11: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/11.jpg)
Query Optimiser
• 1st Part - Searching or enumerating candidate plans
• 2nd Part - Estimates the cost of each physical operator in that plan (I/O, CPU, and memory)
• This cost estimation depends mostly on the algorithm used by the physical operator, as well as the estimated number of records that will need to be processed (Cardinality Estimation).
PRIMARY WAY to interact with Query Optimizer is through EXECUTION PLANS
www.dageop.comOptimizing Queries
![Page 12: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/12.jpg)
Query Optimiser
• Execution Plan• Trees consisting of a number of physical operators
• Physical operators• Index Scan
• Hash Aggregate
• result operator (root element in the plan)
• Nested Loops Join
• Merge Join
• Hash Join
• Algorithms
• Can be saved as .sqlplan (Can be viewed in SSMS)
www.dageop.comOptimizing Queries
![Page 13: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/13.jpg)
Query Optimiser• Operators
• Logical• Relational algebraic operation• Conceptually describes what operation needs to be performed
• Physical• Implement the operation described by logical operators• Access columns, rows, index, table, views, calculations, aggregations, data integrity
checks • 3 methods
• Init() – Initializes a connection• GetNext() – 0 to many calls to get data• Close() – Some cleanup and close the connection.
• Query Optimizer chooses efficient physical operator based on logical.
www.dageop.comOptimizing Queries
![Page 14: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/14.jpg)
Query Optimiser
www.dageop.comOptimizing Queries
Logical Physical Logical & PhysicalBitmap Create Assert AggregateBranch Repartition Bitmap Clustered Index ScanCache Clustered Index Delete Clustered Index ScanDistinct Clustered Index Insert Clustered Index UpdateDistinct Sort Clustered Index Merge CollapseDistribute Streams Hash Match Compute ScalarEager Spool Merge Join ConcatenationFlow Distinct Nested Loops Cursor
Full Outer Join Nonclustered Index Delete Inserted ScanGather Streams Index Insert Log Row ScanInner Join Index Spool Merge IntervalInsert Nonclustered Index Update Index ScanLazy Spool Online Index Insert Index SeekLeft Anti Semi Join Parallelism Parameter Table ScanLeft Outer Join RID Lookup Remote DeleteLeft Semi Join Stream Aggregate Remote Index ScanPartial Aggregate Table Delete Remote Index SeekRepartition Streams Table Insert Remote InsertRight Anti Semi Join Table Merge Remote QueryRight Outer Join Table Spool Remote ScanRight Semi Join Table Update Remote UpdateRow Count Spool SegmentSegment Repartition SequenceUnion Sequence ProjectUpdate Sort
SplitSwitchTable ScanTable-valued FunctionTopWindow Spool
![Page 15: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/15.jpg)
Query Optimiser
• Invalid Plans• Removal of an index
• Removal of a constraint,
• Significant changes made to the contents of the database
• SQL Server under memory pressure
• Changing configuration options - max degree of parallelism
• Discard the plan cache and new optimization generated
www.dageop.comOptimizing Queries
![Page 16: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/16.jpg)
Query Optimiser
• Highly complex pieces of software
• Even 30 years of research, still has technical challenges.
HINTS
• override the operations of the Query Optimizer
• caution and only as a last option
• Influence Query Optimizer to use certain plans
www.dageop.comOptimizing Queries
![Page 17: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/17.jpg)
Query Optimiser - Interaction
www.dageop.comOptimizing Queries
Query Optimizer
Execution Plans
Trees, Algorithm, Physical operator
ACTUAL ESTIMATED
Graphics, Text, XML format
![Page 20: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/20.jpg)
Phases
• Once we execute the SQL Statement, it will follow certain procedures from SQL Statement to Query Result
www.dageop.comOptimizing Queries
Parsing Query Compilation
Query Optimization
Query Execution
![Page 21: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/21.jpg)
Phases
• If the Query is already in the plan cache then it will not generate any new execution plan for that query.
• Parsing:• The Query’s syntax will be validated and query is transformed in a tree.
Checks the objects for its existence which are used in the query.
• After the query validated, the final tree is formed.
• Query Compilation:• Query Tree will be compiled here.
www.dageop.comOptimizing Queries
![Page 22: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/22.jpg)
Phases
• Query Optimization• The query optimizer takes as input the compiled query tree generated in the
previous step and investigates several access strategies before it decides how to process the given query.
• It will analyse the query to find most efficient execution plan.
• Query Execution• After the execution plan is generated, it is permanently stored and executed• Note: For Some statements, parsing and optimization can be avoided if the
database engines know that there is only one viable plan. This is called trivial plan optimization.
www.dageop.comOptimizing Queries
![Page 24: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/24.jpg)
Strategies
• As a DBA, we need to choose right strategy for each SQL statement like scripts should be well written before executing the SQL statement.
• Statistics will be used to understand the performance of each query.
• Enable Set Statistics IO before query run. It will display the following information• How many scans were performed
• How many logical reads (reads in cache) were performed
• How many physical reads (reads on disk) were performed
• How many pages were placed in the cache in anticipation of future reads (read-ahead reads)
www.dageop.comOptimizing Queries
![Page 25: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/25.jpg)
Strategies
• We can understand the query performance from statistics IO, If the query is good then the logical reads of the query result should be lower and few physical reads and scans.
• Enable Set Statistics Time, it will display the execution time of the query. It purely depends on the total activity of the server.
• Enable show execution plan, to see how the query performed.
• These are things will helpful for choosing the right strategy.
www.dageop.comOptimizing Queries
![Page 26: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/26.jpg)
www.dageop.comOptimizing Queries
StrategiesChoose the right strategies based on below information from SQL Server
![Page 29: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/29.jpg)
Data Access Plans
• If we need to optimize the data access plans then we need to start from creating files for database, Index on the tables and T SQL Statements.
• Steps:• Organize the file groups and files
• Apply partitioning in big tables
• Create the appropriate indexes/covering indexes
• Defragment the indexes
• Identify inefficient TSQL
• Diagnose performance problems
www.dageop.comOptimizing Queries
![Page 30: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/30.jpg)
Data Access Plans
• Organize the file groups and files• Initially two files will be created while created a database (.mdf & .ldf). • .mdf file : Primary data file for each database. All system objects will be stored in this
file, including the user defined objects if .ndf file is not there.• .ndf file: These are secondary data files, these are optional. These files will have user
created objects.• .ldf file: These are the transaction log files. This could be one or more files for single
database.
• File Group:• Database files are logically grouped for better performance and improved
administration on large databases. When a new SQL Server database is created, the primary file group is created and the primary data file is included in the primary file group. Also, the primary group is marked as the default group.
www.dageop.comOptimizing Queries
![Page 31: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/31.jpg)
Data Access Plans
• To obtain the performance of the data access, Primary file group must be separate and it should be only for system objects.
• Need to create one more file called secondary data file for user defined objects.
• Separating the system objects will improve the performance enhance the ability to access tables in cases of serious data failures.
• For frequently accessed tables containing indexes, put the tables and the indexes in separate file groups. This would enable reading the index and table data faster
www.dageop.comOptimizing Queries
![Page 32: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/32.jpg)
Data Access Plans
• Apply partitioning in big tables• Table partitioning is nothing but splitting the large table into multiple small
tables so that queries have to scan less amount for data retrieving.
• Consider partitioning big fat tables into different file groups where each file inside the file group is spread into separate physical disks (so that the table spans across different files in different physical disks). This would enable the database engine to read/write data operations faster.
• Partitioning is very much needed for history tables.
• 2 types• Physical
• Logical
www.dageop.comOptimizing Queries
![Page 33: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/33.jpg)
Data Access Plans
• Create the appropriate indexes/covering indexes
• Create Non-Clustered index on frequently used columns
• Create index on column which is used for joining the tables
• Index for foreign key columns
• Create covering index for particular columns which are using frequently.
• Defragment the indexes
• Once index has created , it should maintain properly to avoid defragmentation.
• Maintaining the index will lead to performance gain.
www.dageop.comOptimizing Queries
![Page 34: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/34.jpg)
Data Access Plans
• Identify inefficient TSQL• Don’t use * in the select statements. Mention column names while retrieving
the data. It will improve the performance of the query
• Avoid Deadlocks between two objects.
• Diagnose the performance issue• SQL Server has many tools to Monitor and diagnose the issues.
• Accessing the data will be more easier.
www.dageop.comOptimizing Queries
![Page 37: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/37.jpg)
Auto-Parameterisation
• SQL Server Query Optimizer might decide to parameterize some of the queries.
• In this case the specific parameter will not make any impact on the execution plan. It will return the same execution plan.
• In SQL Server 2005 forced parameterization has been introduced and is disabled by default and can be enabled in database level.
• To differentiate from forced parameterization, auto-parameterization is also referred as simple parameterization.
www.dageop.comOptimizing Queries
![Page 40: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/40.jpg)
Avoiding Recompilation of Queries
• In SQL Server 2005 recompiles the stored procedures, only the statement that causes recompilation is compiled, rather than the entire procedure.
• Recompilation will occur in following ways,• On Schema change of objects
• On Change of the SET options
• On statistics change of tables.
www.dageop.comOptimizing Queries
![Page 41: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/41.jpg)
Avoiding Recompilation of Queries
• On Schema Change of Objects• Adding and dropping column, constraints, index, indexed view and trigger.
• On change of the SET options• When executing the stored procedure, the compiled plan is created and it will store
the environment setting of a connection (SET OPTION) .• Recompilation will occur, if the stored procedure run on different environment and
with different SET option then it will not use the existing plan which it is created first time.
• On statistics change of tables• SQL server maintains a modification counter for each table and index.• If the counter values exceed the defined threshold, the previously create compiled
plans is considered stale plan and new plan will be created.
www.dageop.comOptimizing Queries
![Page 42: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/42.jpg)
Avoiding Recompilation of Queries
• Temporary table modification counter threshold is 6. Stored procedure will be recompiled when stored procedure create a temp table insert 6 or more rows into this table.
• For Permanent table the counter threshold is 500.
• We can increase the temp table counter threshold to 500 as same as permanent table.
• Use table variable instead of Temporary table.
www.dageop.comOptimizing Queries
![Page 44: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/44.jpg)
Review
Query Optimiser ArchitecturePhases
Strategies
Data access plans
Auto-parameterisation
Avoiding recompilation of queries
www.dageop.comOptimizing Queries
![Page 46: Optimising Queries - Series 1 Query Optimiser Architecture](https://reader034.vdocuments.site/reader034/viewer/2022042619/5a6cece37f8b9a04428b469f/html5/thumbnails/46.jpg)
®
www.dageop.com