building scalable application with sql server
DESCRIPTION
TRANSCRIPT
![Page 1: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/1.jpg)
Building Scalable Applications With
SQL ServerChris Adkin
![Page 2: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/2.jpg)
15 years of IT experience 14 years of DBA experience
across a variety of vertical sectors.
Over 10 years experience of SQL Server from 2000.
Contact◦ Email:
[email protected]◦ Twitter: ChrisAdkin8
About me
![Page 3: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/3.jpg)
The following material contains general rules of thumb.
Always test your assumptions.There will always be edge cases in which conventional wisdom may not apply.
Disclaimer
![Page 4: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/4.jpg)
Characteristic OLTP OLAP
I/O Random Serial
Transactions Short Longer during ETL
Execution plans Simple Complex
Plan nature Serial with index seeks
Parallel with index/table /partition scans
SQL Complexity Simpler Complex, often generated by BI tools
Number of queries Finite Highly variable
(re)compiles impact on performance
Critical Execution plan run time more important
SQL Selectivity Queries typically only affect a few rows
Queries aggregate many rows and affect many rows during ETL
OLAP Compared To OLTP
![Page 5: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/5.jpg)
Where the foundation for scalability is laid.
Consider your most performance critical queries and the number of joins they require.
#1 Schema Design
![Page 6: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/6.jpg)
Data model design “Usual suspects”:◦Over normalisation.◦1:1 relationships, where most / all the attributes from both entities are retrieved together.
◦Logical data model sprawl, as the model evolves over time.
#1 Schema Design
![Page 7: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/7.jpg)
1:1 relationships edge cases:◦Sparse columns, which cannot be compressed or used in clustered indexes.
◦Infrequently accessed columns with wide data types.
#1 Schema Design
![Page 8: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/8.jpg)
“Walking with keys” We only want columns from tables A and E and have to go via B, C and D in order to facilitate the join.
The exception is when entities are introduced to resolve many to many relationships.
#1 Schema Design
A EDCB
![Page 9: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/9.jpg)
Test drive your logical model design as early as possible in the project.
Failure to do so may result in performance issues built into the foundations of the application.
#1 Schema Design
![Page 10: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/10.jpg)
“Chattiness” => heavy network conversation volume between software components.
Chatter takes place via linked servers and /or between tiers.
More time is spent on the network than anywhere else.
#2 “Chatty” Designs
![Page 11: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/11.jpg)
Low IIS and SQL Server CPU utilisation is a symptom.
Cache static / relatively static data outside the database.
Put CRUD intensive operations in stored procedures.
Always put the processing closest to the resource that requires it.
#2 “Chatty” Designs
![Page 12: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/12.jpg)
There is a wider debate about where it is best to put the business logic.
This is beyond the scope of this presentation, however you may wish to look at:◦To sp Or Not sp by Adam Machanic◦Pet vsPet: MS opens .Net benchmarking wars
#2 “Chatty” Designs
![Page 13: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/13.jpg)
Start with a minimal set of indexes and then add.
Use filtered indexes for selective SQL. Indexing GUIDs:
◦Causes page splits and fragmentation◦GUIDS are large, 16 bytes◦Inflates the size of secondary indexes created on clustered indexes.
# 3 Indexing
![Page 14: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/14.jpg)
‘Scatter gun’ indexing strategies can lead to indexes that add little or no value.
Glenn Berry’s diagnostics script includes a query to highlight indexes with lots of writes and little or no reads.
# 3 Indexing
![Page 15: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/15.jpg)
For GUIDs and clustered indexes, refer to “The Clustered Index Debate” by SQL Skills.
If you really need GUIDs, consider:◦NEWSEQUENTIALID().◦SSD technology as blogged about by Tony Rogerson.
Otherwise avoid GUIDs in general.
# 3 Indexing
![Page 16: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/16.jpg)
Know your critical indexes and when to rebuild and re-organise them.
The maintenance plan index rebuild tasks rebuilds ALL indexes.
Use index fragmentation and index read intensity to determine rebuild / reorg order.
Some good free tools are available, the procedures from Ola Hallengren to name one.
# 4 Index Maintenance
![Page 17: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/17.jpg)
A compile takes place when a plan cannot be found for a statement in the plan cache.
Recompiles take place when the execution engine deems that a plan is no longer valid.
All are to be avoided where possible.
# 5 Compiles and Recompiles
![Page 18: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/18.jpg)
Factor Causes A Compile
Causes A Recompile
Use Of temp tables √
Changes in statistics √
DBCC FREEPROCCACHE / instance restart
√
Schema changes √
Use of the RECOMPILE hint √
Non parameterised SQL √
Dynamic SQL √
Plans aged out the plan cache √
Changes in SET options (Refer to Erland Sommarskog’s blog)
√
# 5 Compiles and Recompiles
![Page 19: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/19.jpg)
# 5 Compiles and Recompiles
There are two hints that help to minimise recompiles.
The KEEP PLAN hint causes the execution engine to treat temporary tables as permanent tables.
The KEEPFIXED PLAN hint will prevent recompiles if there are changes to the statistics of base tables used in a query.
Forced parameterisation can minimise compiles, but use it with caution.
![Page 20: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/20.jpg)
There are many causes of RBAR.Cursors.User defined functions, table level functions are the exception.
EXISTS sub queries.Recursive CTEs. . . . and many more, search on SQL Server Central for RBAR
#6 Row By Agonising Row (RBAR)
![Page 21: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/21.jpg)
Avoid RBAR wherever possible.Row by agonising row, as coined by Jeff Moden, is covered extensively in his articles on SQL Server Central.
#6 Row By Agonising Row
![Page 22: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/22.jpg)
Segregate different types of querying activity.
I/O best practise is to segregate random from sequential I/O.
Segregate OLAP/reporting queries from OLTP queries.
# 7 Segregate OLAP and OLTP Workloads
![Page 23: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/23.jpg)
ETL aside, OLTP applications are more write intensive than OLAP ones.
Optimisations for reads can decimate write performance:◦Heavy indexing◦Covering indexes◦Index views◦Column store indexes◦Aggregation
# 7 Segregate OLAP and OLTP Workloads
![Page 24: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/24.jpg)
The Command Query Responsibility Segregation model provides a solution.
Queries insert data into a database optimised for fast writes.
The same data is queued (service broker).
The data is de-queued and written to a database optimised for fast reads.
# 7 Segregate OLAP and OLTP Workloads
![Page 25: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/25.jpg)
An online retailer services thousands of concurrent customers.
A single table holds the balance for the retailer in a single row.
If all transactions can update the balance table at the same time, this creates a unique point of serialisation.
# 8 Unique Points Of Serialisation
![Page 26: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/26.jpg)
Queue the debits credits from the user transactions, two options.
Table with a clustered index, as per Remus Rasanu’s blog and DELETE OUTPUT destructive read.
Service broker is the 2nd option.De-queue values and update the table.
# 8 Unique Points Of Serialisation
![Page 27: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/27.jpg)
Prior to SQL Server 2005, unless dirty reads are used, readers block writers.
This throttles back OTLP through put. The solution; multi-version concurrency control.
Facilitated by two new isolation levels available in SQL Server 2005 onwards.
# 9 Locking
![Page 28: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/28.jpg)
Read committed snapshot◦Data is consistent with beginning of the statement.
◦An extension of the read committed isolation level.
Snapshot isolation◦Data is consistent with the beginning of transaction.
# 9 Locking
![Page 29: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/29.jpg)
Snapshot / RCSI isolation cannot be enabled for databases with filestream file groups.
Queries may have to traverse the row version store => more logical and physical reads.
This Microsoft paper provides guidance as to which isolation level to use.
# 9 Locking
![Page 30: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/30.jpg)
14 bytes of information is added to each row in the database as it is modified or created.
Snapshot isolation can lead to update conflicts, retry logic is required to work around this.
Ensure tempdb resides on fast disks, as this is where the version store lives.
# 9 Locking
![Page 31: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/31.jpg)
Latches protect the integrity of internal SQL Server memory structures.
Three latch wait types:◦Buffer (BUF) : PAGELATCH_* wait type◦Non buffer : LATCH_* wait type◦IO latch : PAGEIOLATCH wait type
# 10 Latching
![Page 32: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/32.jpg)
When multiple threads try to obtain the same latch in incompatible modes, contention takes place.
Some latch contention is to be expected.
Refer to this SQLCAT paper for detailed information on latch trouble shooting.
# 10 Latching
![Page 33: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/33.jpg)
If problems exist, long PAGELATCH_* and PAGEIOLATCH waits will be visible.
Common causes of latch contention include:◦Heavy CRUD activity affecting rows in the same page.
◦Solution is to pad rows out with char[X] columns.
# 10 Latching
![Page 34: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/34.jpg)
An index has a leading column populated by an identity, sequence etc.
Solution: hash partition the table on the indexes leading column.
# 10 Latching
![Page 35: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/35.jpg)
Diagnosing and Resolving Latching Issuesby SQL Customer Advisory Team
Designing High Scale OLTP Systems by Thomas Kejser
Lessons Learned From 128 Core OLTP Testing
by Thomas Kejser
Further Reading
![Page 36: Building scalable application with sql server](https://reader033.vdocuments.site/reader033/viewer/2022061209/5486e659b4af9fca3b8b4781/html5/thumbnails/36.jpg)
Questions ?