Christian Bolton, Justin Langford, Brent Ozar, James Rowland-Jones, Steven Wort

Wrox Programmer to Programmer™Join the discussion @ p2p.wrox.com

ProfessionalSQL Server® 2008Internals and Troubleshooting

$59.99 USA $71.99 CAN

When faced with a variety of SQL Server issues, you need to be able to troubleshoot SQL Server as well as its underlying operating system. To that end, this author group of SQL Server experts presents you with invaluable advice that brings together—and simplifies—practical internal information concerning SQL Server along with troubleshooting techniques and methodologies. Packed with real-world examples of ways to effectively use troubleshooting tools, this book shows you how to efficiently and accurately determine the root of your SQL Server problem so you can successfully resolve it.

• Escorts you through the lifecycle of a query to establish an under-standing of the fundamental concepts and architectures

• Examines the use of memory including the physical components, Windows memory management, and SQL Server’s own memory architecture

• Reviews SQL Server wait types and how to monitor them with Extended Events

• Equips you with the knowledge to confidently specify and monitor your storage performance requirements

• Details SQL Server’s locking model, the internal use of latches, and row versioning

• Shows you how to use SQLDiag, RML Utilities, SQL Nexus, the Management Data Warehouse, and how to build your own SQL Server Management Studio reports

Christian Bolton is coauthor of Professional SQL Server 2005 Performance Tuning and Technical Director for Coeo Ltd.

Justin Langford is coauthor of Professional SQL Server 2005 Performance Tuning and leads the Managed Support team for Coeo Ltd.

Brent Ozar is a SQL Server expert for Quest Software.

James Rowland-Jones is an advisory consultant for EMC Consulting.

Steven Wort is a developer in the Windows Division at Microsoft.

Wrox guides are crafted to make learning programming languages and technologies easier than you think. Written by programmers for programmers, they provide a structured, tutorial format that will guide you through all the techniques involved.

Database Management / General

An in-depth look at SQL Server and best practices for troubleshooting

wrox.comProgrammer ForumsJoin our Programmer to Programmer forums to ask and answer programming questions about this book, join discussions on the hottest topics in the industry, and connect with fellow programmers from around the world.

Code Downloads Take advantage of free code samples from this book, as well as code samples from hundreds of other books, all ready to use.

Read More Find articles, ebooks, sample chapters and tables of contents for hundreds of books, and more reference resources on programming topics that matter to you.

Bolton, Langford, et al.

ProfessionalSQ

L Server® 2008

Internals and Troubleshooting

Get more out of wrox.com

Programmer to Programmer™

InteractTake an active role online by participating in our P2P forums @ p2p.wrox.com

Wrox Online LibraryHundreds of our books are available online through Books24x7.com

Wrox Blox Download short informational pieces and code to keep you up to date and out of trouble!

Join the CommunitySign up for our free monthly newsletter at newsletter.wrox.com

BrowseReady for more Wrox? We have books and e-books available on .NET, SQL Server, Java, XML, Visual Basic, C#/ C++, and much more!

Contact Us. We always like to get feedback from our readers. Have a book idea? Need community support? Let us know by e-mailing wrox-partnerwithus@wrox.com

PROFESSIONAL

SQL SERVER® 2008 INTERNALS

AND TROUBLESHOOTING

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

CHAPTER 1 SQL Server Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

CHAPTER 2 Understanding Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

CHAPTER 3 SQL Server Waits and Extended Events . . . . . . . . . . . . . . . . . . . . . . . . . . 59

CHAPTER 4 Working with Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

CHAPTER 5 CPU and Query Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137

CHAPTER 6 Locking and Latches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

CHAPTER 7 Knowing Tempdb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

CHAPTER 8 Defi ning Your Approach To Troubleshooting . . . . . . . . . . . . . . . . . . . . . 305

CHAPTER 9 Viewing Server Performance with PerfMon and the PAL Tool . . . . . . 329

CHAPTER 10 Tracing SQL Server with SQL Trace and Profi ler . . . . . . . . . . . . . . . . . . 369

CHAPTER 11 Consolidating Data Collection with SQLDiag and the

PerfStats Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437

CHAPTER 12 Introducing RML Utilities for Stress Testing and Trace

File Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459

CHAPTER 13 Bringing It All Together with SQL Nexus . . . . . . . . . . . . . . . . . . . . . . . . . 481

CHAPTER 14 Using Management Studio Reports and the Performance

Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .505

CHAPTER 15 Using SQL Server Management Data Warehouse . . . . . . . . . . . . . . . . . 539

CHAPTER 16 Shortcuts to Effi cient Data Collection and Quick Analysis . . . . Wrox.com

INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555

84289ffirs.indd i84289ffirs.indd i 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

84289ffirs.indd ii84289ffirs.indd ii 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

PROFESSIONAL

SQL Server® 2008 Internals

and Troubleshooting

84289ffirs.indd iii84289ffirs.indd iii 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

84289ffirs.indd iv84289ffirs.indd iv 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

PROFESSIONAL

SQL Server® 2008 Internals

and Troubleshooting

Christian Bolton

Justin Langford

Brent Ozar

James Rowland-Jones

Jonathan Kehayias

Cindy Gross

Steven Wort

84289ffirs.indd v84289ffirs.indd v 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

Professional SQL Server® 2008 Internals and Troubleshooting

Published byWiley Publishing, Inc.10475 Crosspoint BoulevardIndianapolis, IN 46256www.wiley.com

Copyright © 2010 by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 978-0-470-48428-9

Manufactured in the United States of America

10 9 8 7 6 5 4 3 2 1

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifi cally disclaim all warranties, including without limitation warranties of fi tness for a particular purpose. No warranty may be created or extended by sales or pro-motional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the pub-lisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Web site may provide or recommendations it may make. Further, readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Control Number: 2009941346

Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Wrox Programmer to Programmer, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affi liates, in the United States and other coun-tries, and may not be used without written permission. SQL Server is a registered trademark of Microsoft Corporation in the United States and/or other countries. All other trademarks are the property of their respective owners. Wiley Publishing, Inc. is not associated with any product or vendor mentioned in this book.

84289ffirs.indd vi84289ffirs.indd vi 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

For Gemma, with all my love and thanks

—Christian Bolton

This is dedicated to Erika, who has been dedicated

to me despite my long working hours. I love you

dearly, and I love working on next chapters.

—Brent Ozar

84289ffirs.indd vii84289ffirs.indd vii 11/23/09 1:35:42 PM11/23/09 1:35:42 PM

84289ffirs.indd viii84289ffirs.indd viii 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

ABOUT THE AUTHORS

CHRISTIAN BOLTON is the Technical Director for Coeo Ltd., a leading provider of SQL Server consulting and managed support services in the UK and Europe. Prior to this Christian worked for 5 years at Microsoft, leading the SQL Server Premier Field Engineering team in the UK. He is a Microsoft Certifi ed Architect, Master and MVP for SQL Server, and co-author of Professional SQL Server 2005 Performance Tuning.

He works out of London and lives in the south of England with his wife and children. He can be contacted at christian@coeo.com. Christian authored chapters 1, 2, 7, 13 and the online chapter 16 in addition to lending his authoring expertise where needed on other chapters and functioned as the lead author for the entire project.

JUSTIN LANGFORD leads the Managed Support team for Coeo Ltd, delivering out-sourced 24x7 operations for mission-critical SQL Server platforms. Before join-ing Coeo, Justin worked for Microsoft in the Premier Field Engineering team and has worked with some of Microsoft’s largest fi nance and government customers in Europe. Justin co-authored Wrox Professional SQL Server 2005 Performance Tuning

and lives in London with his girlfriend, Claire. Outside work he enjoys sailing and has a keen inter-est in classic British sports cars. Justin can be contacted at justinl@coeo.com. Justin authored chap-ters 9, 11, 12, and 15.

BRENT OZAR is a SQL Server Expert for Quest Software. He has over a decade of broad IT experience, performing SQL Server database administration, systems admin-istration, SAN administration, virtualization administration, and project management. In his current role, Brent trains DBAs on performance tuning, disaster recovery, and virtualization. He has spoken around the globe at events for PASS, SQLBits, SSWUG,

and other organizations. Brent founded the Virtualization Virtual Chapter for the Professional Association for SQL Server (PASS), and serves as the Editor-in-Chief at SQLServerPedia.com.

Brent blogs at www.BrentOzar.com and discusses diverse topics at http://twitter.com/BrentO. When he’s not talking SQL Server, he enjoys traveling, working with social media, snorkeling, and sam-pling new restaurants. He is the author of chapters 4 and 14.

JAMES ROWLAND-JONES works for EMC Consulting EMEA as an Advisory Con-sultant. His principle focus is the delivery of large, scalable, data warehousing, and business intelligence projects. Within this fi eld James specializes in data integration, database architecture, and performance tuning. He is very active in the technical community and is one of the organizers for SQLBits, Europe’s largest SQL Server

community conference. James has received the Microsoft MVP award for 2009 and 2010. You can fi nd him online at http://consultingblogs.emc.com/jamesrowlandjones, twitter (@jrowlandjones), linkedin http://www.linkedin.com/in/jrowlandjones, or even using an old fashioned email, james.rowland-jones@emc.com. James authored chapters 6 and 10.

84289ffirs.indd ix84289ffirs.indd ix 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

JONATHAN KEHAYIAS is a SQL Server MVP, MCITP Database Administrator and Developer, who got started in SQL Server in 2004 as a database developer and report writer in the natural gas industry. After spending two and a half years working in T-SQL, in late 2006, he transitioned to the role of Database Administrator. He has experience in upgrading and consolidating SQL environments, and has experience in

running SQL Server in Virtual Environments on VMWare ESX 3.5+. He is a member of the Tampa SQL Server User Group and a regular speaker about SQL Server at events. Jonathan authored chapter 3.

CINDY GROSS started her SQL Server life as a DBA with a hospital and health plan company in 1993, and moved to Microsoft in 2000 where she has worked ever since. Her roles at Microsoft have included PSS Product Support Engineer, SQL Content Lead, Yukon Readiness Lead, and most recently Dedicated Support Engineer (DSE), all for SQL Server. Cindy enjoys delivering training throughout the United States as well as in Europe and India, including presentations at SQL PASS. In 2008 she started the Boise SQL Server User Group, an affi liate of the SQLCommunity.org site (where she is a worldwide SQL Server Community Leader) to share SQL Server wisdom in the Idaho SQL Server community. Most recently she obtained the Microsoft Certifi ed Master: SQL Server Qualifi cation, which was a hard-fought prize. Over the years Cindy has learned from many won-derful friends and colleagues and they all deserve a word of thanks for contributing to her success. Cindy authored chapter 8.

STEVEN WORT has been working with SQL Server since the early days of SQL Server way back in 1992-93. He is currently a developer in the Windows Division at Microsoft, where he works on performance and scalability issues on large database systems for the Windows Telemetry team. Steven has been at Microsoft since 2000. Prior to working in the Windows Division, Steven spent 2 years working in the SQL

Server group, working on performance and scalability. Steven’s fi rst 3 years at Microsoft were spent working in support as an escalation engineer on the SIE team. During this time, Steven was able to travel the world working with some of Microsoft’s customers on their performance and scal-ability issues. Before coming to Microsoft, Steven spent 20 years working in the United Kingdom as a freelance consultant, specializing in database application development. When Steven isn’t busy working, he can be found spending time with his family and enjoying many fi tness activities in the outdoors of the Pacifi c Northwest. Steven authored chapter 5.

84289ffirs.indd x84289ffirs.indd x 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

EXECUTIVE EDITOR

Robert Elliott

SENIOR PROJECT EDITOR

Ami Frank Sullivan

TECHNICAL EDITORS

James Boother

Jimmy May

Paul Nielsen

Tony Rogerson

James Rowland-Jones

Simon Sabin

Steven Wort

SENIOR PRODUCTION EDITOR

Debra Banninger

COPY EDITOR

Luann Rouff

EDITORIAL DIRECTOR

Robyn B. Siesky

EDITORIAL MANAGER

Mary Beth Wakefi eld

MARKETING MANAGER

Ashley Zurcher

PRODUCTION MANAGER

Tim Tate

VICE PRESIDENT AND

EXECUTIVE GROUP PUBLISHER

Richard Swadley

VICE PRESIDENT AND

EXECUTIVE PUBLISHER

Barry Pruett

ASSOCIATE PUBLISHER

Jim Minatel

PROJECT COORDINATOR,

COVER

Lynsey Stanford

COMPOSITOR

Chris Gillespie,

Happenstance Type-O-Rama

PROOFREADER

Nancy Carrasco

INDEXER

Robert Swanson

COVER DESIGNER

Michael E. Trent

COVER IMAGE

Punchstock / Glowimages

CREDITS

84289ffirs.indd xi84289ffirs.indd xi 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

84289ffirs.indd xii84289ffirs.indd xii 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

ACKNOWLEDGMENTS

FROM CHRISTIAN BOLTON: This book has been a far greater challenge and rewarding experience than I ever imagined. As with anything worth shedding blood, sweat, and tears over, it has taken a number of people, generous with their time and talents, to bring this project to life.

First of all, I’d like to thank my wife, Gemma, for her support and eternal patience during the many evenings and weekends I spent researching, writing, and reviewing content for “just a little longer.” My children, Ava and Leighton, deserve a special thank you also for frequently pulling me back to what really matters with cheeky grins, demands to ride on my shoulders, spin in my chair, and an offer to draw a picture of a princess for me to put in the book.

I’d also like to thank my fellow authors and contributors for their outstanding efforts in bringing new, interesting, and well researched material to make this book unique:

Justin Langford has been a great friend for many years and is always the fi rst to offer support ➤

and encouragement to any project.

James Rowland-Jones for setting impossible goals for his chapters on Locking and SQL Trace ➤

and then going past them.

Brent Ozar, for transferring his easy-reading blog style into professional book chapters on ➤

Storage and Management Studio Reports that are a pleasure to read.

Jonathan Kehayias for his excellent chapter on Waits and Extended Events that would have ➤

been a pale shadow of its current form had he not been involved in the book.

Cindy Gross for bringing her years of experience at Microsoft and defi ning what it takes to ➤

be a Professional when troubleshooting SQL Server.

Steven Wort for working to extremely tight timescales on the CPU and Query Processing ➤

chapter and working tirelessly to improve and expand on the original specs.

Starting a new chapter from a blank page is one of the hardest things you can do as an author, but it is ultimately rewarding when you see the fi nished product. I often think that Technical Editors don’t get enough praise for the work they do to make the authors look good, so I want to say a big thank you to our team of Technical Editors for their diligent research, tips, advice, and patience through multiple revisions: Simon Sabin, James Rowland-Jones, James Boother, Jimmy May, Paul Nielsen, Steven Wort, and Tony Rogerson. I’d also like to thank Ami Sullivan, our Project Editor at Wiley, for keeping the schedule moving and working very hard to compensate for our writing delays, and Robert Elliott, our Executive Editor, for buying into the original vision and helping me to refi ne the proposal that eventually became this book.

Writing about SQL Server Internals with a suffi cient abstraction in certain areas to introduce key topics and then drilling down into the heart of the product in others presents a diffi cult challenge for anyone, but it is much harder writing from outside Microsoft; and I’d like to thank Bob Ward,

84289ffirs.indd xiii84289ffirs.indd xiii 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

Ewan Fairweather, and Thomas Kejser from Microsoft for their enthusiastic support for the project and for clarifying some of the fi ner details of exactly what the code in SQL Server is doing.

I’d also like to thank the following SQL Server professionals, MVPs, and Microsoft staff for their inspiration and support whether knowingly or not: Mark Anderson, Sunil Agarwal, Chirag Roy, Aaron Bertrand, Denny Cherry, Grant Fritchey, and Paul Randal.

Finally, I’d like to thank you for buying this book. Everyone involved has given their best game to make it stand-out; especially the authors and contributors who have given a little bit of what makes them special in their fi eld to create a book that they’re proud to put their name to. I hope it lives up to your expectations and you fi nd it a worthy investment.

FROM JAMES ROWLAND-JONES: Firstly I’d like to thank Christian for the opportunity to be involved with the book, Simon Sabin and Tony Rogerson (my TE’s) for their efforts and constructive feed-back, and also to Ami Sullivan, our editor, for her endless support and patience. The management team at EMC Consulting and especially Rob Grigg have been a constant source of support; both on the book and in the community with user groups and SQLBits, so a big thank you to them. I’d also like to thank Bob Ward and Thomas Kejser for their reviews and insight during some tough times on the locking chapter. Finally, I’d like to thank all my family, but especially my wife Jane and our children Lucy, Kate, and Oliver. Without you life is strangely pointless.

FROM BRENT OZAR: Thanks to Jimmy May for hooking me up with such a great team of authors. My involvement with this book wouldn’t have happened without his encouragement and trust. Similarly, I’d like to thank the authors for giving me an opportunity to work with them. I’m hum-bled to have my names next to such great technical minds. Thanks also to Christian Hasker, Rony Lerner, Don Duncan, and Joe Sullivan; I have consistently hit the lottery when it comes to manag-ers, and I couldn’t have picked a better string of guys to mentor me and grow my career. If anybody doesn’t succeed under any of them, it’s their own fault.

To my coworkers Andy Grant, Brett Epps, Heather Eichman, and everybody else at Quest Software, thanks for making my work so much fun. Thanks to everybody on Twitter for laughing with me while I keep punching harder and faster. My day is infi nitely more enjoyable thanks to folks like @SQLRockstar, @SQLChicken, @SQLAgentMan, @KBrianKelley, @Wendy_Dance, @GFritchey, @MrDenny, @StatisticsIO, @MikeHillwig, @Peschkaj, @SQLSarg, @SQLCraftsman, and many others.

Finally, I’d like to thank Dad, Mom, and Caryl for my dashing good looks and brilliant wit. I just wish you’d given me some humility so that I could be perfect.

84289ffirs.indd xiv84289ffirs.indd xiv 11/23/09 1:35:43 PM11/23/09 1:35:43 PM

CONTENTS

INTRODUCTION XXV

SQL SERVER ARCHITECTURE CHAPTER 1: 1

Database Transactions 2

ACID Properties 2

SQL Server Transactions 3

The Life Cycle of a Query 3

The Relational and Storage Engines 4

The Buff er Pool 4

A Basic Select Query 5

A Simple Update Query 14

Recovery 16

The SQLOS (SQL Operating System) 20

Summary 22

UNDERSTANDING MEMORY 2CHAPTER 2: 3

Windows and Memory 24

Physical Memory 24

Virtual Address Space 27

Virtual Memory Manager 29

Tuning 32-Bit Systems 31

Tuning 64-Bit Systems 45

SQL Server Memory 47

Memory Nodes 47

Memory Clerks, Caches, and the Buff er Pool 47

Summary 58

SQL SERVER WAITS AND EXTENDED EVENTS 5CHAPTER 3: 9

Waits 60

SQL Server Execution Model 60

Understanding Wait Statistics 61

Wait Types 64

84289ftoc.indd xv84289ftoc.indd xv 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xvi

CONTENTS

Extended Events 70

Architecture 71

Event Sessions 81

Using the Extended Events Manager 87

Examining Waits with Extended Events 90

Summary 94

WORKING WITH STORAGE 9CHAPTER 4: 5

Types of Storage 95

Understanding Individual Drives 96

Protecting Data with RAID 98

Direct Attached Storage 105

Storage Area Networks 105

Storage Performance Testing 112

Choosing a Storage Testing Tool 113

Interpreting Storage Test Results 119

Confi guring Software for Storage 123

Confi guring Windows Server 123

Confi guring SQL Server 2008 128

Corruption 131

Summary 135

CPU AND QUERY PROCESSING 13CHAPTER 5: 7

The CPU 138

The CPU and SQL Server 139

Processor Speed 139

Hyper-Threading 140

Licensing with Multicore and Hyper-Threading 141

Cache 141

Multicore Processors 143

System Architecture 144

Symmetric Multiprocessing 145

NonUniform Memory Access 145

Query Processing 148

Parsing 148

Binding 149

Query Optimization 149

Parallel Plans 151

Algebrizer Trees 151

84289ftoc.indd xvi84289ftoc.indd xvi 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xvii

CONTENTS

sql_handle or plan_handle 152

Statistics 152

Plan Caching and Recompilation 154

Infl uencing Optimization 159

Query Plans 165

Query Plan Operators 168

Reading Query Plans 171

Executing Your Queries 175

SQLOS 175

Summary 183

LOCKING AND LATCHES 18CHAPTER 6: 5

Transactions 187

Atomic 187

Consistent 187

Isolated 188

Durable 188

Consequence of Concurrent Access 188

Lost Updates 189

Dirty Reads 191

Non-Repeatable Reads 192

Phantom Reads 193

Double Reads 196

Halloween Eff ect 198

Locking 198

Viewing Locks — sys.dm_tran_locks 199

Lock Granularity 202

Lock Modes 205

Lock Hierarchy 212

Lock Compatibility 213

Lock Escalation 214

Deadlocking 216

Pessimistic Concurrency 219

Pessimistic Isolation Levels 220

Concurrency vs. Isolation 231

Optimistic Concurrency 231

Optimistic Isolation Levels 232

How Row Versioning Works 240

Row Versioning Deep Dive 240

Monitoring Row Versioning 249

84289ftoc.indd xvii84289ftoc.indd xvii 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xviii

CONTENTS

Latches 251

Latch Types 251

BUF Latch Architecture 255

Latch Modes 258

Grant Order 260

Latch Waits and Blocking 261

Sub-latches and Super-latches 262

Latching in Action 263

Without Latching 266

With Latching 267

Summary 268

KNOWING TEMPDB 26CHAPTER 7: 9

Overview and Usage 270

User Temporary Objects 271

Internal Temporary Objects 275

The Version Store 276

Troubleshooting Common Issues 279

Latch Contention 279

Monitoring Tempdb Performance 289

Troubleshooting Space Issues 291

Transaction Log Growing Too Big? 295

Confi guration Best Practices 296

Tempdb File Placement 296

Tempdb Initial Sizing and Autogrowth 300

Confi guring Multiple Tempdb Data Files 303

Summary 303

DEFINING YOUR APPROACH TO TROUBLESHOOTING 30CHAPTER 8: 5

Approaching the Problem Correctly 306

Having the Right Attitude 306

Dealing with Management 307

When to Call for Outside Help 308

Defi ning the Problem 309

Tips for Identifying a Problem 309

Bite-Size Chunks 310

Service-Level Agreements 311

Defi ning Exit Criteria 311

Understanding Your Baselines 311

Events and Alerts 315

84289ftoc.indd xviii84289ftoc.indd xviii 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xix

CONTENTS

Gathering Data 315

Understanding the Data Gathering Process 315

Tools and Utilities 316

Analyzing Data 317

SQL Nexus 317

Profi ler 317

Database Tuning Advisor 318

Visual Studio Database Edition 318

Testing Solutions 318

Troubleshooting Other Components 318

Failover Clustering 318

Replication 319

Analysis Services 322

Summary 327

VIEWING SERVER PERFORMANCE WITH CHAPTER 9: PERFMON AND THE PAL TOOL 329

Performance Monitor Overview 330

Reliability and Performance Monitor in Windows Server 2008 330

New Counters for SQL Server 2008 in PerfMon 333

Troubleshooting SQL Server Problems 335

Getting Started with PerfMon 338

Monitoring Real-Time Server Activity 338

Starting Out with Data Collector Sets 341

Working with User-Defi ned Data Collector Sets 342

What to Be Aware of When Running PerfMon 346

The Impact of Running PerfMon 346

Servers with Very Poor Performance 348

Common PerfMon Problems 348

Getting More from Performance Monitor 349

Identifying SQL Server Bottlenecks 350

Wait Stats Analysis 356

Getting a Performance Baseline 357

Getting Started with Performance Analysis for Logs (PAL) 357

Templates and PAL 358

Capturing PerfMon Logs 358

Using PAL for Log Analysis 359

Other PerfMon Log Analysis Tools 363

Using SQL Server to Analyze PerfMon Logs 364

Combining PerfMon Logs and SQL Profi ler Traces 364

84289ftoc.indd xix84289ftoc.indd xix 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xx

CONTENTS

Using Relog 365

Using LogMan 366

Using LogParser 367

Summary 368

TRACING SQL SERVER WITH SQL TRACE CHAPTER 10: AND PROFILER 369

Tracing 101 370

Why Trace 370

When to Trace 370

Where to Trace 370

What to Trace 371

How to Trace 371

The Architecture of SQL Trace 372

Event Classifi cation and Hierarchies 374

SQL Trace Catalog Views 376

sys.traces 376

sys.trace_categories 386

sys.trace_events 387

sys.trace_columns 387

sys.trace_subclass_values 389

sys.trace_event_bindings 391

SQL Trace Procedures and Functions 392

sp_trace_create 392

sp_trace_setevent 394

sp_trace_setfi lter 400

sp_trace_setstatus 403

sp_trace_generateevent 404

fn_trace_gettable 406

fn_trace_geteventinfo 408

fn_trace_getfi lterinfo 409

Securing SQL Trace 410

Tracing Login Creation/Deletion 411

Logins Changing Passwords 412

When Tracing Login Can View Object Defi nitions

and Parameter Values 414

Securing the Output of SQL Trace 418

84289ftoc.indd xx84289ftoc.indd xx 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xxi

CONTENTS

Profi ler 420

Advanced Features of Profi ler 420

Exporting a Trace Defi nition 424

Exporting an Existing Server-Side Trace 431

Tips & Tricks 433

Summary 435

CONSOLIDATING DATA COLLECTION WITH CHAPTER 11: SQLDIAG AND THE PERFSTATS SCRIPT 437

Approaching Data Collection 438

What Is SQLDiag? 439

An Overview of SQLDiag 440

Using SQLDiag 444

Running SQLDiag in Production 447

Using the PerfStats Script 451

What Is the PerfStats Script? 451

Where to get the PerfStats Script 452

Confi guring the PerfStats Script 454

Running the PerfStats Script 454

Analyzing PerfStats Output 455

Summary 457

INTRODUCING RML UTILITIES FOR STRESS CHAPTER 12: TESTING AND TRACE FILE ANALYSIS 459

When to Use RML Utilities 460

Testing New Applications before Going Live 461

Validating the Impact of a Change 461

Determining the Purpose of Testing 462

What Are RML Utilities? 462

History of RML Utilities 463

What’s in the Download 463

Components of RML Utilities 464

Performance Testing 473

Testing Scenarios 473

Ensuring a Fair Test 473

Capturing a SQL Trace 476

Analyzing Large Datasets 477

Summary 478

84289ftoc.indd xxi84289ftoc.indd xxi 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xxii

CONTENTS

BRINGING IT ALL TOGETHER WITH SQL NEXUS 48CHAPTER 13: 1

Getting Started 484

Data Collection 484

Is the Default Data Collection Good Enough? 485

Modifying the Data Collection 487

Collecting Data for a Specifi c Instance 487

Knowing How Much Data to Collect 488

Importing Data 489

Available Reports 491

Example Scenario Using SQL Nexus 491

Confi guring the Data Collection 492

Generating the Workload 493

Importing the Data 494

Looking at the Bottlenecks 494

Testing the Resolution 501

Adding Your Own Reports 502

Where Are the PerfMon Reports? 503

Summary 504

USING MANAGEMENT STUDIO REPORTS CHAPTER 14: AND THE PERFORMANCE DASHBOARD 505

Using the Standard Reports 506

Interpreting the Standard Server Reports 507

Interpreting the Database Reports 519

Using the Performance Dashboard 524

Troubleshooting Common Errors 526

Interpreting the Performance Dashboard Reports 527

Building Custom Reports 531

Building a Custom Report with BIDS 532

Picking Custom Reports Candidates 536

Summary 537

USING SQL SERVER MANAGEMENT CHAPTER 15: DATA WAREHOUSE 539

Introducing Management Data Warehouse 540

Background to MDW 540

MDW Architecture 541

Implementing MDW 543

Creating a Management Data Warehouse 544

Set Up Data Collection 545

84289ftoc.indd xxii84289ftoc.indd xxii 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

xxiii

CONTENTS

System Collection Sets 545

Uses for MDW 547

Performance Overhead 548

Reporting from MDW 549

Custom Data Collection and Reporting 551

Defi ning Custom Collections 551

SSAS Monitoring Scripts for the MDW 554

Summary 554

SHORTCUTS TO EFFICIENT DATA CHAPTER 16: COLLECTION AND QUICK ANALYSIS Wrox.com

INDEX 555

84289ftoc.indd xxiii84289ftoc.indd xxiii 11/23/09 2:50:08 PM11/23/09 2:50:08 PM

84289flast.indd xxiv84289flast.indd xxiv 11/23/09 2:49:49 PM11/23/09 2:49:49 PM

INTRODUCTION

WHILE PUTTING TOGETHER THE PROPOSAL that eventually became this book, the aim was to write a troubleshooting guide that covered the additional tools available from the SQL Server community.

It soon became clear, however, that to effectively talk about the tools, so many digressions were necessary to explain the results that the fl ow and impact were interrupted. The decision was made to alter the approach to include architectural information, not just on SQL Server, but on the whole platform on which SQL Server depends.

If you’re troubleshooting an apparent “SQL Server” issue, you need to be able to troubleshoot the underlying operating system and storage as well as SQL Server, so we wanted to bring together and simplify the architectural details of these components too.

A fair amount of Windows and storage internals information is available already, but very little of it that condenses and fi lters the right material to be easily consumed by SQL Server professionals. The available material is either too light or too in depth — with nothing to help bridge the gap.

Combining this need with the need for practical internals information on SQL Server, a look at building a troubleshooting methodology, and relevant information on all the extra tools, three goals for the book were established:

To provide in-depth architectural information on SQL Server (and the environment on which ➤

it depends) that is easy to consume

To introduce a troubleshooting approach using the same techniques and methodologies that ➤

Microsoft uses internally

To present some of the additional free SQL Server troubleshooting tools that are available ➤

with real-world examples demonstrating how they can be used together to effi ciently and accurately determine the root cause of issues on systems running SQL Server

WHO THIS BOOK IS FOR

This book is intended for those people who regard themselves as, or who aspire to be, SQL Server professionals in predominantly relational environments. What I mean by a SQL Server professional, is anyone that regards SQL Server as one of their core product skills and continually strives to develop their knowledge of the product and how to use it.

It is not a beginner’s book and makes assumptions that the reader knows the basics of how to install, use, and confi gure SQL Server and is aware of some of the challenges that troubleshooting SQL Server problems using only the native tools presents. Every effort has been made however, to provide a gentle route into each area for those readers who are less confi dent in some of the topics presented.

84289flast.indd xxv84289flast.indd xxv 11/23/09 2:49:50 PM11/23/09 2:49:50 PM

xxvi

INTRODUCTION

The book is presented in two parts. The fi rst covers internal information which is intended to pro-vide an in-depth grounding in core concepts and provides the knowledge to help understand the output and positioning of the tools covered in part two. Those readers who are confi dent with the subject matter presented in part one will fi nd that they can start reading from part two and dip back into part one as required to clarify any understanding.

WHAT THIS BOOK COVERS

Before launching into a description of the structure of the book and each chapter, it’s important for you to understand some key drivers and assumptions that originally led to the topics the book covers.

Understanding Internals

You don’t need to understand too much about how SQL Server works to be successful in many SQL Server–based job roles. You can fi nd numerous well-established, prescriptive guidelines and a very active and helpful community to help you. Eventually, however, you will reach a point when that’s just not enough (usually when something serious has gone wrong).

During an unexpected service outage, for example, you need to make quick decisions in order to balance the demands of restoring the service as quickly as possible while gathering enough data to help you diagnose the issue so you can prevent it from happening again. In that situation you cannot depend on external help or goodwill; it won’t arrive fast enough to help you. Understanding inter-nals will enable you to make quick decisions for problem resolution independently.

When I worked for Microsoft, one of our customers encountered corruption in a large business-critical database running on SQL Server. The business decided to take the database offl ine until it was fi xed because it held fi nancial trade data, and mistakes would have been disastrous.

They ran DBCC CHECKDB, which can be used in SQL Server to help detect and resolve corruption, but killed it after eight hours in favor of a database restore. The backup was corrupt so they had no option but to run CHECKDB again, which fi xed the problem after another 12 hours. It was a time-consuming disaster and led to a large fi ne having to be paid by the customer for failing to provide a service to the fi nancial markets.

The simple lessons to learn from this example are to test your backups and to know how long CHECKDB takes to run (and that it takes longer when corruption is detected, as it takes another pass with deeper checks). These are “best practices” that can be followed with little understanding of actual internals.

However, the reason for including this example is the information that resulted from the postmortem. The original error message that detected the problem contained details of a corrupt page. Armed with a data page number, the troubleshooting team could have used DBCC PAGE to look at the header and determine to which database object it belonged.

84289flast.indd xxvi84289flast.indd xxvi 11/23/09 2:49:50 PM11/23/09 2:49:50 PM

xxvii

INTRODUCTION

In this case it actually belonged to a non-clustered index that could have just been rebuilt without having to take the entire database down to run CHECKDB or restore the entire database. This is why it’s useful to know the “internals”; so you can work things out for yourself and take the best course of action.

This book covers internals information for Windows and SQL Server that helps you understand the environment in which your application(s) works, how to confi gure your server to optimize for differ-ent requirements, and how to avoid making blind decisions in the heat of the moment because you don’t know why you’re seeing a particular behavior.

Additional Troubleshooting Tools

The second part of this book, which was actually its original source of inspiration, deals with a range of free troubleshooting tools that can be used together to form a structured, effective, trouble-shooting strategy. We wanted to write a practical guide to these useful tools that can make your life so much easier on a daily basis (but which can seem overly complicated and diffi cult to learn to the uninitiated).

HOW THIS BOOK IS STRUCTURED

The fi rst part of the book starts with a high-level overview of SQL Server’s architecture, leading into chapters on the three core resources that are important to SQL Server: memory, storage, and CPU. Nestled in between these chapters at strategic points are additional chapters which cover material that is critical to understand for effective troubleshooting: SQL Server Waits and Extended Events, Locking and Latches, and tempdb.

This section provides an overview of each chapter to put it into context within the book and to help you decide where to start reading.

Chapter 1: SQL Server Architecture

This chapter takes you lightly through the life cycle of a query, with enough depth to help you under-stand fundamental concepts and architectures without getting lost in the complexities of individual components (some of which are looked at closely in later chapters).

This chapter will appeal to readers at all levels of skill, whether you’re a developer, a DBA, or a sea-soned SQL Server veteran.

Chapter 2: Understanding Memory

With this chapter we wanted to expand the scope of “memory” to include the physical components and Windows memory management, rather than just cover SQL Server’s internal usage so you’ll be able to read about the different types of memory modules you can buy and learn how Windows manages physical memory using a Virtual Memory Manager. It also compares 32-bit and 64-bit architectures and the options you have for tuning them.

84289flast.indd xxvii84289flast.indd xxvii 11/23/09 2:49:50 PM11/23/09 2:49:50 PM

xxviii

INTRODUCTION

For SQL Server itself, you’ll learn about architectural elements such as memory nodes, clerks, caches, and the buffer pool, as well as the often misunderstood concept of memtoleave and how to measure it. You will also read all about AWE usage and how to implement it in this chapter.

The objective of this chapter is to provide you with a thorough understanding of how SQL Server uses memory. Once you understand the core architecture of SQL Server’s memory management, you will be well prepared to diagnose memory-related problems as well.

Chapter 3: SQL Server Waits and Extended Events

This chapter introduces the benefi ts of reviewing SQL Server Waits and the architecture that supports this feature. It looks into how they occur and what they mean; the common wait types of concern; which wait types can be safely ignored; and what new wait types there are in SQL Server 2008.

It then covers a new feature in SQL Server 2008 called Extended Events and shows how they can be used to get a deeper look into what waits are occuring for individual tasks and why. Finally, it demonstrates how to implement and manage events using a freeware tool called the Extended Events Manager which was written by the author.

The objective of this chapter is to introduce and reinforce the benefi ts of adding waits analysis to your troubleshooting method and how to take it to the next level with Extended Events.

Chapter 4: Working with Storage

This chapter equips you with the knowledge to confi dently specify and monitor your storage require-ments, from understanding the uses for different types of physical disks and knowing the real-world implications for different RAID levels, to being comfortable with the technologies that make up a storage area network (SAN) and various optimization tips that you can employ on different types of storage.

We compare SANs and direct-attached storage (DAS) so you can be clear about the environment in which each is appropriate, and you’ll learn about the implementation details and implications of performance-tuning tips like increasing HBA queue depth and implementing disk sector alignment.

The objective of this chapter is to ensure that you and your storage administrator can communicate using common terminology and address key storage performance bottlenecks in a cooperative and collaborative way.

Chapter 5: CPU and Query Processing

This chapter covers two key, interrelated areas. First you’ll read about CPUs, looking at how they work and where the technology is heading so you can decide what features are more important to you when choosing the processors that will be available in your server.

Then, the chapter takes an in-depth look at how SQL Server processes queries because understand-ing how a query is processed will help you determine how to tune it. However, understanding what drives SQL Server to determine this plan will also aid you in understanding why SQL Server made

84289flast.indd xxviii84289flast.indd xxviii 11/23/09 2:49:50 PM11/23/09 2:49:50 PM