Integrating Microsoft SQL Server Event Tracing with OS-Level Events and Database Client EventsBob BeaucheminDeveloper Skills PartnerSQLskills

SESSION CODE: DAT402

About Bob Beauchemin

Independent Consultant/Trainer/Writer/SpeakerDeveloper Skills Partner, SQLskills

Website: http://www.SQLskills.comBlog: http://www.SQLskills.com/blogs/bobb

SQL Server MVP

Author of books and numerous resources

related to SQL Server and data access

A Developer's Guide to SQL Server 2005

A First Look at SQL Server 2005 for Developers

“Best Book” SQL Server Magazine’s Reader’s Choice Awards (2005)

Essential ADO.NET

ContentsIntegrating Tracing With Perfmon in SQL Server 2005/8Event Tracing For Windows

ArchitectureSQL Server and client providers integration points

Extended Events in SQL Server 2008Packages, Events, Actions, and PredicatesETW and other targetsIntegrating SQL Server ETW trace and other ETW providers

Client tracingRegistering client traceConfiguring and running client traceIntegrating SQL Server - server, client, and network trace

SQL Server DiagnosticsSQL Server pre-2008 supports

Performance monitor countersCounters are cumulativeLittle/no built in filtering

SQL Server ProfilerGranular eventsColumn filtersAutomated via server-side trace queues

Event Tracing for Windows (ETW)Exposes profiler events

SQL Server ProfilerProfiler is the most used diagnostic tool for SQL Server

Graphic user interface for configuring trace parameters"Save to file" and "Save to table" optionsExport / Script Trace Definition saves DDL for server-side trace queueView trace in user interface

Real-time viewingLoad trace file

Replay trace Little integration with OS tracing facilities

How does ASP.NET traffic affect SQL Server?What network messages are being exchanged with clients?

SQL Server Profiler - PerfMon Integration Create a Profiler TraceCreate a Performance Monitor LogWorks solely based on time – make sure the two clients (if different) are time correlatedOpen Trace (complete load),

Use File – Import Performance DataSelect Objects/Counters…

Works with SQL Server 2000 as well as SQL Server 2005/2008 It’s client side

Event Tracing For WindowsETW is the built-in Windows trace facility

Since Windows 2000Enables tracing of kernel or application events

In real-timeTo a log file

Minimal overheadSQL Server can use ETW

Integration point with all other system tracing facilities

ETW ComponentsAPI consists of three components

ControllersStart and stop event tracing sessionsEnable and disable providersManage buffer poolsKeep trace session statistics

Providers Generates eventsClassic (pre-Vista) or Manifest-based (Vista and later)

ConsumersOne of more tracing sessions as event source

Integration Via Event Tracing For WindowsETW tracing ships with Windows

Controllers – logman, xperf, perfmon (Data Collector Sets), othersMany built-in providers

Logman -query providers to view listOS providers - Kernel, LSA, Kerberos, NTLM…Application providers - IIS, ASP, ASP.NET…

Classic Providers Uses WMI registryMOF files used to decode trace recordsNot required to run traceOne event session per provider limit

Vista/Windows Server 2008 use manifest filesBetter integration with rest of OS toolsUp to 8 event sessions per provider

Consumers – tracerpt, xperf, xperfview, others

Post-processingETL format is well-defined

...but binary for speed (needs decoding)Tracerpt is the built-in processor

Produces XML by defaultOther formats available (HTML, CSV)Not particularly user-friendly

Logparser 2.2 most well-known Processes ETW, CSV, other formatsSome format conversion (one is load to SQL tables)SQL query language built-in

XPerf is part of Windows Server 2008 SDKCan merge traces and other information (e.g. symbols)Treats classic ETW providers as "legacy" (only event GUIDs available)

Other Post-ProcessorsYou can write your own postprocessor

Non-trivial for rich outputDecoding event detail field most complex

This field could change in futureYou can use SSIS as postprocessor

Load into SQL ServerAllows SQL queriesSubstring into event detail fieldBe careful with timer fields

High resolution timer "always looks the same in Excel"

End-To-End Tracing via ETWSQL Server 2008 ETW Provider

ETW target for SQL Server 2008 Extended EventsSQL Server ETW Provider

SQL Server 2005 trace event provider (will be deprecated)Database (but not Analysis Services) profiler events

StreamInsight (SQL Server 2008 R2) ETW ProviderUsed for real-time and offline tracingCustom client – Event Flow Debugger

SQL Server Clients and Network StackETW provider for ADO.NETETW provider for SQL Native Client (OLE DB and ODBC)Support for MDAC 2.8.1, JDBC

Using ETW with SQL Server ProviderTo run an ETW session

Register etwcls.mof with mofcomp.exeConfigure events to capture in etwcnf.xmlUse logman to stop and start the trace

Provider: MSSQLSERVER Trace Actually [Instance] TraceName event template in etwcnf.xml at trace startup

Post-process with tracerpt

Extended Events – ETW Support And MoreA general event-handling systemPart of SQL Server 2008

Used to implement auditingEngine lives in sqlserver.exe

Consumers can be in-proc or out of procETW consumer is provided

Enables correlation between SQL Server and Windows trace information

How are extended events different?Extended events

Less overhead than server-side trace queuesSingle event cost is ~2us on a 2GHz processor

Event and consumer agnosticAny event can be processed by any consumerNew events can be added, immediately useable

Allows action to be taken when an event firesAdditional information can be included

Rich predicate system for filtering

Starting extended eventsExtended event sessions defined via DDL

CREATE EVENT SESSIONALTER EVENT SESSIONDROP EVENT SESSION

Event sessions specifyEventsActionsTargets (consumers)

What's available?

Extended Event ObjectsA module is a container for packages - loaded on demandA package is a container for event objects

Any package's objects usable with any packages (except auditing objects)An event is a trace point in code

Events can be added to running EVENT SESSIONSEvents have a versionable schema

Actions can add data or additional context to the eventExamples - SQL trace, stack trace

Predicates filter events at publish timeTypes are a data types supported by extended eventsMaps are enumerations

TargetsTarget is an event consumer

Can be synchronous or asynchronousEvent Session can specify what to do if target can't keep upEvent Session can specify buffering specifics

Available TargetsETW classic sync targetSynchronous/Asynchronous bucketizerPair matchingAsynchronous file targetSynchronous event counterRing bufferAudit targets - file, security/application logs

Only useable with audit DDL

ETW Target SpecificsThe ETW target is a classic ETW provider

Has to work with pre-Vista operating systemsIt’s a synchronous targetService Account must be a member of Performance Log Users groupMust be configured and started via DDL

Not controllable via logman or other controllersCreates an ETW event session, XE_DEFAULT_ETW_SESSION

If session already exists, additional event sessions use itShared among all SQL Server instances on the computerRuns until machine shut down

Can stop it with logmanlogman stop XE_DEFAULT_ETW_SESSION -ets

EVENT SESSION RevisitedCREATE EVENT SESSION

Specify events, targets, actions, predicatesEvent session options

Max memory, memory partitioning, max event sizeDispatch latency and event retentionStartup stateCausality tracking

ALTER EVENT SESSIONCan start and stop sessionCan add or drop events, targets, objects

Extended Event MetadataCatalog views for defined session info

server_event_sessionsserver_event_session_targetserver_event_session_fieldsserver_event_session_actionsserver_event_session_events

DMVs for Event System Metadatadm_xe_packagedm_xe_objectsdm_xe_object_columnsdm_xe_map_values

DMVs for currently active session infodm_xe_sessionsdm_xe_session_targetsdm_xe_eventsdm_xe_event_actionsdm_xe_object_columns

Extended Event Use CasesCorrelate OS, SQL Server, Client activity

ETW Kernel providerUse with client API for end-to-end tracing

Any other ETW providerDifficult to correlate if many threads for same clients

Monitor LocksBucketizer and event pairer useful here

Wait analysisAudits built with extended events

Client-side TraceSQL Server 2005 and 2008 have trace in

ADO.NET 2.0 and aboveIn SqlClient, OleDbClient, OdbcClientIn System.Data.OracleClientIn System.Data.CommonIn .NET DataSetIn Entity Framework (.NET 3.5 SP1)SNI network APIs

SQL Native ClientSQL Native Client OLE DB providerSQL Native Client ODBC driverSNI network APIs

MDAC 2.81 and JDBC

Client Tracing ArchitectureADONETDIAG is an enabling DLL

Passes information to trace facilityTrace facility is currently ETW

...but could be changed is to use other trace facilitiesADONETDIAG provider for startup and shutdown information

MSDADIAG is an alternate enabling DLLIf .NET 2.0 is not installed

But wait, there's more...Trace is

High performanceNon-intrusiveUses standard trace mechanismFlexible architectureConfigurableMMC controllable

Enabling the TraceClient trace is not ready to run at install

Install registry entryInstall .mof file(s)Set up trace control fileTurn the trace on/offPost-process the .etl file

Trace ConfigurationConfigurable Through

Registry settingsmay be replaced by a control panel appletdon't depend on being able to write registry key

Provider config file-pf option on logman

Provider-specific bits in config filemean something different to each provider

ADO.NET Configuration Bits

ETW Session Configuration SettingsBit Setting Meaning

0x00000002 Regular Tracepoints

0x00000004 Execution Flow

0x00000080 Advanced Output

0x00001000 Connection Pooling Specific (System.Data.1 only)

0x01000000 Disable Tracing

0x10000000 Convert Unicode To Ascii

Trace Use CasesYou can use the trace

When program disguises underlying errorCorrelate server activity to client and network

Or client and other applications (ie IIS, ASP.NET)You have to know how to read network trace

To troubleshoot connection pool problemsFor problems in multiple-layered providers

Using ADO.NET with third party providersWhen using OleDb/Odbc .NET providerUsing MSDASQL with SQL Server (NOT a recommended configuration)

Tracing Oracle activityPSS doesn't have Oracle Client Stack source

Client Trace caveatsTracing is still in its infancy

No control panel applet, use registryBut don't depend on always having access

Only works with ADO.NET, SNAC, MDAC commonNo support for Analysis Service providerNo third party providers

Does not integrate with third party traces (much)Different from ODBC trace

Hooks to ADONETDIAG not publishedThird party integration not possible

Event detail field difficult to decode

SummarySide by Side Viewing of SQL Profiler and Perfmon

In SQL Profiler 2005 and aboveTracing is included in all components

Server, clients, and networkEvent Tracing for Windows is the glue

Can integrate multiple ETW sessionsCan start ETW session with multiple providersUses provider model for plug-in architectureLow overhead

Extended Events go beyond ETWEvent agnostic targets

Client trace permits end-to-end tracing of database and applications

ReferencesMSDN Webcast: SQL Server 2008 Advanced Troubleshooting with Extended EventsBob Beauchemin's Blog: Series in August 2007

http://www.sqlskills.com/blogs/bobb/default,month,2007-08.aspxData Access Tracing in SQL Server 2005, Bob Beauchemin, Jennifer Perret, Acey Bunch, MSDN OnlineInside Event Tracing For Windows, Darren Mar-Elia, WindowsITPro Magazine - Dec 2003http://blogs.msdn.com/sqlqueryprocessing/archive/2006/11/12/using-etw-for-sql-server-2005.aspxhttp://blogs.msdn.com/pigscanfly/pages/xperf-articles.aspx

DAT Track Scratch 2 Win

Find the DAT Track Surface Table in the Yellow Section of the TLCTry your luck to win a Zune HDSimply scratch the game pieces on the DAT Track Surface Table and Match 3 Zune HDs to win

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Complete an evaluation on CommNet and enter to win!

Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st

http://northamerica.msteched.com/registration

You can also register at the

North America 2011 kiosk located at registrationJoin us in Atlanta next year

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

JUNE 7-10, 2010 | NEW ORLEANS, LA