T-SQL and XML T-SQL and XML Enhancements in SQL Enhancements in SQL Server 2005 Server 2005 Eric NelsonEric NelsonApplication ArchitectApplication ArchitectMicrosoftMicrosofthttp://blogs.msdn.com/ericnelhttp://blogs.msdn.com/ericnel (SQL Dev and UK (SQL Dev and UK

ISV)ISV)

Many TSQL enhancements…Many TSQL enhancements…

Exception handlingException handlingTOP(expression)TOP(expression)PIVOT/UNPIVOTPIVOT/UNPIVOTAPPLYAPPLYCommon Table Common Table ExpressionsExpressionsHierarchical queriesHierarchical queriesRANKRANKSNAPSHOT IsolationSNAPSHOT IsolationVARCHAR(MAX), VARCHAR(MAX), VARBINARY(MAX)…VARBINARY(MAX)…XML TypeXML Type

Full Text Search Full Text Search improvementsimprovementsWAITFORWAITFOREvent Notifcations on Event Notifcations on DDL and DMLDDL and DMLFOR XMLFOR XMLTABLESAMPLETABLESAMPLEService BrokerService BrokerBULK INSERTBULK INSERT……Plus “none dev TSQL Plus “none dev TSQL stuff”stuff”

Online Index RebuildOnline Index RebuildSchemasSchemasCertificatesCertificatesStatement Level Statement Level Recompile …Recompile …

Exception HandlingException Handling““If @@Error” programming If @@Error” programming sucks! There must be a better sucks! There must be a better way…way…

Exception Handling: Exception Handling: TRY/CATCHTRY/CATCH

Eliminate tedious “if @@error” codeEliminate tedious “if @@error” codePerform logging/cleanup when exceptions Perform logging/cleanup when exceptions occuroccurAbility to re-raise exceptions after cleanupAbility to re-raise exceptions after cleanup

BEGIN TRY<core logic>

END TRYBEGIN CATCH TRAN_ABORT<exception handling logic>

END CATCH

Exception handlingException handling

In CATCH block you canIn CATCH block you canUse new built-in functions to retrieve Use new built-in functions to retrieve error-number, message, severityerror-number, message, severity

Re-raise original exception or raise an altRe-raise original exception or raise an alt

Transaction abortTransaction abortTx remains in “doomed” state until Tx remains in “doomed” state until explicitly rolled backexplicitly rolled back

No actions which result in log writes may No actions which result in log writes may be performed in a doomed transaction – be performed in a doomed transaction – SELECT only until you ROLLBACKSELECT only until you ROLLBACK

TOPTOPDrat …. I will not know until Drat …. I will not know until runtime how many rows I need…runtime how many rows I need…

TOP (<expression>)TOP (<expression>)

SQL 7.0 and 2000 SQL 7.0 and 2000 Provided TOP (n) with constant Provided TOP (n) with constant expression expression

Only for SELECTOnly for SELECT

SQL Server 2005 SQL Server 2005 Provides TOP (<expression>)Provides TOP (<expression>)

Also available on Also available on INSERT/UPDATE/DELETEINSERT/UPDATE/DELETE

TRY/CATCHTRY/CATCHTOPTOP

PIVOTPIVOTColumns, columns everywhere Columns, columns everywhere …. …. I need rows! I need rows! (And Access can already do it!)(And Access can already do it!)

PIVOTPIVOT

PIVOTPIVOTTransforms a set of rows to columnsTransforms a set of rows to columns

Similar to Access TRANSFORMSimilar to Access TRANSFORM

Useful for data analysisUseful for data analysis

Useful for open shemasUseful for open shemasE.g. Products with different propertiesE.g. Products with different properties

UNPIVOTUNPIVOTReverse operation of PIVOTReverse operation of PIVOT

PIVOTPIVOT

MakeMake YearYear SalesSales

HondaHonda 19901990 20002000

AcuraAcura 19901990 500500

HondaHonda 19911991 30003000

AcuraAcura 19911991 600600

MakeMake 19901990 19911991

HondHondaa

20002000 30003000

AcuraAcura 500500 600600

SELECT *SELECT *FROM SalesTable FROM SalesTable PIVOT(SUM(PIVOT(SUM(SalesSales) ) FOR FOR YearYear IN IN ([1990], [1991])) s([1990], [1991])) s

Common Table Common Table ExpressionsExpressionsTemporary tables make some Temporary tables make some things so much easier to code– things so much easier to code– but that is so clunky… I need but that is so clunky… I need something better…something better…

Common Table Common Table ExpressionsExpressionsAs per SQL-99As per SQL-99

Syntax:Syntax:WITH WITH <<CTENameCTEName>> ( < ( <column-listcolumn-list> )> )ASAS( <( <CTECTE>)>)<<SELECT using CTESELECT using CTE>>

Both recursive and non-recursive formsBoth recursive and non-recursive formsNon-recursive:Non-recursive:

Tidy code, avoid temp tables, views, sub Tidy code, avoid temp tables, views, sub selectsselects

Recursive:Recursive:Rewrite queries with derived tables to be Rewrite queries with derived tables to be more readablemore readable

Simple CTESimple CTE--Average number of times a Product was ordered--Average number of times a Product was ordered--for all Products that appeared on an order--for all Products that appeared on an order--more than 50 times --more than 50 times WITH WITH SalesCTESalesCTE(ProductID, SalesOrderID)(ProductID, SalesOrderID)AS AS ((

SELECT ProductID, COUNT(SalesOrderID) SELECT ProductID, COUNT(SalesOrderID) FROM Sales.SalesOrderDetailFROM Sales.SalesOrderDetailGROUP BY ProductIDGROUP BY ProductID

))SELECT AVG(SalesOrderID) SELECT AVG(SalesOrderID) FROM FROM SalesCTESalesCTEWHERE SalesOrderID > 50WHERE SalesOrderID > 50

Recursive CTEsRecursive CTEs

Recursive, when <CTE> Recursive, when <CTE> references itselfreferences itselfRecursive form of CTERecursive form of CTE

<non-recursive SELECT><non-recursive SELECT>UNION ALLUNION ALL<SELECT referencing CTE><SELECT referencing CTE>

Recursion stops when 2Recursion stops when 2ndnd SELECT SELECT produces empty resultsproduces empty results

Initialize

Accumulate

EXAMPLE: “Org Chart” EXAMPLE: “Org Chart” No Recursive Queries No Recursive Queries DECLARE @RowsAdded int DECLARE @RowsAdded int

-- table variable to hold accumulated results -- table variable to hold accumulated results DECLARE @reports TABLE (empid nchar(5) primary key, empname nvarchar(50) NOT NULL, mgrid DECLARE @reports TABLE (empid nchar(5) primary key, empname nvarchar(50) NOT NULL, mgrid

nchar(5), title nvarchar(30), processed tinyint default 0) nchar(5), title nvarchar(30), processed tinyint default 0)

-- initialize @Reports with direct reports of the given employee -- initialize @Reports with direct reports of the given employee INSERT @reports INSERT @reports SELECT empid, empname, mgrid, title, 0 SELECT empid, empname, mgrid, title, 0 FROM employees FROM employees WHERE empid = ‘12345’WHERE empid = ‘12345’

SET @RowsAdded = @@rowcount SET @RowsAdded = @@rowcount

-- While new employees were added in the previous iteration -- While new employees were added in the previous iteration WHILE @RowsAdded > 0 WHILE @RowsAdded > 0 BEGIN /*Mark all employee records whose direct reports are going to be found in this BEGIN /*Mark all employee records whose direct reports are going to be found in this

iteration with processed=1.*/ iteration with processed=1.*/ UPDATE @reports UPDATE @reports SET processed = 1 SET processed = 1 WHERE processed = 0 WHERE processed = 0

-- Insert employees who report to employees marked 1. -- Insert employees who report to employees marked 1. INSERT @reports INSERT @reports SELECT e.empid, e.empname, e.mgrid, e.title, 0 SELECT e.empid, e.empname, e.mgrid, e.title, 0 FROM employees e, @reports r FROM employees e, @reports r WHERE e.mgrid=r.empid and e.mgrid <> e.empid and r.processed = 1 WHERE e.mgrid=r.empid and e.mgrid <> e.empid and r.processed = 1

SET @RowsAdded = @@rowcount SET @RowsAdded = @@rowcount /*Mark all employee records whose direct reports have been found in this /*Mark all employee records whose direct reports have been found in this

iteration.*/ iteration.*/ UPDATE @reports SET processed = 2 WHERE processed = 1 UPDATE @reports SET processed = 2 WHERE processed = 1

ENDEND

EXAMPLE: “Org Chart” EXAMPLE: “Org Chart” With Recursive QueriesWith Recursive QueriesWITH WITH EmpCTEEmpCTE(empid, empname, mgrid)(empid, empname, mgrid)ASAS( ( SELECT empid, empname, mgridSELECT empid, empname, mgrid FROM EmployeesFROM Employees WHERE empid = ‘12345’WHERE empid = ‘12345’ UNION ALLUNION ALL SELECT E.empid, E.empname, E.mgridSELECT E.empid, E.empname, E.mgrid FROM Employees AS E JOIN EmpCTE AS MFROM Employees AS E JOIN EmpCTE AS M ON E.mgrid = M.empidON E.mgrid = M.empid))SELECT * FROM SELECT * FROM EmpCTEEmpCTE

DDL TriggersDDL TriggersI need to control how people I need to control how people change the schema of my change the schema of my database…how?database…how?

DDL TriggersDDL Triggers

Extension of traditional triggers for Extension of traditional triggers for DDL eventsDDL events

Triggering events include all DDL Triggering events include all DDL statementsstatements

CREATE_TABLE, ALTER_PROCEDURE, CREATE_TABLE, ALTER_PROCEDURE, DROP_LOGIN, etc.DROP_LOGIN, etc.

Scoping at Database and Server Scoping at Database and Server levelslevels

Event data available inside trigger Event data available inside trigger through eventdata() functionthrough eventdata() function

DDL DDL TriggersTriggers

-- Log tables being dropped to dropLog-- Log tables being dropped to dropLog

CREATE TABLE dropLog (id INT PRIMARY KEY IDENTITY, CREATE TABLE dropLog (id INT PRIMARY KEY IDENTITY, logTxt VARCHAR(MAX)) logTxt VARCHAR(MAX))

GO GO

-- Trigger to log drops -- Trigger to log drops

CREATE TRIGGER ddlDropCREATE TRIGGER ddlDrop

ON DATABASE ON DATABASE

AFTER DROP_TABLEAFTER DROP_TABLE

AS AS

INSERT INTO dropLog INSERT INTO dropLog

VALUES('A table has been dropped') VALUES('A table has been dropped')

DML with OutputDML with OutputIt is a shame that I need to do It is a shame that I need to do two things to find out what rows I two things to find out what rows I UPDATE or DELETE… is there a UPDATE or DELETE… is there a better way?better way?

DML with OUTPUTDML with OUTPUTOUTPUT clause for DMLOUTPUT clause for DMLAbility to return rows as part of DML Ability to return rows as part of DML operationsoperationsUse “Inserted” and “Deleted” columns Use “Inserted” and “Deleted” columns available to get pre- and post-update available to get pre- and post-update valuesvaluesOption to store returned rowsOption to store returned rows

OUTPUT… INTO…OUTPUT… INTO…

DECLARE @MyTableVar TABLE (orderId int)DECLARE @MyTableVar TABLE (orderId int)

-- Update all 'unprocessed' to 'processed’-- Update all 'unprocessed' to 'processed’

UPDATE OrdersUPDATE Orders

SET status='processed'SET status='processed'

OUTPUT INSERTED.orderId INTO @MyTableVarOUTPUT INSERTED.orderId INTO @MyTableVar

WHERE status='unprocessed'WHERE status='unprocessed'

RankingRankingI want to rank my data based on I want to rank my data based on criteria … no, I don’t just mean criteria … no, I don’t just mean order it…order it…

Ranking Functions: Ranking Functions: ScenariosScenarios

Data analysis (RANK, DENSE_RANK, Data analysis (RANK, DENSE_RANK, NTILE)NTILE)

Ability to generate ranks based on Ability to generate ranks based on different criteria in same querydifferent criteria in same query

Ability to separate presentation order Ability to separate presentation order from ranksfrom ranks

Paging using ROW_NUMBERPaging using ROW_NUMBERCommon scenario for walking through Common scenario for walking through result setsresult sets

DML with DML with OutputOutputRANKRANKSELECTSELECT

RANKRANK()OVER(ORDER BY City)()OVER(ORDER BY City)as RANK,as RANK,

RANKRANK()OVER(()OVER(PARTITIONPARTITION BY City ORDER BY LastName) BY City ORDER BY LastName) as PART_RANK,as PART_RANK,

DENSE_RANKDENSE_RANK() OVER(ORDER BY City) () OVER(ORDER BY City) as DENSE_RANK, as DENSE_RANK,

ROW_NUMBERROW_NUMBER() OVER(ORDER BY City) () OVER(ORDER BY City) as ROW_NUM,as ROW_NUM,

NTILENTILE(4) OVER(ORDER BY City) (4) OVER(ORDER BY City) as NTILE_4,as NTILE_4,

LastName, LastName, FirstName, FirstName, City City

FROM Employees FROM Employees ORDER BY City, LastNameORDER BY City, LastName

SNAPSHOT IsolationSNAPSHOT IsolationHmmm …… I need more work to Hmmm …… I need more work to happen in parallel. How?happen in parallel. How?ORORI am migrating from Oracle and I am migrating from Oracle and want to do as few changes as want to do as few changes as possiblepossible

Snapshot IsolationSnapshot Isolation

SQL Server 2000 Transaction isolation SQL Server 2000 Transaction isolation levelslevels

Read UncommittedRead UncommittedRead CommittedRead CommittedRepeatable ReadRepeatable ReadSerializableSerializable

SQL Server 2005 adds...SQL Server 2005 adds...SnapshotSnapshotTwo flavours:Two flavours:

Statement = READ_COMMITTEDStatement = READ_COMMITTEDTransaction = SERIALIZABLETransaction = SERIALIZABLE

Snapshot IsolationSnapshot Isolation

Increased data availability for Increased data availability for readread applicationsapplications

Allows non-blocking consistent reads in an OLTP Allows non-blocking consistent reads in an OLTP environmentenvironmentWriters don’t block readersWriters don’t block readersReaders don’t block writersReaders don’t block writers

Permits writes, which can cause conflictsPermits writes, which can cause conflictsBUT…includes mandatory conflict detectionBUT…includes mandatory conflict detection

Snapshot Isolation trades:Snapshot Isolation trades:cost of concurrency (locking exclusion) for cost of concurrency (locking exclusion) for cost of CPU & I/O to construct transaction cost of CPU & I/O to construct transaction consistent view and read over versionsconsistent view and read over versions

Snapshot Isolation: Snapshot Isolation: ScenariosScenarios

Reporting and ad-hoc queries running Reporting and ad-hoc queries running concurrently with OLTP concurrently with OLTP Read-mostly database with relatively Read-mostly database with relatively few writesfew writesApplications prone to deadlocks may Applications prone to deadlocks may behave betterbehave betterConsistent aggregates (e.g., AVG, Consistent aggregates (e.g., AVG, SUM)SUM)Migration from versioning databasesMigration from versioning databases

E.g. Oracle to SQL ServerE.g. Oracle to SQL Server

Large data typesLarge data typesI hate chunking data!!!! Please, I hate chunking data!!!! Please, tell me there is a better way…tell me there is a better way…

……(max) Type(max) TypeText/nText and Image have problemsText/nText and Image have problems

Most varchar functions don’t work (e.g. Most varchar functions don’t work (e.g. Like)Like)

Not directly updateableNot directly updateable

Not allowed as procedure variablesNot allowed as procedure variables

Extension to varchar, nvarchar, Extension to varchar, nvarchar, varbinary up to 2GBvarbinary up to 2GB

CREATE TABLE myTableCREATE TABLE myTable

(Id int, (Id int,

Picture varbinary(max))Picture varbinary(max))

XMLXML

I have rich XML outside the db – I have rich XML outside the db – and strings inside the db and strings inside the db . Is . Is there a better way?there a better way?

XML Support in SQL 2005XML Support in SQL 2005

SQL Server 2005 as an XML store SQL Server 2005 as an XML store Durable, recoverable, consistentDurable, recoverable, consistent

Mature management toolsMature management tools

Integration with existing relational dataIntegration with existing relational data

SQL Server 2005 as an XML sourceSQL Server 2005 as an XML sourceAccessible over HTTP and SOAPAccessible over HTTP and SOAP

Exposes all data as XMLExposes all data as XML

Can be programmed just like a web Can be programmed just like a web serviceservice

XML data typeXML data typeNative SQL typeNative SQL typeUse for column, variable or parameterUse for column, variable or parameter

CREATE TABLE docs (id INT PRIMARY KEY, xCol CREATE TABLE docs (id INT PRIMARY KEY, xCol XML NOT NULL)XML NOT NULL)

Well-formed and validation checksWell-formed and validation checksOptional XML Schema enforcementOptional XML Schema enforcementBehaviours allow XQuery and Behaviours allow XQuery and extensionsextensions

Exist, Query, Modify, ValueExist, Query, Modify, Value

Native XML StoreNative XML StoreXML Data TypeXML Data Type

Native XML StoreNative XML StoreXML IndexXML Index

Create XML index on XML columnCreate XML index on XML columnCREATE PRIMARY XML INDEX idx_1 ON T (xCol)CREATE PRIMARY XML INDEX idx_1 ON T (xCol)

CREATE XML INDEX idx_2 ON T(xCol)CREATE XML INDEX idx_2 ON T(xCol)

USING XML INDEX idx_1 FOR PATHUSING XML INDEX idx_1 FOR PATH

Creates indexes on structure (PATH), Creates indexes on structure (PATH), properties (PROPERTY) values (VALUE)properties (PROPERTY) values (VALUE)

Speeds up queriesSpeeds up queriesEntire query is optimisedEntire query is optimised

Same industry leading cost based optimiserSame industry leading cost based optimiser

Indexes are used as availableIndexes are used as available

XMLXML

SummarySummary

T-SQL is alive and kickingT-SQL is alive and kicking

There is LOTS more than covered todayThere is LOTS more than covered today

Use T-SQL and CLR judiciouslyUse T-SQL and CLR judiciously

Links:Links:Sample chapter Sample chapter

http://www.yukonxml.com/chapters/aw/sql2005dehttp://www.yukonxml.com/chapters/aw/sql2005dev/v/

““A First Look at SQL Server 2005 for Developers”A First Look at SQL Server 2005 for Developers”Addison Wesley: Bob Beauchemin, Niels Berglund, Dan Addison Wesley: Bob Beauchemin, Niels Berglund, Dan SullivanSullivan

Books OnlineBooks Onlinehttp://blogs.msdn.com/ericnelhttp://blogs.msdn.com/ericnel - click on - click on “Download Books Online” on right“Download Books Online” on right

© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.