capturing changed data using sql server 2008 - srikant jahagirdar
TRANSCRIPT
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
1/19
Change Data Capture
Srikant JahagirdarDevelopment LeadMSIT - Microsoft
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
2/19
Session Objectives And Takeaways
Session Objective(s):
Describe Change Data Capture architecture
Build a data tracking solution using Change DataCapture
Key Takeaways
Best practices and recommendations for change datacapture solutions
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
3/19
Tracking Methods
The traditional tracking methods provides differentlevels of change information and can be intrusive
Trigger based trackingTimestamp columns
Join queriesAdditional table to track deletes
Scenario requirements vary and one size fits allmay not be the best approach when buildingtracking solutions
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
4/19
Typical Scenario
Right time BI is becoming more critical for businesses
Data volume is increasing and batch windows aredecreasing
Efficient ETL through incremental data load is keyto reducing the overall ETL time
Upfront information about what changed helpsbuild efficient ETL solutions
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
5/19
SQL Server 2008 TrackingFeatures
Change Data CaptureProvide rich change information harvested fromdatabase log
Change TrackingLight weight tracking providing real-time changedetection
SQL AuditingLow impact tracking providing auditing information
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
6/19
Value Proposition
Change Data Capture provides valuable changeinformation about DML changes on a tableefficiently and close to real-time
Eliminates expensive techniques:
user triggers, timestamp columns, expensive join queries
Can provide answers to a variety of key questionsWhat are ALL the changes that happened between 12:00AM and12:00PM?I want to get a transactional consistent set of all changes thathappened in Order table and Order_Detail since 12:00PMWhich column changed?Was the change an insert or an update?
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
7/19
Functional Overview
Configuration
Tracking Mechanism
Querying for changes
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
8/19
Configuration
Change Data Capture enabled at two levelsDatabase - sp_cdc_enable_dbIndividual tables - sp_cdc_enable_table
sysadmin privilege required to enable CDC at thedatabase levelRequires dbo privilege to enable CDC at the table level
System metadata available to track configuration
Supported only on enterprise edition
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
9/19
Tracking MechanismAsynchronous log reading technology populatechange tables
Provide the flexibility to stop capture to minimize impact onsourceBuild on top of the replication log reader technology
Provides guarantees on transactionalconsistency in case of failures
Tracking infrastructure allows time-based rangequeries
All changes are trackedCaptures the before and after image of updates from the log
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
10/19
Store_Num Order_Status Store_desc
1001 A Mystore1
1002 B Mystore2
STORE TABLE
Store_Id Order_Status Store_desc Operation Column Info TransactionTimestamp
1001 H Mystore1 Insert 0x00 1
1002 D Mystore2 Insert 0x00 2
1001 A Mystore1 Update 0x10 3
1002 B Mstore2 Update 0x10 4
Lo gBased
CaptureJob
Db Log
Store_Id Order_Status Store_desc Operation Column Info TransactionTimestamp
C DC
A P I s
Select * fromfn_cdc_get_net_changes_*(starttime,endtime)
DW
CDC ConsumerSSIS Package
CleanupJob
STORE Change Table
Insert 1001 ,HInsert 1002, D
Update 1001, AUpdate 1002, B
Store_Num Order_Status Store_desc
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
11/19
Capturing Changes
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
12/19
Querying for ChangesTwo table valued functions (TVF) for eachchange tableAllows bounded range based queries
Row filter option provides filtering of result setConsistent result set guaranteed when using time-based ranges
All changes TVFUseful in extracting all changes very efficientlyOptionally provide before and after image of update
Net changes TVFAccurately identifies net operation to a row in a rangeProvides information about which column(s) changed for the netchange rowMore expensive query depending on filter options
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
13/19
Querying for Changes
TVF uses LSN to specify the rangeselect * from cdc.fn_cdc_get_all_changes_Customer(@from_lsn, @to_lsn, 'all')
Log sequence number (LSN) is a binary(10) typeYuk!!!......Why cant I use date time to specify the range?
Time to LSN mapping function enables range tobe specified using time
Wrapper function generator provides an easy wayto deploy time based enumeration functions
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
14/19
Querying for ChangesAdvanced system functions provide mapping andother functionality
Easily determine the lowest and the highest watermark forchanges available in the change tablesDetermine which column changed given a bitmask
Security ModelAccess to change data controlled through TVFRequire access to all tracked columns in the base table at aminimumCDC role provide additional layer of security
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
15/19
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
16/19
Best Practices and RecommendationsChange tracking Solution
Use the change tracking that suits your applicationrequirement
PerformanceUse separate file-groups for change tracking tableTrack only the required columnsWrites to the change table are logged
Querying change dataMake sure that the fromLSN and toLSN are in the lowend and high end of change tracking windowsys.sp_cdc_generate_wrapper_function
-
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
17/19
Resources
http://www.databasejournal.com/features/ms
sql/article.php/3720361
http://www.databasejournal.com/features/ms
sql/article.php/3725476
http://www.databasejournal.com/features/mssql/article.php/3725476http://www.databasejournal.com/features/mssql/article.php/3725476http://www.databasejournal.com/features/mssql/article.php/3720361http://www.databasejournal.com/features/mssql/article.php/3720361 -
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
18/19
Contact
mailto:[email protected] -
8/8/2019 Capturing Changed Data Using SQL Server 2008 - Srikant Jahagirdar
19/19
2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.MICROSOFT MAKES NO W ARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.