DistributingDistributing Data Techniques withData Techniques withDistributing Distributing Data Techniques with Data Techniques with SQL Server 2008 R2SQL Server 2008 R2QQ
Speaker: Paul Bertuccid b hi hFounder, Database Architechs
Silicon Valley SQL Server User GroupOctober 19, 2010
Mark Ginnebaugh, User Group Leader, [email protected]
Distributing Data TechniquesDistributing Data Techniques with SQL Server 2008 R2
Copyright 2010 All Rights Reserved
Agendag• Determining where data is needed
• What SQL Server options do you have
• How to get it there
• Work within your constraints Security
Regulatory (like PII…)
Ad i i t ti l Administratively
Data volatility/sizes
SLA’s you must deliver
Disaster Recovery/Business Continuity
Skills/Knowledge of your team
Resources (like $$’s, so on)
Reasons to Distribute DataReasons to Distribute Data
19FIG02OLTP DB
SQL Server 2008
PrimaryOLTP
Rpt DB
SQL Server 2008
ReportingServer
Reporting/ODS
OLTP DB pt
SQL Server 2008
EuropeEnabling/Partitioning
xyz DB
SQL Server 2008
USA(Headquarters)
xyz DB
Server
SQL Server 2008
AsiaServerxyz DB
xyz DB
Server
xyz DB
SQL Server 2008
North AmericaRegion
xyz DB
SQL Server 2008
EuropeRegion
Regionalization
(multiple owners)y y
SQL Server 2008
Primary
SQL Server 2008
Hot Spare
(multiple owners)
Failover
xyz DB
Primary
xyz DB
Hot Spare(Fail-over)
Distributing periodic updates toData Marts from a “master” Data Warehouse
DataMart
SQL Server2008
MasterData Warehouse
DataDataMart
DataMart
SQL Server2000
SQL Server2005
ORACLE
Enabling a Data Warehousefrom Legacy Systems
MasterData Warehouse
SQL Server2005
SQL Server2008
HeterogeneousInitialization of tables HeterogeneousSubscriber
Initialization of tablesfor a heterogeneous Subscriber
in Replication
ORACLE
Publisher
Subscriber
SQL Server 2008
SQL Server2008
Subscriber
SQL Server 2005
Pulling in Sales Order informationfrom disparate sources
for import into SQL Server 2008Sales Force Automation
Application for import into SQL Server 2008
Access
Application
Access
SQL Server2008Excel 2008Excel
Data Mart
SQL Server2008
OLTP Database
2008
Export from the OLTP Database
ExcelSQL Server
2008
Export from the OLTP Database to a business intelligence
Data Mart & Excel
Connection
SQL Server2008
SQL Server2008
Results Results
2008
Client Local Server2008
Remote Server
Decision‐Tree Path Traversal
I t ti
DatabaseMirroring
With S h tIntegrationServices
With Snapshots
1a2c3 1a2d3 Detach/
1a2
1a2c3
1 2 3
1a2b3
1a2d3
1a2e3
ab c d
e
BCP
Detach/Attach
1 LogShippingSync
Framework
1b2bac
de
1c2
1e2
1a2a3
Framework 1c21d2
Backup/Restore
Data Replication
LinkedServersServers
DataReplication
LogShipping
Database Mirroringwith Snapshots
LinkedServers
SSIS BCPBackup/Restore
Detach/Attach
SyncFramework
LargeData
HighVolatilityVolatility
On Line
Many Targets
LowLatency
Transformation/Transformation/Filtering
Data ReplicationData Replication
Central Publisher(default option)SQL Server
2008SQL Server
20052008
AdventureWorks2008
translog
2005
AdventureWorks2008
translogSQL Server
19FIG10
translog translogSQL Server2008
AdventureWorks2008
SubscriptionServer
SubscriptionServer
translogPublication
Server
Oracle
SQL Server2000
distribution
DistributionServer
AdventureWorks2008
AdventureWorks2008
translog
Server
Subscription Subscription
1515
SubscriptionServer Server
19FIG11
Central PublisherRemote DistributorSQL Server
2008SQL Server
2005
SQL Server2008
2008
AdventureWorks2008
translog
2005
AdventureWorks2008
translog2008
AdventureWorks2008
translog
translogSubscription
Server
translogSubscription
Server
translogPublication
Server
distribution
SQL Server2008
Oracle
SQL Server2000
Remote DistributionServerAdventure
Works2008
Subscription
AdventureWorks2008
translogSubscriptionSubscription
Server Server
Publishing Subscriber
SQL Server2008
SQL Server2005
19FIG12
SQL Server2008
AdventureWorks2008bl
icat
ion
Serv
er
2008
AdventureWorks2008
translog
2005
AdventureWorks2008
translogtranslog
distribution
PuS
istr
ibut
ion
Serv
er
translogSubscription
Server
translogSubscription
Server
Dpt
ion
erSQL Server
2008
tion
erSu
bscr
iSe
rve
AdventureWorks2008
translogOracle
SQL Server2000
2008
Publ
icat
Serv
e translogAdventureWorks2008
Subscription
AdventureWorks2008
translogSubscriptionio
nr
1717
SubscriptionServer Serverdistribution
Dis
trib
uti
Serv
er
19FIG13 CentralSQL Server
2008tio
ner
SQL Server2008
tion
er19FIG13 CentralSubscriber
AdventureWorks2008
translogPubl
ica
Serv
eio
nr
AdventureWorks2008
translogPubl
icat
Serv
eon
SQL Server 2008
SubscriptionServer
distribution
Dis
trib
uti
Serv
er
distribution
Dis
trib
uti
Serv
er
AdventureWorks2008
Server
SQL Server2008
atio
nve
r
SQL Server2008
atio
nve
r
AdventureWorks2008
translogPubl
icSe
rvut
ion
er
AdventureWorks2008
translogPubl
ica
Serv
utio
ner
distribution
Dis
trib
uSe
rve
distributionD
istr
ibu
Serv
e
19FIG14
Multiple Publishersof a Single Table
on onSQL S SQL S19FIG14
on rSu
bscr
iptio
Serv
er
AdventureWorks2008 on
Subs
crip
tioSe
rver
AdventureWorks2008
SQL Server2008
SQL Server2008
Publ
icat
ioSe
rver translog
Publ
icat
ioSe
rver translog
distribution
Dis
trib
utio
nSe
rver
distribution
Dis
trib
utio
nSe
rver
Territory 1
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDateCustomers Table
Territory 2
1345 1 AW1345 I X69G9.. 120203
1356 2 AW1356 I W211G.. 051605
2354 1 AW2354 S 7SQ78K.. 106705
3346 2 AW3346 I W12DV.. 022305
7643 2 AW7643 S WZ8R4.. 122205
7901 2 AW7901 I S2345X.. 041506
8921 1 AW8921 I RT66Y.. 031206
U d ti
19FIG15
Updating Subscriber
SQL Server2008
AdventureWorks2008bl
icat
ion
Serv
er
SQL Server Queue(Queue Reader Agent)
Works2008
translog
distributionPu
b Sst
ribut
ion
Serv
er
Update Customersset AccountNumber = null
where CustomerID = 12345
Dis S
SQL Server2008
Adventure
SQL Server2008
AdventureW k 2008Works2008
translogSubscription
Server
Works2008
translogSubscription
Server
19FIG16SQL Server
2008SQL Server
200819FIG16
Peer-to-PeerAdventureWorks2008
translogPublication
AdventureWorks2008
translogPublicationPublication
ServerPublication
Server
North American Asiao e caServer
s aServerSQL Server 2008
distribution
Distribution ServerPeer Originator id: 1 Peer Originator id: 2
C t ID T it ID A tN b C T id M difi dD tCustomers Table
1345 1 AW1345 I X69G9.. 120203
1356 2 AW1356 I W211G.. 051605
2354 1 AW2354 S 7SQ78K.. 106705
3346 2 AW3346 I W12DV 022305
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate
3346 2 AW3346 I W12DV.. 022305
7643 3 AW7643 S WZ8R4.. 122205
7901 5 AW7901 I S2345X.. 041506
8921 4 AW8921 I RT66Y.. 031206
Publisher
P bli ti
19FIG05AdventureWorks2008
AW_VerticalPublication
19FIG05SQL Server
2008
translog CustomerV (Article)
1345 1 AW1345 I X69G9.. 120203
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate
1345 1 I
CustomerID TerritoryID CustomerType
1356 2 AW1356 I W211G.. 051605
2354 1 AW2354 S 7SQ78K.. 106705
3346 2 AW3346 I W12DV.. 022305
7643 3 AW7643 S WZ8R4.. 122205
1356 2 I
2354 1 S
3346 2 I
7643 3 S
7901 5 AW7901 I S2345X.. 041506
8921 4 AW8921 I RT66Y.. 031206
7901 5 I
8921 4 I
Publisher
AW_HorizontalPublication
SQL Server
AdventureWorks2008
translog
19FIG06CustomerH (Article)
1345 1 AW1345 I X69G9.. 120203
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate
SQL Server2008
1345 1 AW1345 I X69G9.. 120203
1356 2 AW1356 I W211G.. 051605
2354 1 AW2354 S 7SQ78K.. 106705
3346 2 AW3346 I W12DV.. 022305
7643 3 AW7643 S WZ8R4.. 122205
Only these Rows!7643 3 AW7643 S WZ8R4.. 122205
7901 5 AW7901 I S2345X.. 041506
8921 4 AW8921 I RT66Y.. 031206
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate
2354 1 AW2354 S 7SQ78K.. 106705
7643 3 AW7643 S WZ8R4.. 122205
yp g
Publisher
19FIG07Ad t
AW_H_and_V
Publication
SQL Server
AdventureWorks2008
translog07FIG08
CustomerHV (Article)
1345 1 AW1345 I X69G9.. 120203
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate
SQL Server2008
1345 1 AW1345 I X69G9.. 120203
1356 2 AW1356 I W211G.. 051605
2354 1 AW2354 S 7SQ78K.. 106705
3346 2 AW3346 I W12DV.. 022305
7643 3 AW7643 S WZ8R4.. 122205
Only these Columns and these Rows!
7643 3 AW7643 S WZ8R4.. 122205
7901 5 AW7901 I S2345X.. 041506
8921 4 AW8921 I RT66Y.. 031206
CustomerID TerritoryID CustomerType
2354 1 S
7643 3 S
y CustomerType
AW_H_and_J (Publication)
19FIG08
07FIG09CustomerH (Article)
OrdersHJ (Article)
1345 1 AW1345 I X69G9 120203
CustomerID TerritoryID AccountNumberCustomerType rowguid ModifiedDate Only the orders for1345 1 AW1345 I X69G9.. 120203
1356 2 AW1356 I W211G.. 051605
2354 1 AW2354 S 7SQ78K.. 106705
3346 2 AW3346 I W12DV.. 022305
7643 3 AW7643 S WZ8R4 122205
2354
7643
CustomerID
for customer type “S”
SalesOrderID RevisionNumber OrderDate DueDate Status CustomerID
7643 3 AW7643 S WZ8R4.. 122205
7901 5 AW7901 I S2345X.. 041506
8921 4 AW8921 I RT66Y.. 031206
7643
join
43659 1 7/1/2001 7/13/2001 5 1345
43660 1 7/1/2001 7/13/2001 5 1356
43661 1 7/1/2001 7/13/2001 5 2354
43662 2 7/1/2001 7/13/2001 5 3346SalesOrderID OrderDate CustomerID
2525
43663 1 7/1/2001 7/13/2001 5 7643
43664 1 7/1/2001 7/13/2001 5 7901
43665 1 7/1/2001 7/13/2001 5 8921
43661 7/1/2001 2354
43663 7/1/2001 7643
Publisher SubscriberAW (P bli ti )
19FIG17
SubscriberSQL Server
2008
Adventurecatio
nrv
er
AW (Publication)
Customers (Article)SQL Server
2008AdventureWorks2008
translog
di t ib ti
Publ
icSe
rbu
tion
rver PUSH
2008
AdventureWorks2008
translog
PUSH: initiated by Publisher
distribution
Dis
trib
Ser
SubscriptionServer
PULL(agent will be on the Publisher if local distribution,
or on the Distributor if remote distribution) PULL: initiated by Subscribery
(agent will be on the subscriber)
Distributing DataData Access Latency Autonomy Sites(locations) Frequency Network Machines Owner Other REPLICATIONDistributing Data(locations)
Read OnlyReporting short high many high fast/
stablen
server/site1 OLTP
site
Each site only needs regionaldata
Central PublisherTransactional replfilter by region
Read OnlyReporting long high many low fast/
stablen
server/site1 OLTP
site
Each site only needs regionaldata
Central PublisherSnapshot replfilter by region
Read Mostly fast/ n Regional updates Central Publisheread ost yA few updates short high < 10 medium fast/
stablen
server/site1 OLTP
site
g pon one table Transactional repl
Updating Subs
Read MostlyA few updates medium high < 10 medium slow/
unreliabn
server/siteAll
update
Regional updateall tables
Central PublisherMerge repl
Read equalEqual updates short high < 10 medium fast/
stable1
server/siteAll
update
Regional updateall tables
Peer-to-PeerTransactional
repl
Inserts (new orders) short high many high fast/
stable1
server/site
1reportsite
Each site only needs regionaldata
Central SubscriberTransactional repl
/ f / Central PublisherHot/WarmSpare
Veryshort
high < 2 high fast/stable
1server/site
1 OLTPsite
Fail-overCentral PublisherRemote DistributorTransactional repl
Log ShippingYes, it is still here..,
Log Shipping
SQL Server 2008
“Source”
SQL Server 2008
“Destination”
T LCallOne DB
CallOne DB
Q
SecondaryServertranslog
TxnLogbackupsPrimary
Server
CallOne DB
\Backup\CallOne_tlog_200405141120.TRN
T L \LogShare\CallOne tlog 200405141120 TRN TxnLogpped
wer
TxnLogCopies
\LogShare\CallOne_tlog_200405141120.TRN TxnLog Restores
SQL S 2008
“Monitor”
Last
log
ship
Del
ay A
nsw
MSDB DB
SQL Server 2008
MonitorServer
Delay between logs loaded
Delay Answer
Server
Don’t build on this, it is being deprecated !
Database Mirroringwith Snapshotswith Snapshots
Network
Client Client ClientClient
SQL Server 2008 SQL Server 2008Mi S
A
DB MirroringAdventure Works DB
Mirror Server
t l
PrincipalServer
Adventure Works DBB
DD
translog translogC
SQL Server xyz
Witness
3131
MSDB DBt ess
Server
10SQL Server 2008 SQL Server 2008
Mirror ServerPrincipal
AdventureWorksDB
translog
Server
Database
AdventureWorksDB
translogDatabase
SQL Server 2008
SnapshotDatabase Snapshot
node
m
irror
.
MSDB DBWitnessServer
this
ser
ver n
com
es th
e m
If t
bec
3232
01SQL Server 2008
01AdventureWorks DB
translog
SourceServer
D t b
CREATE DB ‘xyz’AS SNAPSHOT OF
AdventureWorks
Database Snapshot
--- Reverting --- if needed ---Reverting if needed RESTORE DB
AdventureWorksFROM DATABASE_SNAPSHOT = ‘xyz’
Source DataPPages
Source
02SQL Server 2008
SQL SparseServer
AdventureWorksDB
SnapshotAdventureWorks
DB
FilePages
DB
Empty Sparse fileof a Snapshot just created
(no updates to original
3434
( p gdata pages have occurred yet)
SourceData
PPages
Source
03SQL Server 2008
SQL SparseServer
AdventureWorksDB
SnapshotAdventureWorks
DB
FilePages
DB
System Catalog Copy of originalSystem Catalogof changed pages
Copy of originalpages for snapshot
only when a page is changed(Copy-on-write)
3535
(Copy-on-write)
Source DataPages
Snapshot Users
PagesSELECT …..data……. FROM AdventureWorks
SNAPSHOT
04SQL Server 2008
SQLSServer
AdventureWorksDB
SnapshotAdventureWorks
DB
System Catalog SparseSystem Catalogof changed pages
SparseFile
Pages
3636
10SQL Server 2008 SQL Server 2008
Mirror ServerPrincipal
AdventureWorksDB
translog
Server AdventureWorksDB
translogDatabase
SQL Server 2008
Database Snapshot
Database Snapshot
node
m
irror
.
MSDB DBWitnessServer
this
ser
ver n
com
es th
e m
If t
bec
3737
PH Topology With SnapshotsPH Topology With SnapshotsReal timeR ti
SQL Server 2008
ReportingUsers
Net
wor
k
SQL Server 2008
PrincipalServer
n
N
Adventure Works DB
SQL Server 2008
Mirror Server
Adventure Works DB
Active
ster
ed
pplic
atio
n
icat
ion
translog translog
PHPassive
Clu
s
OLT
P A
Rep
l
Database SnapshotPH
TopologyPrincipalServer
Network
3838SQL Server 2008
Point in timeReporting Users
DB Mirror and ReplicationSQL Server 2008 SQL Server 2008
Publisher
PrincipalServer
MirrorServer
SQL Server 2008
Subscriber
Distributor
SQL Server 2008
Principal
SQL Server 2008
Mirror
s bu o
SQL Server 2008
PrincipalServer
MirrorServer
WitnessServer
SQL Server 2008
Subscriber
3939
Integration Services
Excel
OLAP Data Warehouse
Access
MultidimensionalMultidimensionalCubeOLTP
Databases
Other Data Sources
OLTP
Other Data Sources
OLTPDatabases
SQL Server
Flat filesWeb Services
XML
P k
Task
Package
Task
Task
aine
rSource e
Con
taData Flow Task
Source
Transformation
D ti ti
Cac
he
Task
Destination
Task
Task
Data MartSSIS
e
SQL Server2008
OLTP Database
sk to
d
Agg
rega
te
AdventureWorks 2008
Tas
Join
and
UnleashedDataMart
AdventureWorks
Export from the OLTP Database
Excel.csv flat file
SQL Server2008
Export from the OLTP Database to a business intelligence
Data Mart & Excel/.csv flat file
Linked Servers
SQL Server
OLE DB Providerfor MS SQL Server
SQL Server6.5 – 2008
ODBC
OLE DB Providerfor ODBC
SybaseInformixDB2Othersde
r DLL
’s
ODBC
MS OLE DBP id
OthersOLE DB Provider
for JetExcel spreadsheetFoxPro DatabaseDbase fileAccess Database
Jet database
fileOLE DB Providerf O lE
DB
Pro
viSQL Server
2008
Provider for Oracle
ORACLE
OLE DB P id
OLE
OLE DB Providerfor Indexing Service Active Directory Services
Index ServerOthers
Client Tier Server Tier Remote Data Source Tier (can be “Linked”)
Might be to easy?Might be to easy?--------------------------------------------------------------- Selecting data from Excel spreadsheet via Linked Server --Selecting data from Excel spreadsheet via Linked Server -------------------------------------------------------------SELECT substring(CustomerID,1,5) as 'CustID', substring(ContactName,1,18) as ContactName,
b t i (Cit 1 10) 'Cit ' b t i (C t 1 15) 'C t 'substring(City,1,10) as 'City', substring(Country,1,15) as 'Country'FROM [Linked ExcelSW]...[SWCustomers$]ORDER BY ContactNamegogCustID ContactName City Country------ ------------------ ---------- ---------------PICCO Adam Greifer Los Angele USABLAUS John Monroe Concord USABLAUS John Monroe Concord USACHOPS Martin Sommer Berkeley USA
Microsoft Sync FrameworkMicrosoft Sync FrameworkA comprehensive synchronization platform that enables collaboration and offline access forenables collaboration and offline access for applications, services, and devices. It features technologies and tools that enable roaming, sharing, and taking data offline.
B i S F k d l b ildBy using Sync Framework, developers can build sync ecosystems that integrate any application with any data from any store that uses any y y yprotocol over any network. Mentioned here because of its replication‐like behavior for “occasionally connected” applicationsoccasionally connected applications.
OthersOthers
• BCPBCP
• Detach/Attach
k /• Backup/Restore
Copy on Write Technology
DB Mirroring
Network
Client Client ClientClient
SQL Server 2008 SQL Server 2008Mi S
A
Adventure Works DB
Mirror Server
t l
PrincipalServer
Adventure Works DBB
DD
translog translogC
SQL Server xyz
Witness
5050
MSDB DBt ess
Server
SourceData
PPages
Source
03SQL Server 2008
SQL SparseServer
AdventureWorksDB
SnapshotAdventureWorks
DB
FilePages
DB
System Catalog Copy of originalSystem Catalogof changed pages
Copy of originalpages for snapshot
only when a page is changed(Copy-on-write)
5151
(Copy-on-write)
Database Architechs
Paul BertucciPaul Bertucci Founder Database ArchitechsFounder Database Architechs Former Former Chief Data Architect Chief Data Architect –– VeritasVeritas & Symantec& Symantec C rrentl Chief Architect A todesk Currently Chief Architect - Autodesk Author of SQL Server 2000, 2005 & 2008/R2 Unleashed CoCo--Author of ADO.NET in 24 hours Author of ADO.NET in 24 hours Author MS SQL Server High AvailabilityAuthor MS SQL Server High Availability Author Sybase Performance & TuningAuthor Sybase Performance & Tuning Author Sybase Performance & TuningAuthor Sybase Performance & Tuning Author Sybase Physical DB Author Sybase Physical DB DesignDesign Author Chen Entity/Relationship ModelingAuthor Chen Entity/Relationship Modeling VeritasVeritas SQL Server Performance Series SQL Server Performance Series
[email protected]@dbarchitechs.com
To learn more or inquire about speaking opportunities, please contact:
Mark Ginnebaugh User Group LeaderMark Ginnebaugh, User Group [email protected]