metalogix replicator for sharepoint - wordpress.com · 4 supports sharepoint foundation 2010,...
Post on 17-Apr-2020
27 Views
Preview:
TRANSCRIPT
W H I T E PA P E R
m e t a l o g i x . c o m
INTRODUCTION............................................................................................. 2
REPLICATOR PERFORMANCE AND SCALABILITY FEATURES .................. 4
Replicator Multi-Engine Deployment ................................................. 7
Multi-Threaded Replication Queue Architecture ............................. 7
Customized Replication Interval....................................................... 8
ConfigurablePackageEventCount................................................8
ConfigurablePackageEventProcessingDuration..........................9
ReplicationPackageCompression..................................................9
HardwareNetworkCompressionSupport.....................................9
SoftwarePackageCompression...................................................10
RemoteDifferentialCompression(Rdc)........................................10
IndependentOutboundEventAndInboundEvent
ProcessingControls.......................................................................11
SelectiveStructureReplication.....................................................11
Rule-BasedItem-LevelContentReplication.................................11
SelectableReplicationEvents.......................................................12
SharedReplicationPackages.......................................................12
PackageDatabaseCaching..........................................................13
ConfigurableReplicationMonitorUpdateLevel...........................13
ImprovedInitialReplicationSupport.............................................14
MICROSOFT TECHNOLOGY CENTER PERFORMANCE
ANDSCALABILITYBENCHMARK...............................................................14
METALOGIX NETWORK TEST LAB PERFORMANCE BENCHMARKING...16
UNDERSTANDING REPLICATOR PERFORMANCE AND SCALABILITY.......................................................................17
MetalogixReplicatorPipeline.......................................17
ReplicatorPerformance...........................................18
OutboundEventProcessing..................................19
PackageTransfer......................................................20
InboundEventProcessing......................................21
Performance Monitoring .........................................22
Replicator Pipeline Performance Features .............24
ReplicatorScalability..............................................25
APPENDIX A – ABOUT WINDOWS REMOTE DIFFERENTIAL COMPRESSION ................................................................ 25
Copyright © 2012 Metalogix International GmbH. All rights reserved. Metalogix is a trademark of Metalogix International GmbH. StoragePoint is a registered trademark of BlueThread Technologies, Inc. Microsoft, Exchange Server, Microsoft Office, SharePoint, and SQL Server are registered trademarks of Microsoft Corporation.
Metalogix Replicatorfor SharePointENTERPRISE PERFORMANCE AND SCALABILITYVersion4.1June2012
CONTENTS
m e t a l o g i x . c o m W H I T E PA P E R
MetalogixReplicatorisanenterprise-classsolutionforreplicatingcontent,metadata,structure,security,look&feelandbusinessprocessesfromoneSharePointwebapplicationtoantherwebapplication.Replicatorversion4supportsSharePointFoundation2010,SharePointServer2010,OfficeSharePointServer2007andWindowsSharePointServices3.0.
MetalogixReplicatorsupportsthereplicationneedsofMicrosoft’s largest(andsmallest)SharePointcustomers.Version4providesadditionalperformanceandscalabilitycapabilitiesthesecustomerswillneedastheycontinuetogrowtheirDistributedSharePointenvironments.
The release ofMetalogix Replicator version 4 represents a new performance and scalabilitymilestone forSharePointreplicationsolutions.Version4includesnewfeaturesforsupportingmassivelyscalableSharePointreplicationnetworks:
SharedReplicationPackages,
PackageDatabaseCaching,
Improved Initial Replication Support, and
ImprovedRemoteDifferentialCompression(RDC).
Together,thesefeaturesbuildontheperformancefeaturesthathavebeenbuiltintoReplicatorovertheprevious3major releasesand7yearsofdevelopmenteffort.TheReplicatorversion4performance featuresetnowincludesthefollowingcomponents:
1.MultipleReplicationEngineDeployment
2. Multi-threaded Replication Queue Architecture
3.CustomizableReplicationInterval
4.ConfigurablePackageEventCount
5.ConfigurablePackageEventProcessingDuration
6.ReplicationPackageCompression
7.HardwareNetworkCompressionSupport
8.SoftwarePackageCompression
9.RemoteDifferentialCompression(RDC)
10.IndependentControlsforInboundProcessingandOutboundEventProcessing
11.SelectiveStructureReplication
INTRODUCTION
2
m e t a l o g i x . c o m W H I T E PA P E R
3
BENCHMARKING AT THE MICROSOFT TECHNOLOGY CENTER
12.Rule-basedItem-levelContentReplication
13.SelectableReplicationEvents
14.SharedReplicationPackages
15.PackageDatabaseCaching
16.ConfigurableReplicationMonitorUpdateLevel
Thisdocumentprovidesadditionaldetailed informationon theReplicatorperformance featureset.Thefirstsectiondescribeseachcomponentof theReplicatorperformance featureset.Thesecondsection includesin-depthdiscussionsofeachfeatureaswellasnewbenchmarktestingperformedattheMicrosoftTechnologyCenter inNewYork,NY and theMetalogixNetwork Test Lab. The final section of the document analysesReplicator from an operational point-of-view.
InAugust2010,withthesupportoftheMicrosoftTechnologyCenterinNewYork,NY,theMetalogixMassiveScalabilityTeamexecutedseveralperformanceteststoexerciseapre-releaseversionofMetalogixReplicatorversion4.TheScalabilityTeamconfiguredthelargestknownDistributedSharePoint2010environmentintheworldforthesetests:
61SharePointfarmsconnectedintoasinglereplicationnetwork
Microsoft Hyper-V server virtualization technology
65SharePoint2010Windows2008virtualservers
16Windows2008R2hostphysicalservers
128GBofphysicalRAM
1.2TBofphysicaldiskstorage
m e t a l o g i x . c o m W H I T E PA P E R
4
TheReplicator version4performance feature set builds uponmany components that havebeenbuilt intoReplicator over the previous 3major releases and 7 years of development effort. The new or significantlyimprovedversion4featuresaremarkedwithanasterisk.
Figure1.MicrosoftTechnologyCenter,NewYorkBenchmark:MetalogixConnectfor Replicator Visualization
REPLICATOR PERFORMANCE AND SCALABILITY FEATURES
Apre-releaseversionofMetalogixConnectforReplicatorwasusedtovisualizeandreportonthisconfiguration.Thefullydeployed61SharePointfarmenvironmentisdepictedinFigure1.
m e t a l o g i x . c o m W H I T E PA P E R
5
Feature DescriptionReplicator Multi-EngineDeployment
ReplicatorEnterpriseEditionenablestheReplicatorEnginetobedeployedonmultiplewebfront-endsineachfarm.Withmultiplewebfront-endshostingReplicatorEngineinafarm,Replicatorprovidesahigheravailabilityandhigherperformancereplicationsolution;inadditiontosupportingincreasedscalability.
Multi-Threaded Replica-tion Queue Architecture
Replicator’sReplicationEnginearchitectureisbasedonamulti-threadedreplica-tionqueuearchitecturethatsupportsparallelReplicationPackageprocessing.
CustomizableReplicationInterval
Replicatorsupportsseveralreplicationschedulingpolicies:Immediately,Interval,Daily,Weekly,MonthlyandManual.
ConfigurablePackageEvent Count
Bydefault,ReplicatorincludestheabilitytogrouporbatchmultipleReplicationEventsintoasingleReplicationPackageforprocessingandtransfertotheTargetWebAp-plication.ThisConfigurablePackageEventCountdeterminesthemaximumnumberofEventsthatcanbepackagedintoasingleReplicationPackage.ThisenablesReplica-tortobetunedfordifferentlevelsofreal-timereplication,wideareanetworkperfor-mancecharacteristicsandavailableInboundandOutboundEventProcessingmemoryand CPU processing resources.
ConfigurablePackageEvent Processing Duration
SimilartotheConfigurablePackageEventCountfeature,theConfigurablePack-age Event Processing Duration allows control over how many Replication Events are groupedorbatchedtogetherintoasingleReplicationPackagebasedonelapsedOutboundEventProcessingtime.ThissettingenablesReplicatorlocalserverresourcerequirementstobetunedforoptimalperformance.
ReplicationPackageCompression
Replicator supports several forms of software compression in addition to support for andcompatibilitywithvarietyofhardwarenetworkcompressiondevices.Replicatorcan use either software or hardware compression to reduce the amount of Replication Packagedatatransferredoverawideareanetwork.
SoftwarePackageCompression
ReplicatorsupportsZIPsoftwarepackagecompressionandacustomimplementationofMicrosoft’sRemoteDifferentialCompression(RDC)thatishighlyoptimizedfortheone-wayandbi-directionalreplicationofSharePointdata.
HardwareNetworkCompression Support
Whenhardwarenetworkcompressiondevices,bandwidthoptimizationappliances,ornetworkacceleratorsolutionsareavailable(suchastheRiverbed®Steelhead®Appli-ance),Replicatorcanbeconfiguredtominimizetheserverresourcesusedforsoftwarecompressionandmaximizetheeffectiveofthenetworkcompressiondeviceorappli-cation.
RemoteDifferentialCompression(RDC)*
Metalogix Replicator supports a custom implementation of Microsoft’s RemoteDifferentialCompression(RDC)technologythatMetalogixhasoptimizedtoreducethetotalamountofReplicationPackagedatathatneedstobetransferredbe-tweentheSourceWebApplicationtotheTargetWebApplicationduringreplication.
IndependentOutboundEventandInboundEventProcessing Controls
ReplicatorsupportsindependentcontrolsforOutboundEventProcessingandIn-boundEventProcessingattheWebApplicationandReplicationConnectionlevelsforaSourceWebApplicationandTargetWebApplication.Inaddition,replicationforaparticularWebApplicationcanbeconfiguredasone-wayorbi-directional.
m e t a l o g i x . c o m W H I T E PA P E R
6
Feature DescriptionSelective Structure Replication
Bydefault,manycustomersusewebapplication-to-webapplicationreplication – replicating the entire content of each site collection inonewebapplicationtothesecondwebapplication.SelectivestructurereplicationenablestheSharePointAdministratortoselectaspecificsubsetoftheSourceWebApplicationstructuretobereplicatedtotheTargetWebApplication.
Rule-basedItem-levelContent Replication
Rule-baseditem-levelcontentreplicationusestheMetalogixReplicatorRulesEngineto process custom rule sets to determine if an item in a particular list or document libraryshouldbereplicated.
SelectableReplicationEvents
ReplicatorsupportsselectablereplicationofchangesthatoccurinaSharePointWebApplication,SiteCollection,WebSite,ListorDocumentLibrary.Thedifferenttypesofindividual changes are called Replication Events. Replication Events are categorized intohigher-levelEventGroups.ByselectingwhichEventsneedtobereplicatedandwhichEventsdon’tneedtobereplicated,theSelectableReplicationEventsfeatureprovidestheSharePointAdministratorwithfine-grainedcontrolovertheEventsprocessingduringInboundEventProcessing,PackageTransferandOutboundEventProcessing; which in turn help improve overall Replicator performance.
Shared Replication Packages*
InReplicatorversion3,aseparateReplicationPackagewascreatedforeachoutboundReplicationConnection.TheSharedReplicationPackagefeatureinReplicatorversion4eliminatesthetimerequiredtocreateaseparatePackageforeachoutboundReplicationConnection–asinglesharedPackagecontainingthebatchofReplicationEventsiscreated.DependingonthenumberofEventsinthepackage,thesizeandtypeoftheSharePointchange,andthenumberofoutboundReplicationCon-nections,thiscansignificantlyreduceOutboundEventProcessingtimeandresources.
PackageDatabaseCaching*
InReplicatorversion4,uponacceptanceofthetransferofanInboundReplicationPackage,ReplicatorimmediatelycachesthemetadataforeachReplicationEventintheReplicatorConfigurationdatabase.ThisimprovestheperformanceoftheReplica-torEnginebyeliminatingtheneedtorepeatedlyaccesstheserializedPackagedata;especially in SharePoint farms that host multiple Replicator Engines.
ConfigurableReplicationMonitorUpdateLevel*
DuringnormalInboundEventProcessing,Replicator,runningonaTargetWebApplication,willmakeseveralcallstotheReplicatorWebServicerunningontheSourceWebApplicationtokeeptheMonitorReplicationreportsontheSourceWebApplicationascurrentaspossible.TohelpminimizetheQueuedItemupdatenet-worktrafficbetweentheSourceandTargetWebApplications,Replicatorversion4.1supportsanewReplicationConnectionpropertycalled“quietmode”.
Improved Initial ReplicationSupport*
ThenewBackupModefeatureprovidesamoreefficientmethodofreplicatingalargesitecollectionorhierarchyofwebsites.UsingBackupMode,ReplicatorusestheSharePoint import and export operations to create a single archive of the entire group ofwebsites,andqueuesthereplicationofthisarchiveasasingleReplicationEvent.
*NeworsignificantlyimprovedReplicatorversion4feature
m e t a l o g i x . c o m W H I T E PA P E R
7
ReplicatorEnterpriseEditionenablestheReplicatorEnginetobedeployedonmultiplewebfront-endsineachfarm.Withmultiplewebfront-endshostingReplicatorEngineinafarm,Replicatorprovidesahigheravailabilityandhigherperformance replicationsolution; inaddition tosupporting increasedscalability. Ifoneengine isstopped, the other engines are still operational.
Thefollowingisbasedonaversion4benchmarkperformedattheMicrosoftTechnologyCenterinNewYorkusingahub-and-spoketopologyconnecting61SharePoint2010farms.Thefigureshows:
Multiplereplicationenginesprocessingandtransferringmultiplepackagessimultaneously
MultiplethreadsperenginesupportingparallelReplicationPackageprocessingandtransfer
Smooth scaling of the replication processing under heavy load
Replicator’sReplicationEnginearchitectureisbasedonamulti-threadedreplicationqueuearchitecturethat supports parallelReplicationPackageprocessing.ReplicatorStandardEdition supports 4 threadsperreplicationenginewithasingleactiveenginerunningonasinglewebfront-endperSharePointfarm.ReplicatorEnterpriseEditionsupports10toanunlimitednumberofthreadsrunningonasinglereplicationenginerunningonanunlimitednumberofwebfront-endsperSharePointfarm.
Figure2.MultipleEngine,MultipleThreadPerformanceandScalability showsReplicatorEnterpriseEditionrunningon4webfront-endsconfiguredtoperformparallelReplicationPackageprocessingandtransfer.
Figure2.MultipleEngine,MultipleThreadPerformanceandScalability
REPLICATOR MULTI-ENGINE DEPLOYMENT
MULTI-THREADED REPLICATION QUEUE ARCHITECTURE
m e t a l o g i x . c o m W H I T E PA P E R
8
CUSTOMIZED REPLICATION INTERVAL
CONFIGURABLE PACKAGE EVENT COUNT
Replicatorsupportsseveraldifferentreplicationschedulingpolicies:
Immediately(defaultreal-time,event-drivenreplicationpolicy)
Interval(seconds,minutes,hours)
Daily
Weekly
Monthly
Manual
Bydefault,ReplicatorincludestheabilitytogrouporbatchmultipleReplicationEventsintoasingleReplicationPackageforprocessingandtransfertotheTargetWebApplication.ThisConfigurablePackageEventCountdetermines themaximumnumberofEvents that canbepackaged into a singleReplicationPackage. ThisenablesReplicator to be tuned for different levels of real-time replication,wide area network performancecharacteristicsandavailableInboundandOutboundEventProcessingmemoryandCPUprocessingresources.
AhigherPackageEventCountwillresultinfewer,largerPackagesbeingcreatedwhichcanincreaseoverallperformance.TheWindowsBackground IntelligentTransferProcess (BITS)protocol usedbyReplicator forPackageTransferisdesignedforhighperformanceandreliability(restartable)withlargefiledownloadsaswellassmallerdownloads.Largerpackages,somewhatsurprisingly,canresultinbetterthroughputandperformanceinslowandunreliablewideareanetworkscenarios.
ThesesettingsenablefinegrainedcontroloverwhenOutboundEventProcessingandPackageTransferwilloccur.This in turnsallows theSharePointAdministrator tobettermanageSharePointserverandwideareanetworkresources.
Figure3.MapFamilyReplicationSchedule
m e t a l o g i x . c o m W H I T E PA P E R
9
SimilartotheConfigurablePackageEventCountfeature,theConfigurablePackageEventProcessingDurationallowscontroloverhowmanyReplicationEventsaregroupedorbatchedtogether intoasingleReplicationPackage based on elapsedOutbound Event Processing time. This setting enables Replicator local serverresourcerequirementstobetunedforoptimalperformance.
Replicatorsupportsseveral formsofsoftwarecompression inadditiontosupportforandcompatibilitywithvarietyofhardwarenetworkcompressiondevices.ReplicatorcanuseeithersoftwareorhardwarecompressiontoreducetheamountofReplicationPackagedatatransferredoverawideareanetwork.
ThefollowingsectionsdescribethebenefitsofReplicator’ssupportforhardwarenetworkcompressionandsoftwarepackagecompression.Replicator’shighlyoptimizedimplementationofMicrosoft’sRemoteDifferentialCompression(RDC)isalsodescribed.
When hardware network compression devices, bandwidth optimization appliances, or network acceleratorsolutions are available (such as the Riverbed® Steelhead® Appliance), Replicator can be configured tominimizetheserverresourcesusedforsoftwarecompressionandmaximizetheeffectivenessofthenetworkcompression device or application. A sample deployment is illustrated in Figure 4. Metalogix Replicator and Riverbed®Steelhead®ApplianceCompressedReplicationPackageSolution.
Figure4.MetalogixReplicatorandRiverbed®Steelhead®ApplianceCompressedReplicationPackageSolution
CONFIGURABLE PACKAGE EVENT PROCESSING DURATION
REPLICATION PACKAGE COMPRESSION
HARDWARE NETWORK COMPRESSION SUPPORT
BranchOffice
LAN
WAN
Metalogix ReplicatorData Center
Metalogix Replicator on RSP
m e t a l o g i x . c o m W H I T E PA P E R
10
ConfiguringReplicatortosupporthardwarenetworkcompressiondevicesavoidstheredundantandpotentiallyexpensiveoverheadofcompressingpackagesinsoftwarewhenhardwarecompressionisavailable.
SOFTWARE PACKAGE COMPRESSION
REMOTEDIFFERENTIALCOMPRESSION(RDC)
ReplicatorsupportsZIPsoftwarepackagecompressionandacustomimplementationofMicrosoft’sRemoteDifferential Compression (RDC) that is highly optimized for the one-way and bi-directional replication ofSharePoint data.
MetalogixReplicatorsupportsacustomimplementationofMicrosoft’sRemoteDifferentialCompression(RDC)technologythatMetalogixhasoptimizedtoreducethetotalamountofReplicationPackagedatathatneedstobetransferredbetweentheSourceWebApplicationtotheTargetWebApplicationduringreplication.
“Remote Differential Compression (RDC) allows data to be synchronized with a remote sourceusingcompressiontechniquestominimizetheamountofdatasentacrossthenetwork.RDCissuitableforapplicationsthatmovedataacrossawideareanetwork(WAN)wherethedatatransmissioncostsoutweightheCPUcostofsignaturecomputation.RDCcanalsobeusedonfasternetworksiftheamountofdatatobetransferredisrelativelylargeandthechangestothedataaretypicallysmall.”1MorespecificdetailsonRDCcanbefoundinAppendixA–AboutWindowsRemoteDifferentialCompression.
Metalogix’sDynamicDifferentialCacheimprovementstoMicrosoftRDCuseserver-sidepre-processingandpost-processingtosignificantlyreducetheamountofSharePointdatathatistransferredoverthewideareanetwork.
DuringperformancetestingintheMetalogixNetworkTestLab,usingreal-lifeandsimulatedtestdata(SharePointtasks,multi-leveldocumentlibraries,Office2003documents,Office2007documents,Office2010documentsaswellasotherdesktopfileformatssuchasAutoCADdrawings,photosandimages),MetalogixReplicatorRDCreducedthetotalnetworktrafficbymorethan90%comparedtoZIPcompression.
m e t a l o g i x . c o m W H I T E PA P E R
INDEPENDENT OUTBOUND EVENT AND INBOUND EVENT PROCESSING CONTROLS
SELECTIVE STRUCTURE REPLICATION
RULE-BASED ITEM-LEVEL CONTENT REPLICATION
ReplicatorsupportsindependentcontrolsforOutboundEventProcessingandInboundEventProcessingattheWebApplicationandReplicationConnectionlevelsforaSourceWebApplicationandTargetWebApplication.Inaddition,replicationforaparticularWebApplicationcanbeconfiguredasone-wayorbi-directional.
Whenonlyone-wayreplication isrequired,performance isoptimizedwhenEventprocessing isdisabledontheappropriateWebApplication.Inthebi-directionalreplicationscenario,separateReplicationConnectionsenableseparatereplicationpoliciestobeusedfortheoutboundandinboundconnections.
Bydefault,manycustomersusewebapplication-to-webapplicationreplication–replicatingtheentirecontentofeachsitecollection inonewebapplicationtothesecondwebapplication.SelectivestructurereplicationenablestheSharePointAdministratortoselectaspecificsubsetoftheSourceWebApplicationstructuretobereplicatedtotheTargetWebApplication.
Forexample:
One or more site collections
Rootwebsite
Anyindividualchildwebsite
Anychildwebsiteandsubhierarchyofthechildwebsite’schildwebsites
Individuallistsand/ordocumentlibrariesinawebsite.
Selectivestructurereplicationcanbeusedtosignificantlyreducethevolumeofreplicationdatabyselectingthespecificsitecollectionsandwebsitestobereplicated.
Rule-based item-levelcontent replicationuses theMetalogixReplicatorRulesEngine tooptionallyprocesscustomrulesetstodetermineifaniteminaparticularlistordocumentlibraryshouldbereplicated.
Althoughrules-basedcontentreplicationisnormallyconsideredasecurityandcontentpromotionfeature,rulesetscanalsobeusedtoblockthereplicationoflargerfilesorotherfilesbasedoneachitem’smetadatavaluesand improve performance.
11
1Source:http://msdn.microsoft.com/en-us/library/aa372948(v=VS.85).aspx)
m e t a l o g i x . c o m W H I T E PA P E R
SELECTABLE REPLICATION EVENTS
SHARED REPLICATION PACKAGES
12
Replicator supports selectable replication of changes that occur in a SharePoint Web Application, SiteCollection,WebSite,ListorDocumentLibrary.ThedifferenttypesofindividualchangesarecalledReplicationEvents. Replication Events are categorized into higher-level Event Groups. By selecting which Events need tobereplicatedandwhichEventsdon’tneedtobereplicated,theSelectableReplicationEventsfeature provides the SharePoint Administrator with fine-grained control over the Events processing duringInboundEventProcessing, Package Transfer andOutboundEventProcessing;which in turn help improveoverall Replicator performance.
Figure5.MetalogixReplicatorReplicationEventshighlightstheReplicationEventssectionof theConfigureMapFamilywebpagefromReplicatorCentralAdministration.EntireEventGroupsorindividualEventscanbeenabledordisabled.Thisfigureshowsthedefaultsettings.
InReplicatorversion3,aseparateReplicationPackagewascreatedforeachoutboundReplicationConnection.The Shared Replication Package feature in Replicator version 4 eliminates the time required to create aseparatePackageforeachoutboundReplicationConnection–asinglesharedPackagecontainingthebatchofReplicationEvents iscreated.Dependingon thenumberofEvents in thepackage, thesizeand typeoftheSharePointchange,andthenumberofoutboundReplicationConnections, thiscansignificantly reduceOutboundEventProcessingtimeandresources.
Figure6.MicrosoftTechnologyCenterBenchmark:SharedReplicationPackageOutboundProcessingTimeillustrateshowtheelapsedOutboundProcessingtimeforthefirstPackage(greenportionoffirstbarinthechart)wassignificantlyreducedfortheremaining59instancesofthePackagereplicatedtoTargetWebApplication2through60.
Replication version4.1 also includesnewWebSiteReplicationEvents for replicating SharePoint 2010’ssocialnetworkingfeatures:ratings,commentsandkeywordtags.
Figure 5. Metalogix Replicator Replication Events
m e t a l o g i x . c o m W H I T E PA P E R
PACKAGE DATABASE CACHING
CONFIGURABLE REPLICATION MONITOR UPDATE LEVEL
In Replicator version 4, upon acceptance of the transfer of an Inbound Replication Package, ReplicatorimmediatelycachesthemetadataforeachReplicationEvent in theReplicatorConfigurationdatabase.ThisimprovestheperformanceoftheReplicatorEnginebyeliminatingtheneedtorepeatedlyaccesstheserializedPackagedata;especiallyinSharePointfarmsthathostmultipleReplicatorEngines.
PackageDatabaseCachingspecificallyappliestoInboundEventProcessingontheTargetWebApplicationServerandimprovesoverallInboundEventProcessingbyreducingthenumberoffilesystemaccesses,reducingthevolumeofdatareadfromthefilesystemandloweredfiledecompressionCPUusage.
DuringnormalInboundEventProcessing,Replicator,runningonaTargetWebApplication,willmakeseveralcallstotheReplicatorWebServicerunningontheSourceWebApplicationtokeeptheMonitorReplicationreportsontheSourceWebApplicationascurrentaspossible.
TohelpminimizetheQueuedItemupdatenetworktrafficbetweentheSourceandTargetWebApplications,Replicatorversion4.1supportsanewReplicationConnectionpropertycalled“quietmode”.Itcanbeconfigured through theConfigureReplicationConnectionadministrationpagewhere it is referred toas theConfigurableReplicationMonitorUpdate Level. This new setting has 2 values:Normal andMinimal (quitemode).
DuringperformancetestingintheMetalogixNetworkTestLabreplicating10001KBfiles,thenetworktrafficwasreducedby55%andthenumberofHTTPPUToperationswasreducedby60%.
Figure6.MicrosoftTechnologyCenterBenchmark:SharedReplicationPackageOutboundProcessingTime
13
m e t a l o g i x . c o m W H I T E PA P E R
ThenewBackupModefeatureprovidesamoreefficientmethodofreplicatingalargesitecollectionorhierarchyofwebsites.UsingBackupMode,ReplicatorusestheSharePointimportandexportoperationstocreateasinglearchiveoftheentiregroupofwebsites,andqueuesthereplicationofthisarchiveasasingleReplicationEvent.
BackupMode significantly improves the performance and reliability of a large initial replication: instead ofcreatingandtransferring10,000sofqueueditemsandpackagesinaverylargeSharePointenvironment,onlyafew(large)Packagesneedtobepackagedandtransferred.
Inaddition,ifRDCcompressionisenabled,theamountofdatatransferredoverawideareanetworkwillbereduceddramatically.Whenreplicatingtomultiplelocations,onlyonePackageiscreatedforalldestinationswiththeversion4SharedReplicationPackagefeature.
Figure 7. Microsoft Technology Center Windows Server 2008 R2 Hyper-V Host Environment: MetalogixReplicatorBenchmark
InAugust2010,MetalogixwasinvitedtotheMicrosoftTechnologyCenter(MTC)inNewYork,NYtomeasuretheperformanceandscalabilityofanearlyreleaseofMetalogixReplicatorforSharePointversion4.0.
WorkingcloselywiththeMTCtechnicalstaff,theMetalogixMassiveScalabilityTeamdeployedandconfiguredtheworld’slargestSharePoint2010DistributedSharePointenvironment,measuredintermsofthenumberof independent SharePoint farms. Eleven large scale physical servers from Dell and HP running Microsoft WindowsServer2008R2wereusedtodeployandconfigure85WindowsServer2008
R2 virtual machines. Microsoft Windows Hyper-V was used for the operating system virtualization solution. The entireconfigurationwasdeployedandmanagedusingMicrosoftWindowsVirtualMachineManager.
IMPROVED INITIAL REPLICATION SUPPORT
MICROSOFT TECHNOLOGY CENTER PERFORMANCE AND SCALABILITY BENCHMARK
14
HUB Servers
64GB
x 16
Core
s
Hype
r-V H
osts
3 x 200GB LUNs 200GB LUN 200GB LUN 200GB LUN
x 8
32GB
x 16
Core
s
32GB
x 16
Core
s
32GB
x 16
Core
s
VPN VMM RouterAD/DNS LabWorkstations
TARGET Servers TARGET Servers TARGET Servers
m e t a l o g i x . c o m W H I T E PA P E R
15
Fivevirtualmachinesrunningon16processorcoreswithaccessto64GBofRAMand600GBofdiskstoragewereusedtohost4WebFront-EndServersand1SQLServer2008databaseserverintheHubSharePoint2010farm.Eighty(80)additionalvirtualmachinesrunningon96processorcoreswithaccessto320GMofRAMand2TBofdiskstoragewereusedtohost60single-serverSharePoint2010farms.ThespecificconfigurationoftheWindowsServer2008R2physicalhostscanbefoundinTable1.MicrosoftTechnologyCenterWindowsServer2008R2Hyper-VHostEnvironment:VirtualMachineConfiguration.
Table1.MicrosoftTechnologyCenterWindowsServer2008R2Hyper-VHostEnvironment:VirtualMachineConfiguration
Thefollowingperformancemonitoringtoolswereusedduringthebenchmark:
Windows Performance Monitor
Metalogix Replicator Replication Monitor
Metalogix Replicator Replication Status
Metalogix Connect for Replicator
Custom Replication Data Collection PowerShell Scripts
CustomReplicatorDeployment,Configuration,andManagementPowerShellScripts
CustomSharePointBenchmarkingPowerShellScripts
HostType
No.Hosts
Cores RAM(GB)
No.LUN
LUNSize(GB)
TotalCores
TotalRAM
TotalStorage(GB)
VMRAM
No. VM
Hub 1 16 64 3 200 16 64 600 - 5
Targets 1 16 32 1 200 16 32 200 4 8
Targets 1 16 32 1 200 16 32 200 4 8
Targets 8 8 32 8 200 64 256 1600 4 64
Totals 11 112 384 2600 12 85
m e t a l o g i x . c o m W H I T E PA P E R
DuringthedesignanddevelopmentofMetalogixReplicatorversion4,eachnewperformancebuildwastestedandmeasuredalong40dimensions,including:
CPU usage
Availablememory
Workingset
Diskaccess
Networkaccess
Networkutilization
HTTPWebServerperformance
Theconfigurationof theMetalogixNetworkTest Lab is depicted in Figure 8.MetalogixNetworkTest Lab:PerformanceBenchmarking.IncontrastwithvirtualizedMicrosoftTechnologyCenterenvironmentusedforthemassivescalabilityandperformancebenchmarking,theMetalogixNetworkTestLabrunsentirelyonphysicalhardware.
Figure8.MetalogixNetworkTestLab:PerformanceBenchmarking
METALOGIX NETWORK TEST LAB PERFORMANCE BENCHMARKING
16
RDC1 SQL Server
Phys
ical H
osts
RDC1 WFE
WAN Simulator
AD/DNS Lab Workstations
RDC2 WFE RDC2 SQL Server
m e t a l o g i x . c o m W H I T E PA P E R
ThekeytounderstandingReplicatorperformanceandscalability fromaday-to-dayoperationalperspectiveinvolvesunderstandinghowthebasiccomponentsoftheReplicatorPipelinecaptureandpackageReplicationEvents,transferpackagesfromtheSourceWebApplicationtotheTargetWebApplicationandfinallyupdatetheTargetWebApplicationwiththeneworchangedcontent.
OutboundEventProcessing is responsible forcapturingandrecordingReplicationEvents thatoccur in theSourceWebApplication.Forexample,whenausercreatesanewcontactinacontactlistinaSharePointWebsiteoreditsandsavesaMicrosoftWorddocumentthatisstoredinaSharePointdocumentlibrary,eachoftheseis captured and recorded as a Replication Event. Replication Events are captured and stored in a set of SQL ServertablesinaReplicatordatabase.OutboundEventProcessingiscontrolledbyReplicationMapswhichdeterminewhatEventsneedtobecaptured,packagedandtransferredtotheTargetWebApplication.GroupsofReplicationEventsarepackaged into two typesofmessagesorobjects:Queued ItemsandReplicationPackages.
Figure9.MetalogixReplicatorPipeline
The Replicator Pipeline is a detailed representation of how Replicator divides the overall replication process into 3activities:
CapturesachangeonaSourceWebApplication,packagesagroupofReplicationEventsandassociatedSharePointcontentintoaQueuedItemandcorrespondingReplicationPackage,
TransferstheQueuedItemandPackagefromtheSourcetotheTargetWebApplication,and
WhenreceivedandacceptedontheTargetWebApplication,appliesthechangesrepresentedbytheQueuedItemandPackagetotheTargetWebApplication.
These3activities,OutboundEventProcessing,PackageTransferandInboundEventProcessingareshowninthefollowingfigure.
UNDERSTANDING REPLICATOR PERFORMANCE AND SCALABILITY
METALOGIX REPLICATOR PIPELINE
17
m e t a l o g i x . c o m W H I T E PA P E R
AQueued Item is a unit ofwork to be transferred to a TargetWebApplication for remote execution. TheReplicatorWebServiceonaTargetWebApplication iscalledtopushaQueuedItemfromtheSourceWebApplicationtoaTargetWebApplication.
A Replication Package is a collection of one or more Replication Events plus data about the SharePointinformation that is packaged in a format specific to theReplication Transport being used.When anEventisbeingprocessed,OutboundEventProcessingcalls theSharePointobjectmodel toextract thechangedSharePointinformationfromtheSourceWebApplicationcontentdatabase.
The Package Transfer activity is responsible for the transfer of Queued Items and Replication Packages(Packages)fromtheSourceWebApplicationtotheTargetWebApplication.PackageTransferistheprocessthatsitsbetweenOutboundEventProcessing(ontheSourceWebApplication)andInboundEventProcessing(ontheTargetWebApplications).PackageTransferinvokesReplicatorEnginesoftwarecomponentsonboththeSourceandTargetWebApplications.
InboundEventProcessingisresponsibleforprocessingtheQueuedItemsandPackagesreceivedandacceptedbytheTargetWebApplicationandapplyingthemtothesitecollections,Websites,listsanddocumentlibrariesinthewebapplication’scontentdatabase.TheQueuedItemsandPackagesareappliedtotheTargetWebApplicationcontentbasebycallingtheSharePointObjectModel.
ReplicatorperformanceisbestunderstoodbyanalyzingtheReplicatorPipelineactivitiesdepictedinFigure9.MetalogixReplicatorPipeline.ThesamevariablesthataffectperformanceofaSharePointenvironmentalsoaffectReplicatorperformance.ReplicatorwillperformoptimallyiftheunderlyingSharePointenvironmenthasbeenconfiguredforoptimalperformanceforagivenuserworkload.Inaddition,thereareReplicator-specificconfigurationsettingsandoptimizationthatcanfurtherimproveReplicatorperformance.
TheperformanceprofileofSharePoint,andeachoftheReplicatorPipelineactivities,isbasedonavarietyoffactors including:userworkload,documentfile formats,documentsizedistributions,SharePointserverandfarmconfiguration,databaseserverconfiguration,localareanetwork(LAN)characteristics,wideareanetwork(WAN)characteristics,ReplicationdatacompressionandreplicationmonitorsettingsaswellasotherReplicatorconfigurationsettings.
Userworkloadisdefinedbytwofactors:usercontentchangerateandusercontentchangesizedistribution.Qualitatively, the user content change rate is simply the rate at which users of a particular SharePoint environment makecontentchanges:creatingnewlistitems,editingandsavingexisting
items, creating or uploading new documents, editing and saving existing documents and, less frequently,
REPLICATOR PERFORMANCE
18
m e t a l o g i x . c o m W H I T E PA P E R
creatingnewsitecollections,websites,listsanddocumentlibrariesormakingmetadataorsecuritychanges.
User content change size distribution refers to the size of the content changes and their distribution. Forexample,inadocumentcollaborationsolution,theaveragesizeofaWorddocumentmightbeafewmegabytesandtheaveragesizeofaPowerPointmightbemeasuredin10’sofmegabytes.TheamountofdataneededtorepresentanewSharePointcontactortaskitemwouldbemeasuredin10sor100sofkilobytes.Thedistributionofsizesvaryfromsmall1Kbytedocumentstomediafilesmeasuredin10sofmegabytes.
In Replicator version 4, each Map Family contains a Replication Schedule. A Replication Schedule controls how frequentlyReplicationEventsforaparticularReplicationConnectionareprocessedandpackagedintoQueuedItemsandReplicationPackages.EventProcessingandPackagingcanbescheduledtorun:
Immediately(real-timereplication)
Afteratimedinterval(afteraspecifiednumberofseconds,minutes,hoursordays)
Scheduleddaily,weekly,monthly
Manually initiated
Withtheexceptionofimmediatereplication,OutboundEventProcessingperformanceisprimarilydeterminedbytheReplicationSchedule,usercontentsizedistributionandSharePointsystemperformance.Forimmediatereplication,OutboundEventProcessingperformanceisalsoaffectedbytheusercontentchangerate.Fromasystemperformanceperspective,allOutboundEventProcessingoccurslocallyintheSharePointfarmthatishostingaparticularSourceWebApplication.OnthelocalSharePointfarm,serverRAM,diskorganization,LANnetworkconfigurationanddatabaseserverconfigurationarethegreatestdeterminantsofSharePointandReplicator system performance.
When Replicator is configured for immediate replication, Outbound Event Processing (Event capture andprocessing)isperformedinnearrealtime.
OUTBOUND EVENT PROCESSING
19
m e t a l o g i x . c o m W H I T E PA P E R
PackageTransferreferstothepartoftheReplicatorPipelinethatisresponsiblefortransferringQueuedItems(and their correspondingPackages) fromaSourceWebApplication to a TargetWebApplication. PackageTransferperformanceisdeterminedbyanumberofdifferentfactors:
Userworkload
WANnetworkperformance
SourceWebApplicationsystemperformance
TargetWebApplicationsystemperformance
Replicator data compression
AQueuedItemandcorrespondingReplicationPackagerepresentsabatchofchangesthatusershavemadetotheSourceWebApplicationcontent.QueuedItemsarerelativelysmallinsizeandarepassedfromtheSourceWebApplicationtotheTargetWebApplicationbycallingtheTargetWebApplicationReplicatorWebService.PackagesareeitherdownloadedoruploadedtotheTargetWebApplicationusingtheBITSprotocolrunningoverHTTP/HTTPS.ThemajorityofthePackageTransfertimeisconsumedbythetransferofthePackage.ThisisbecausethePackagecontainsthatchangedSharePointcontentthatneedstobereplicatedfromtheSourceWebApplicationtotheTargetWebApplicationandthechangedcontent,inrelativeterms,maybequitelargedependingonwhetherthechangedcontentissimplelistitemora10MBPowerPointdocument.
Package Transfer performance be improved using Package compression settings that can be set usingReplicatorCentralAdministrationonaReplicationConnectionbyConnectionbasis.The followingPackagecompressionsettingsaresupportedinReplicator:
None
ZIP(default)
RDC(basedonMicrosoftRemoteDifferentialCompressiontechnology)
OncethecharacteristicsoftheQueuedItemandPackagestreamandthePackagecompressionsettingsareunderstood,thenextmajordeterminantofPackageTransferperformancearethecharacteristicsoftheWAN:
Availablebandwidth
Round-trip latency
Packetloss
Reliability
For example, satellite links typically have lower bandwidth, high latency and lower reliability compared toterrestriallinks;andexceptionallypoorerperformancecomparedtoafiberringinametropolitanareanetwork(MAN) or LAN.Metalogix’s advanced use of theMicrosoft BITS protocol in the design of theReplicatorPackageTransferprocessallowsReplicatortofunctionextremelywelloverhighperformanceconnectionsaswellasoversloworunreliablesatellite,ship-to-shoreandbattlefieldnetworks.
PACKAGE TRANSFER
20
m e t a l o g i x . c o m W H I T E PA P E R
IfReplicatorisinstalledandconfiguredonmultipleSharePointfarmsconnectedbyahigh-performancenetwork,thePackageTransferprocesswillperformonanearrealtimebasisbetweenanypairofSourceandTargetWebApplications.
InboundEventProcessingbeginsonceaQueuedItemanditscorrespondingPackagehavebeentransferredtotheTargetWebApplicationbythePackageTransferprocess.InboundEventProcessingexecutesentirelywithintheSharePointfarmfortheTargetWebApplication.
InboundEventProcessingperformanceisdeterminedbytherateatwhichtheReplicatorServicecanprocessthelocalQueuedItemsandtheircorrespondingPackages,decompressthePackagesandapplythecontentchangestotheTargetWebApplicationbycallingtheSharePointObjectModel.TheInboundEventProcessingperformanceisaffectedbythefollowingfactors:
Userworkload
SharePointserverandfarmconfiguration
Databaseserverconfiguration,
Localareanetwork(LAN)characteristics
Subject to theaboveconstraints,Replicator InboundEventProcessing isable tooperateatnear real timeperformance.
INBOUND EVENT PROCESSING
21
m e t a l o g i x . c o m W H I T E PA P E R
Replicator Central Administration provides detailed replication monitoring, replication status and replication eventprocessinghistoryreportsformonitoringtheReplicatorPipeline.ThefollowingtwofiguresareexamplesofthestatisticsavailablefromtheReplicationStatusreportforasampleintranetSourceWebApplicationthatisbeingreplicatedtoanextranetTargetWebApplication.
NOTE:Thefollowingfiguresarefromasmall-scaletestperformedonvirtualsingleserverSharePointfarmthatishostingboththeSourceandTargetWebApplications.Thestatisticsshowninthesefiguresareforillustrationpurposes only.
Intheabovefigure,theOutboundEventProcessingstatisticstellsus:
13Packages(andQueuedItems)wereprocessedrepresenting13batchesofReplicationEvents
TotalnumberofReplicationEvents=212
Totalsizeofthe13Packages(212Events)=13.8MB
Figure10.SampleOutboundEventProcessingStatistics
PERFORMANCE MONITORING
22
m e t a l o g i x . c o m W H I T E PA P E R
Figure11.SampleInboundEventProcessingStatistics
Intheabovefigure,theInboundEventProcessingstatisticstellsus:
14Packages(andQueuedItems)wereprocessedrepresenting14batchesofReplicationEvents
TotalnumberofReplicationEvents=213
Totalsizeofthe14Packages(213Events)=11.82MB
NOTE:InboundandOutboundEventProcessingstatisticsareneverexpectedtomatch,duetoseveralfactors,includingsystemgeneratedQueuedItemsandPackages,retries,filteredPackagesandrejectedPackages.
TomonitorlargeDistributedSharePointenvironmentsconsistingofseveralfarms,MetalogixConnectfor
SharePoint or Metalogix Connect for Replicator is the monitoring tool of choice for these environments.
23
m e t a l o g i x . c o m W H I T E PA P E R
ThefollowingtablemapsthekeyMetalogixReplicatorperformancefeaturestoeachphaseoftheReplicatorPipeline.
NOTE:TheperformancefeaturesthataffectOutboundEventProcessingalsotendtoenhancetheperformanceofthedownstreamReplicatorPipelineactivities:PackageTransferandInboundEventProcessing.Thereisonlyonefeature,PackageDatabaseCaching,whichonlyaffectsInboundEventProcessing.
Table2.MetalogixReplicatorPipelinePerformanceFeatures
REPLICATOR PIPELINE PERFORMANCE FEATURES
24
Feature AllPipelineActivities
OutboundProcessing
PackageTransfer
InboundProcessing
Replicator Multi-Engine Deployment YES
Multi-Threaded Replication Queue Architecture YES
ConfigurablePackageEventCount YES
ConfigurablePackageEventProcessingDuration
YES
IndependentOutboundEventProcessingandIn-boundEventProcessingControls
YES
Selective Structure Replication YES
Rule-basedItem-levelContentReplication YES
SelectableReplicationEvents YES
CustomizableReplicationInterval YES
SharedReplicationPackages YES
ReplicationPackageCompression YES YES
HardwareNetworkCompressionSupport YES YES
SoftwarePackageCompression YES YES
RemoteDifferentialCompression(RDC) YES YES
PackageDatabaseCaching YES
m e t a l o g i x . c o m W H I T E PA P E R
TheMetalogixReplicatorarchitectureisbasedonanunlimitedmulti-master,point-to-pointreplicationnetworkmodelthatdoesnotrequireglobalsharingofanyconfigurationinformation.Arbitrarynetworktopologiesaresupportedincluding:
Serial
Parallel
Hub-and-spoke
Cascadinghub
Federatedhub
Cross domain
Mesh, and
Offline.
MetalogixConnect isadesktopapplication thatprovidesvisualization,monitoringandmanagementof thewide area SharePoint environment. It can be used to visualize the status of each farm and its associatedconnections.Connectisavailableasanadd-onfeaturefortheBasicandtheStandardEdition(andisincludedintheEnterpriseEdition).
MetalogixConnectforReplicatorprovidesthehighlyscalablereplicationnetworkmanagementtoolsneededtomonitorandmaintainthelargest(andsmallest)distributedSharePointenvironments.
Source:http://msdn.microsoft.com/en-us/library/aa372948(VS.85).aspx
RemoteDifferentialCompression(RDC)allowsdatatobesynchronizedwitharemotesourceusingcompressiontechniquestominimizetheamountofdatasentacrossthenetwork.
RDCisdifferentfrompatching-orienteddifferencingmechanisms,suchasBinaryDeltaCompression(BDC),thataredesignedtooperateonlyonknownversionsofasinglefile.BDCrequirestheservertohavecopiesofallversionsofthefile,anddifferencesbetweeneachpairofversionsarepre-computedsothattheycanbedistributedefficientlyfromaservertomultipleclients.
REPLICATOR SCALABILITY
UNLIMITED MULTI-MASTER, POINT-TO-POINT REPLICATION NETWORK MODEL
METALOGIX CONNECT FOR REPLICATOR
APPENDIX A – ABOUT WINDOWS REMOTE DIFFERENTIAL COMPRESSION
25
m e t a l o g i x . c o m W H I T E PA P E R
RDCmakesnoassumptionsaboutfilesimilarityorversioning.Becausedifferencesbetweenfilesarecomputedonthefly,RDCisideallysuitedforsynchronizingfilesthataredifferentorhavebeenupdatedindependently.
RDC does not assume that the file data to be synchronized resides in physical files. Therefore, the RDCapplicationisresponsibleforperformingfileI/OonbehalfoftheRDClibrary.
Becauseitistransportindependent,RDCcanbeusedwithRPC,HTTP,orotherdesiredtransportmechanisms.TheRDCapplicationbearstheresponsibilityforchoosingtheappropriatetransportandperforminganyclientor server authentication that is required to support the transport's security model.
RDCissuitableforapplicationsthatmovedataacrossawideareanetwork(WAN)wherethedatatransmissioncostsoutweightheCPUcostofsignaturecomputation.RDCcanalsobeusedonfasternetworksiftheamountofdatatobetransferredisrelativelylargeandthechangestothedataaretypicallysmall.
RDCdividesafile'sdataintochunksbycomputingthelocalmaximaofafingerprintingfunctionthatis
computedateverybytepositioninthefile.Afingerprintingfunctionisahashfunctionthatcanbecomputedincrementally.Forexample,ifyoucomputethefunctionFoverarangeofbytesfromthefile,Bi...Bj,itshouldthenbepossibletocomputeF(Bi+1...Bj+1)incrementallybyaddingthebyteBj+1andsubtractingthebyteBi.Therangeofbytesfromthefile,Bi...Bj,iscalledthehashwindow.Thelengthofthiswindow,inbytes,iscalledthe hash window size.
TheRDClibrary'sFilterMaxsignaturegenerator"slides"thehashwindowacrosstheentirefilebyaddingthebyteattheleadingedgeandsubtractingthebyteatthetrailingedgeofthewindow.Meanwhile,thegeneratorcontinuallyexaminesthesequenceoffingerprintfunctionvaluesoveragivenrangeofbytes,calledthehorizonsize.Ifafingerprintfunctionvalueisalocalmaximumwithintherange,itsbytepositionischosenasa"cutpoint,"orchunkboundary.
After thefilehasbeendivided intochunks, thesignaturegeneratorcomputesastronghashvalue (anMD4hash),calledasignature,foreachchunk.Thesignaturescanbeusedtocomparethecontentsoftwoarbitrarilydifferentversionsofafile.
Becausethesizeofthesignaturefilegrowslinearlywiththesizeoftheoriginalfile,comparingverylargefilescanbeexpensive.ThiscostisreduceddramaticallybyapplyingtheRDCalgorithmrecursivelytothesignaturefiles.Forexample,iftheoriginalfilesizeis9GB,thesignaturefilesizewouldtypicallybeabout
81MB.IftheRDCalgorithmisappliedtothesignaturefile,theresultingsecond-levelsignaturefilesizewouldbeabout5.7MB.
REMOTE DIFFERENTIAL COMPRESSION ALGORITHM OVERVIEW
26
m e t a l o g i x . c o m W H I T E PA P E R
When developing an application that uses RDC, it is important to understand the following concepts and terminology.
InatypicalRDCscenario,aserverandaclienthavedifferentversionsofafile.(Thetermsclientandserverreferonlytothecomputers'rolesinthisscenario,nottheiroperatingsystems.)Theclient'scopyofthefileiscalledtheseedfile.Theserver'scopyiscalledthesourcefile.TheobjectiveoftheRDCapplicationistodownloadthefileupdatestotheclient,whichusesthemtoconstructatargetfilethatcombinestheupdatesfromthesourcefilewiththeunchangedcontentsfromtheseedfile.
TheRDCclientandservereachusetheRDClibrary'sFilterMaxsignaturegeneratortodividetheircopyofthefileintochunksandcomputeastronghash,calledasignature,foreachchunkoffiledata.Thus,theclienthasalistofsignaturesfortheseedfile,andtheserverhasalistofsignaturesforthesourcefile.Thesesignaturelistscanbecomputedonthefly,ortheycanbepre-computed.
Theclient initiatestheRDCprotocolbyrequestingthesourcesignature list fromtheserver.Thentheclientcompares each source signature against the signatures in its own seed signature list. If a source signature matchesaseedsignature,theclientalreadyhasthefiledataforthatsignature.Ifasourcesignaturedoesnotappearintheclient'slistofseedsignatures,theclientmustrequestthespecifiedchunk(offiledata)fromtheserver.
Theresultofcomparingthetwosignaturelistsisaneedslist,whichdescribeswhichchunksoffiledata,fromwhere(seedorsourcefile),areneededtoconstructthetargetfileontheclientcomputer.Eachentry intheneedslistiscalledaneedsblock.
Theclientiteratesthrougheachneedsblockandcopiesthespecifiedchunkofthesourceorseedfiledatatothetargetfile.Seedfiledataiscopiedlocally.Sourcefiledataisdownloadedfromtheserver.Themoresimilartheseedandsourcefilesare,thelessnetworkbandwidthisrequiredtocreatethetargetfile.
REMOTE DIFFERENTIAL COMPRESSION APPLICATION CONCEPTS
Metalogix is the trusted provider of innovative content lifecycle management solutions for Microsoft SharePoint, ExchangeandCloudplatforms.Wedeliverhigh-performancesolutionstoscaleandcost-effectivelymanage,move, store, archive and protect enterprise content. Metalogix provides global support to thousands ofcustomers and strategic partners and is a Microsoft Gold Partner, a managed partner in Microsoft’s High PotentialISVGroupandGSAprovider.MetalogixisaprivatelyheldcompanybackedbyInsightVenturePartnersand Bessemer Venture Partners.
ABOUT METALOGIX
METALOGIX5335WisconsinAveNW,Suite510,WashingtonDC20015
sales@metalogix.com|www.metalogix.com|1202.609.9100
27
top related