access upsizing event(s)
TRANSCRIPT
Access Upsizing Event(s)Access Upsizing Event(s) http://www.microsoft.com/australia/events/access/http://www.microsoft.com/australia/events/access/A two day introductory hands on workshop A two day introductory hands on workshop covering the following options for Microsoft Access covering the following options for Microsoft Access users looking to upgrade or upsize:users looking to upgrade or upsize:Upgrading from Access 97 to Access 2003Upgrading from Access 97 to Access 2003Upsizing your Access data to SQL ServerUpsizing your Access data to SQL ServerMoving your Access reports to SQL Server Moving your Access reports to SQL Server Reporting ServicesReporting ServicesMoving your Access forms to .NETMoving your Access forms to .NET
CanberraCanberra 11 - 12 Nov 200411 - 12 Nov 2004BrisbaneBrisbane 29 - 30 Nov 2004 29 - 30 Nov 2004 SydneySydney 02 - 03 Dec 2004 02 - 03 Dec 2004 AdelaideAdelaide 06 - 07 Dec 2004 06 - 07 Dec 2004 PerthPerth 09 - 10 Dec 2004 09 - 10 Dec 2004 MelbourneMelbourne 13 - 14 Dec 200413 - 14 Dec 2004
In-depth Look at DTS in YukonIn-depth Look at DTS in Yukon
David LeanDavid LeanPrincipal Technical SpecialistPrincipal Technical SpecialistNational Technology TeamNational Technology TeamMicrosoft AustraliaMicrosoft Australia
where:-Yukon => SQL Server 2005DTS => SSIS (SQL Server Integration Services)
where:-Yukon => SQL Server 2005DTS => SSIS (SQL Server Integration Services)
DTS ArchitectureDTS Architecture
DTS Consists of 4 partsDTS Consists of 4 partsDTS ServiceDTS Service
DTS Run-time Engine & executablesDTS Run-time Engine & executables
DTS Data-flow Engine & componentsDTS Data-flow Engine & components
DTS ClientsDTS Clients
DTS ArchitectureDTS ArchitectureDTS Consists of 4 partsDTS Consists of 4 parts
DTS ServiceDTS ServiceStores Packages & tracks executionStores Packages & tracks execution
DTS Run-time Engine & executablesDTS Run-time Engine & executablesSaves package layoutSaves package layoutRuns packages, Runs packages, Supports logging, debugging, config, connections, & Supports logging, debugging, config, connections, & TransactionsTransactionsDTS Run-time executablesDTS Run-time executables
Are the Containers & the TasksAre the Containers & the Tasks
DTS Data-flow Engine & componentsDTS Data-flow Engine & componentsProvides In-Memory buffers to move dataProvides In-Memory buffers to move dataCalls Source Adaptors to files & DB’sCalls Source Adaptors to files & DB’sProvides Transformations to modify dataProvides Transformations to modify dataDestination Adaptors to load data into data storesDestination Adaptors to load data into data storesComponentsComponents
Source, Destination Adaptors & transformationsSource, Destination Adaptors & transformations
DTS ClientsDTS ClientsTools, Wizards, Command-line Utilities, & Custom Apps that use Tools, Wizards, Command-line Utilities, & Custom Apps that use the Object Modelthe Object Model
DTS DesignerDTS Designer
DTS WizardsDTS WizardsCustom AppsCustom Apps
DTS Service
.dtsx.dtsxfilesfiles
msdbmsdbDBDB
DTS ArchitectureDTS Architecture
DTS RuntimeDTS Runtime
Object ModelObject Model
NativeNative
PackagePackage
Log Log ProvidersProviders
Command Command line Utilsline Utils
ManagedManaged
TaskTask
TaskTaskTaskTask
TaskTask
TaskTask
Data Flow Task(s)Data Flow Task(s)
Container
Tasks
CustomTasks
Tasks
CustomTasks
EnumeratorsEnumeratorsEnumeratorsEnumerators
DataDataSourcesSourcesDataDataSourcesSources
Data Transformation DataflowData Transformation Dataflow
Data Flow TaskData Flow Task
TransformationTransformation
TransformationTransformation
SourceSourceAdapterAdapter
DTP APIDTP API
SourceSourceAdapterAdapter
DestinationDestination AdapterAdapter
DestinationDestination AdapterAdapter
TransformationsTransformations
CustomTransformations
CustomTransformations
Visit www.sqlserver.com.auVisit www.sqlserver.com.auSQL SIG / User Groups in NSW, VIC, QLD & ACTSQL SIG / User Groups in NSW, VIC, QLD & ACT
Monthly Meetings (Free + Pizza & Beer)
Starting in WA email MKleef@ Microsoft.com
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
BI WorkbenchBI Workbench
Business Intelligence WorkbenchBusiness Intelligence WorkbenchOne-stop shopping for projectsOne-stop shopping for projects
Analysis ServicesAnalysis Services
Reporting ServicesReporting Services
Data TransformationData Transformation
Visual Studio-like InterfaceVisual Studio-like InterfaceIntegrated package debuggingIntegrated package debugging
Package host for runtime usePackage host for runtime use
Edit, debug, fix loop in one UIEdit, debug, fix loop in one UI
BI WorkbenchBI Workbench
ProjectProjectA bunch of packagesA bunch of packages
Data Source ViewsData Source Views
Additional filesAdditional files
SolutionSolutionYour settings!Your settings!
BI WorkbenchBI Workbench
Package EditorPackage EditorProperties PaneProperties Pane
ToolboxToolbox
Pop-up editor windowsPop-up editor windows
Design SurfacesDesign SurfacesWork Flow, Data Flow, Event HandlersWork Flow, Data Flow, Event Handlers
Package BrowserPackage Browser
Variables, Connections UIVariables, Connections UI
Output, WatchOutput, Watch
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
Work Flow Vs. Data FlowWork Flow Vs. Data Flow
Work FlowWork FlowProcess-orientedProcess-oriented
Do “something” to “an object”Do “something” to “an object”
Success, fail, or completion resultSuccess, fail, or completion result
Environmental InputsEnvironmental Inputs
Distinct beginning and endDistinct beginning and end
Work Flow Vs. Data FlowWork Flow Vs. Data Flow
Data FlowData FlowData-orientedData-oriented
Produce and/or consume rows of dataProduce and/or consume rows of data
Rows as inputs and outputsRows as inputs and outputs
Can observe environmentCan observe environment
Starts with first row, ends with last rowStarts with first row, ends with last row““for each” rowfor each” row
Work Flow Vs. Data FlowWork Flow Vs. Data Flow
Work Flow Work Flow TasksTasksFile System TaskFile System Task
Execute SQL TaskExecute SQL Task
Send Mail TaskSend Mail Task
Data Flow TaskData Flow Task
Data Flow Data Flow ComponentsComponentsOLEDB Source AdapterOLEDB Source Adapter
SQL Server Destination AdapterSQL Server Destination Adapter
Derived Column TransformDerived Column Transform
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
Work Flow FeaturesWork Flow Features
TasksTasks
Precedence ConstraintsPrecedence Constraints
GroupsGroups
ContainersContainers
LoopingLooping
EventsEvents
Work Flow: TasksWork Flow: Tasks
File System TaskFile System Task
File Transfer Protocol TaskFile Transfer Protocol Task
Execute Process TaskExecute Process Task
Send Mail TaskSend Mail Task
Execute SQL TaskExecute SQL Task
WMI TaskWMI Task
MSMQ TaskMSMQ Task
Database Management TasksDatabase Management Tasks
Execute Package TaskExecute Package Task
ActiveX script, and more!ActiveX script, and more!
Work Flow: TasksWork Flow: Tasks
Smallest Unit of WorkSmallest Unit of Work
Individually successful or Individually successful or unsuccessfulunsuccessful
Disabled or enabledDisabled or enabled
Work Flow: ConstraintsWork Flow: Constraints
Connect one task to anotherConnect one task to another
Give sequential relationship to tasks Give sequential relationship to tasks and their completionand their completion
Establish concurrency!Establish concurrency!
Success/Failure workflowSuccess/Failure workflowExpressions can provide more controlExpressions can provide more control
Work Flow: GroupsWork Flow: Groups
An arbitrary grouping of tasks within a An arbitrary grouping of tasks within a specific container visible in the UI that specific container visible in the UI that imposes no additional execution imposes no additional execution semanticssemantics
Note:Note:GROUPS are only useful in the designer. They GROUPS are only useful in the designer. They are a convenient way of reducing clutter are a convenient way of reducing clutter (detail).(detail).
They provide none of the programmatic They provide none of the programmatic benefits provided by Containers (For Loops, benefits provided by Containers (For Loops, ForEach Loops or Sequences).ForEach Loops or Sequences).
Ie: Transactions, Variable Scope etc.Ie: Transactions, Variable Scope etc.
Work Flow: ContainersWork Flow: Containers
Arbitrary grouping of task listsArbitrary grouping of task listsList of one is allowedList of one is allowed
Tasks connected to others by Tasks connected to others by Precedence ConstraintsPrecedence ConstraintsLists execute concurrentlyLists execute concurrently
Tasks in each list execute sequentiallyTasks in each list execute sequentially
The package is a containerThe package is a containerAdd your own tasksAdd your own tasksInsert your own containersInsert your own containersLoops are containers, tooLoops are containers, too
Work Flow: LoopingWork Flow: Looping
Special ContainerSpecial ContainerFor LoopFor Loop
Uses an expressionUses an expression
For Each LoopFor Each LoopUses an EnumeratorUses an Enumerator
Variable MappingVariable MappingVariables assigned to properties per iterationVariables assigned to properties per iteration
Work Flow: EventsWork Flow: Events
Something happened!Something happened!A special container can be run in A special container can be run in responseresponse
Has information about eventHas information about event
BlockingBlocking
Runtime - Work Flow
WMIWMI
Sources
Flat FileFlat File
DestsOLE
DB
Co
nn
ectio
n F
TP
HTTP
FILE
ODBC
Ma
na
ge
rs C
USTO
M
FLATFILE
CustomCustom
XMLXML
SQLSQLServerServer
Flat FileFlat File
FTPFTP
FileFile
OLE
DB
Co
nn
ectio
n F
TP
HTTP
FILE
ODBC
Ma
na
ge
rs C
USTO
M
FLATFILEWEBWEB
FTPFTP
DBDB
XML Task
FIL
E
FTP Task FT
P
Web Services Task
HT
TP
Data Flow Task
OD
BC
…
FIL
E
…
CustomCustom
Containers
WMI Event Task
WM
I
Event Handlers
Send Mail Task Send Mail Task
Variables Lo
g P
rov
ide
rs
Task HostTask Host
Custom TaskCustom Task
VariablesVariables Lo
g P
rov
ide
rsXMLXML
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
Data FlowData Flow
Components and PathsComponents and Paths
Graph (a “Package”)Graph (a “Package”)
AdaptersAdapters
TransformsTransforms
PerformancePerformance
Data Flow FeaturesData Flow Features
An arbitrary graph of components, An arbitrary graph of components, each with zero or more inputs and zero each with zero or more inputs and zero or more outputsor more outputs
Paths connect outputs to inputsPaths connect outputs to inputs
Inputs or outputs might be left Inputs or outputs might be left “floating”“floating”
Special TaskSpecial Task““It’s just a task!”It’s just a task!”
Data Flow: ComponentsData Flow: Components
ComponentsComponentsData Source AdaptersData Source Adapters
Data Destination AdaptersData Destination Adapters
TransformsTransforms
PathsPathsJust a route between this component and Just a route between this component and the nextthe next
Includes metadata about columns moving Includes metadata about columns moving aroundaround
Data Flow: The GraphData Flow: The Graph
Empty graphEmpty graph
Source-to-DestinationSource-to-Destination
Single TransformSingle Transform
Multiple TransformsMultiple Transforms
Multiple Sources!Multiple Sources!
Multiple Destinations!Multiple Destinations!
Data Flow: AdaptersData Flow: Adapters
OLEDB Source and DestinationOLEDB Source and Destination
Flat File Source and DestinationFlat File Source and Destination
XML SourceXML Source
SQL Server DestinationSQL Server Destination
SQL CE DestinationSQL CE Destination
Recordset DestinationRecordset Destination
Raw Source and DestinationRaw Source and Destination
Data Flow: TransformsData Flow: Transforms
Some transforms work on each row as Some transforms work on each row as it goes byit goes by
Derived columnDerived column
Character MapCharacter Map
Data ConversionData Conversion
LookupLookup
Data Flow: TransformsData Flow: Transforms
Some transforms see whole data set Some transforms see whole data set before emitting even one rowbefore emitting even one row
SortSort
AggregateAggregate
Some will decide to emit rows as they Some will decide to emit rows as they need to, not always for each input rowneed to, not always for each input row
Pivot, UnPivotPivot, UnPivot
MergeJoinMergeJoin
Data Flow: TransformsData Flow: Transforms
Some transforms have multiple inputs Some transforms have multiple inputs and outputsand outputs
Union AllUnion All
MulticastMulticast
Conditional SplitConditional Split
Data Flow: PerformanceData Flow: Performance
As concurrent as possibleAs concurrent as possible
Don’t copy data aroundDon’t copy data around
Some transforms multithreadedSome transforms multithreaded
Data spoolingData spooling
BLOB SupportBLOB Support
Pipeline - Data Flow
XMLXML
DBDB
Sources
Flat FileFlat File
DestsOLE
DB
Da
ta S
ou
rce
ODBC C
USTO
M
Raw
Ad
ap
ters FLA
TFILE
RAWRAW
CustomCustom
DBDB
Flat FileFlat File
CustomCustom
FileFile
OLE
DB
Da
ta D
estin
ation
ODBC
CU
STO
M R
aw A
da
pters
FLATFILE
Derived Derived Column ErrorColumn Error
Conditional Conditional SplitSplit
AggregateAggregate
Fuzzy LookupFuzzy Lookup
Merge JoinMerge Join
RAWRAW
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
Putting It TogetherPutting It Together
Work Flow provides execution Work Flow provides execution environmentenvironment
Data Flow is just one of many tasksData Flow is just one of many tasks
Copy data, just like you used toCopy data, just like you used to
Manipulate data like you never thought Manipulate data like you never thought possible!possible!
AgendaAgenda
Demos!Demos!
BI WorkbenchBI Workbench
Workflow compared to DataflowWorkflow compared to Dataflow
Workflow FeaturesWorkflow Features
Dataflow FeaturesDataflow Features
Putting it TogetherPutting it Together
Where Did it Go?Where Did it Go?
Where Did it Go?Where Did it Go?
Some features have morphed and Some features have morphed and movedmoved
Where did they go?Where did they go?
WDIG: Data PumpWDIG: Data Pump
Pump is now the data flow taskPump is now the data flow task
Different features, slightly different Different features, slightly different paradigmparadigm
WDIG: LoopingWDIG: Looping
No more silly loop hacksNo more silly loop hacks
Functionality baked-in!Functionality baked-in!
Tasks can’t see outside themselvesTasks can’t see outside themselvesLess flexibilityLess flexibility
More featuresMore features
Solutions are now concrete!Solutions are now concrete!
WDIG: Dynamic PropertiesWDIG: Dynamic Properties
ConfigurationsConfigurations
Variable-based propertiesVariable-based properties
Advanced FeaturesAdvanced Features
Work FlowWork FlowConfigurationConfiguration
LoggingLogging
DebuggingDebugging
EventingEventing
Data FlowData FlowError outputsError outputs
SummarySummary
DTS is the tool of choice for data DTS is the tool of choice for data transformation and loading jobs, transformation and loading jobs, including the housekeeping!including the housekeeping!
SQL Server Community sitesSQL Server Community siteshttp://www.microsoft.com/sql/community/default.mspxhttp://www.microsoft.com/sql/community/default.mspxhttp://www.sqlserver.com.auhttp://www.sqlserver.com.auhttp://www.microsoft.com.au/Technetloungehttp://www.microsoft.com.au/Technetlounge
List of newsgroupsList of newsgroupshttp://www.microsoft.com/sql/community/newsgroups/default.mspxhttp://www.microsoft.com/sql/community/newsgroups/default.mspx
Locate Local User GroupsLocate Local User Groupshttphttp://www.microsoft.com/communities/usergroups/default.mspx ://www.microsoft.com/communities/usergroups/default.mspx
Attend a free chat or web castAttend a free chat or web casthttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp http://www.microsoft.com/usa/webcasts/default.asp
Professional Association for SQL ServerProfessional Association for SQL Serverwww.sqlpass.orgwww.sqlpass.org
SQLServer BI Resource SiteSQLServer BI Resource Sitewww.sqlserverbi.comwww.sqlserverbi.com
SQL Server Developers Network SQL Server Developers Network www.sqldev.net (Excellent Site!)www.sqldev.net (Excellent Site!)
SQLDTS Site SQLDTS Site www.sqldts.comwww.sqldts.com
SQLServer CentralSQLServer Centralwww.sqlservercentral.comwww.sqlservercentral.com
Attend a free chat or web castAttend a free chat or web casthttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp http://www.microsoft.com/usa/webcasts/default.asp
List of newsgroupsList of newsgroupshttp://communities2.microsoft.com/http://communities2.microsoft.com/communities/newsgroups/en-us/default.aspxcommunities/newsgroups/en-us/default.aspx
MS Community SitesMS Community Siteshttp://www.microsoft.com/communities/default.mspx http://www.microsoft.com/communities/default.mspx
Locate Local User GroupsLocate Local User Groupshttp://www.microsoft.com/communities/usergroups/default.mspx http://www.microsoft.com/communities/usergroups/default.mspx
Community sitesCommunity siteshttp://www.microsoft.com/communities/related/default.mspxhttp://www.microsoft.com/communities/related/default.mspx
DAT340
SQL Server 2005 (Codenamed “Yukon”):Applying DTS
DAT340
SQL Server 2005 (Codenamed “Yukon”):Applying DTS
Mike BlaszczakMike BlaszczakSoftware ArchitectSoftware ArchitectSQL Server Business IntelligenceSQL Server Business IntelligenceMicrosoft CorporationMicrosoft Corporation
Advanced Talk!Advanced Talk!
Some AssumptionsSome AssumptionsYou understand ETLYou understand ETL
You’ve seen Yukon DTS, You’ve seen Yukon DTS, or use Shiloh DTSor use Shiloh DTS
Ideal Audience MemberIdeal Audience MemberDW or SQL professionalDW or SQL professional
Uses DTS aggressivelyUses DTS aggressively
Experience with SQL Server 2000 DTSExperience with SQL Server 2000 DTS
AgendaAgenda
ToolsTools
Runtime Advanced FeaturesRuntime Advanced Features
Data Flow Task Advanced FeaturesData Flow Task Advanced Features
Integration of Runtime and Integration of Runtime and Data Flow TaskData Flow Task
MigrationMigration
PerformancePerformance
Extra ToolsExtra Tools
DTEXECDTEXEC
DTEXECUIDTEXECUI
Import/Export WizardImport/Export Wizard
Migration WizardMigration Wizard
AgendaAgenda
ToolsTools
Runtime Advanced FeaturesRuntime Advanced Features
Data Flow Task Advanced FeaturesData Flow Task Advanced Features
Integration of Runtime and Integration of Runtime and Data Flow TaskData Flow Task
MigrationMigration
PerformancePerformance
Understanding ConnectionsUnderstanding Connections
Layer between environment and Layer between environment and outside worldoutside world
Allows for indirectionAllows for indirection
Usually allows for easier configurationUsually allows for easier configuration
Provides enumeration servicesProvides enumeration services
Important for:Important for:Isolation, PortabilityIsolation, Portability
MetadataMetadata
… … a future feature?a future feature?
Runtime VariablesRuntime Variables
ScopeScope
NamespaceNamespaceSystem or UserSystem or User
NameName
Value and typeValue and type
Read-onlyRead-only
Package ConfigurationPackage Configuration
Set properties of objects in a package at Set properties of objects in a package at runtime: variables, connection strings, runtime: variables, connection strings, etc.etc.
Configure packages for different Configure packages for different environmentsenvironments
Configure variable values used in data Configure variable values used in data flow – change the value of a conditional flow – change the value of a conditional filter filter
Configure from XML, registry, Configure from XML, registry, environment variables, SQLenvironment variables, SQL
Variable Property MappingVariable Property Mapping
Variable set to PropertyVariable set to PropertyChange in variable causes change in Change in variable causes change in propertyproperty
Each time!Each time!
Runtime ConfigurationsRuntime Configurations
Take something from the systemTake something from the systemEnvironment, registry, explicit options Environment, registry, explicit options setting filesetting file
Apply it to some part of your packageApply it to some part of your package
ThenThen run the package with the new setting run the package with the new setting
Useful for multiple “similar” jobsUseful for multiple “similar” jobsPart of our reusability storyPart of our reusability story
Runtime LoggingRuntime Logging
Tasks (and the runtime engine) raise Tasks (and the runtime engine) raise log eventslog events
They can be saved and viewedThey can be saved and viewedConsoleConsole
SQL Server ProfilerSQL Server Profiler
SQL Server tableSQL Server table
Text fileText file
Get in the habit of using them!Get in the habit of using them!
Controlling ExecutionControlling Execution
PredicatesPredicatesBetween tasksBetween tasks
Determine/predict next executionDetermine/predict next execution
Expressions can be applied to predicatesExpressions can be applied to predicatesUse to examine variablesUse to examine variables
Use to configurations to change/set Use to configurations to change/set variablesvariables
Use Data Flow Task to set output variablesUse Data Flow Task to set output variables
LoopingLooping
No More Hacks!No More Hacks!Self-modifying code is badSelf-modifying code is bad
Object model is not self-visibleObject model is not self-visibleHacks are impossibleHacks are impossible
For Loop For Loop
Executes while some expression Executes while some expression is TRUEis TRUE
Typically regular numberingTypically regular numberingInit: @N = 1Init: @N = 1
Eval: @N <= 25Eval: @N <= 25
Increment: @N =@N + 1Increment: @N =@N + 1
Might be results, or “move next”Might be results, or “move next”
For Each LoopFor Each Loop
Use ForEach LoopUse ForEach LoopLoops over set of objectsLoops over set of objects
FilesFiles
XML nodesXML nodes
Database objectsDatabase objects
Do something with each itemDo something with each item
Property MappingProperty Mapping
Each iteration sets some variables Each iteration sets some variables local to the loop body containerlocal to the loop body container
Runtime can map variables to Runtime can map variables to properties on objectsproperties on objects
Parameterization of complex objectsParameterization of complex objects
Runtime DebuggingRuntime Debugging
Edit BreakpointsEdit Breakpoints
Use Watch WindowUse Watch Window
Trial and Error DebuggingTrial and Error DebuggingDisableDisable
AgendaAgenda
ToolsTools
Runtime Advanced FeaturesRuntime Advanced Features
Data Flow Task Advanced FeaturesData Flow Task Advanced Features
Integration of Runtime and Integration of Runtime and Data Flow TaskData Flow Task
MigrationMigration
PerformancePerformance
Multiple ConnectionsMultiple Connections
Data Flow Task supports:Data Flow Task supports:Multiple sourcesMultiple sources
Multiple destinationsMultiple destinations
Multiple disparate graphsMultiple disparate graphs
Complex GraphsComplex Graphs
Many SourcesMany Sources
Concurrency limited by EngineThreads Concurrency limited by EngineThreads property on Data Flow Taskproperty on Data Flow Task
Pipeline ExecutionPipeline Execution
Runtime blocks are very discreteRuntime blocks are very discreteIndividual start, work, endIndividual start, work, end
Pipeline blocks run and mergePipeline blocks run and mergeSometimes active, sometimes notSometimes active, sometimes not
Vague schedulingVague scheduling
Pipeline DebuggingPipeline Debugging
Data ViewerData ViewerGets a copy of buffer between two Gets a copy of buffer between two transformstransforms
Stops execution while displayedStops execution while displayed
Allows graphing of valuesAllows graphing of values
Data can be copied, selectedData can be copied, selected
Performance CountersPerformance Counters
Pipeline ExtensibilityPipeline Extensibility
Your Own Transforms or AdaptersYour Own Transforms or AdaptersNative CodeNative Code
Base Class, WizardBase Class, Wizard
Designer integration is the hard part!Designer integration is the hard part!
Managed CodeManaged CodeScript TransformScript Transform
AgendaAgenda
ToolsTools
Runtime Advanced FeaturesRuntime Advanced Features
Data Flow Task Advanced FeaturesData Flow Task Advanced Features
Integration of Runtime and Integration of Runtime and Data Flow TaskData Flow Task
MigrationMigration
PerformancePerformance
IntegrationIntegration
CommunicateCommunicate
From the Data Flow Task to the runtimeFrom the Data Flow Task to the runtimeRowset DestinationRowset Destination
Some binding to variables is allowedSome binding to variables is allowed
From the runtime to the Data Flow TaskFrom the runtime to the Data Flow TaskUse variables in the Derived Column Use variables in the Derived Column TransformTransform
Pipeline is Pipeline is notnot discrete! discrete!
AgendaAgenda
ToolsTools
Runtime Advanced FeaturesRuntime Advanced Features
Data Flow Task Advanced FeaturesData Flow Task Advanced Features
Integration of Runtime and Integration of Runtime and Data Flow TaskData Flow Task
MigrationMigration
PerformancePerformance
MigrationMigration
Migration WizardMigration WizardNotNot a solution, just a helper a solution, just a helper
Simple Pacakges mostly workSimple Pacakges mostly work
Test and tune afterwards, iterativelyTest and tune afterwards, iteratively
PortPortThere are paradigm shifts, anywayThere are paradigm shifts, anyway
Reuse of legacy isn’t manageableReuse of legacy isn’t manageable
Legacy is End of Lifecycle!Legacy is End of Lifecycle!
AgendaAgenda
ToolsTools
Runtime Advanced FeaturesRuntime Advanced Features
Data Flow Task Advanced FeaturesData Flow Task Advanced Features
Integration of Runtime and Integration of Runtime and Data Flow TaskData Flow Task
MigrationMigration
PerformancePerformance
PerformancePerformance
Mainly ConcurrencyMainly ConcurrencyAhmdahl’s LawAhmdahl’s LawDon’t over-do it; overhead goes upDon’t over-do it; overhead goes up
RuntimeRuntimeMaxThreadsMaxThreadsLoop ParallelizationLoop Parallelization
PipelinePipelineEngineThreads propertyEngineThreads propertyCollectors and DistributorsCollectors and Distributors
Parallel Loop ExecutionParallel Loop Execution
Runtime Loop ConcurrencyRuntime Loop Concurrency
for six iterations on two threadsfor six iterations on two threads
11 33
22 44
55Thread 1Thread 1
Thread 2Thread 2 66
Collectors/DistributorsCollectors/Distributors
Collectors and DistributorsCollectors and Distributors
Spread independent work across Spread independent work across multiple processorsmultiple processors
Draw what you want:Draw what you want:
SourceSource TransformTransform DestDest
Collectors/DistributorsCollectors/Distributors
Pipeline can fan-out workPipeline can fan-out work
After:After:
SourceSource
TransformTransform(Thread 1)(Thread 1)
DestDestDD CCTransformTransform(Thread 2)(Thread 2)
TransformTransform(Thread 3)(Thread 3)
Attend a free chat or web castAttend a free chat or web casthttp://www.microsoft.com/communities/chats/default.mspxhttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asphttp://www.microsoft.com/usa/webcasts/default.asp
List of newsgroupsList of newsgroupshttp://communities2.microsoft.com/http://communities2.microsoft.com/communities/newsgroups/en-us/default.aspxcommunities/newsgroups/en-us/default.aspx
MS Community SitesMS Community Siteshttp://www.microsoft.com/communities/default.mspxhttp://www.microsoft.com/communities/default.mspx
Locate Local User GroupsLocate Local User Groupshttp://www.microsoft.com/communities/usergroups/default.mspx http://www.microsoft.com/communities/usergroups/default.mspx
Community sitesCommunity siteshttp://www.microsoft.com/communities/related/default.mspxhttp://www.microsoft.com/communities/related/default.mspx