ssis custom components
DESCRIPTION
SSIS Custom Components. Dave Ballantyne [email protected] @ davebally. Why ?. Provide new functionality not provided as standard. Why ?. Reusability Component is a DLL Single code base Can be used multiple times in a single project Can be shared across multiple projects - PowerPoint PPT PresentationTRANSCRIPT
Why ?• Provide new functionality not provided as
standard
Why ?• Reusability
o Component is a DLLo Single code baseo Can be used multiple times in a single projecto Can be shared across multiple projectso Easy to test Component version
• Performanceo Faster than scripting
• Well documentedo Though not a how-to guide
Types Of Component• Data Connections• Log Providers• For Each Loops• Control Flow Tasks• Data Flow Pipeline Component• Custom User Interface
Pipeline Component Types
• Sources• Transforms• Destinations
Design/Run Time• Design Time
o Work done in BIDSo attachments / detachmentso Validationo Column usage
• Run Timeo Metadata interrogationo DTEXECo Flow of data
Demo 1• Reuse and Performance
Performance Comparison
100000 500000 1000000 5000000 10000000 20000000 319996800%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
ScriptCustom
100,000
500,000
1,000,000
5,000,000
10,000,000
20,000,000
31,999,680
Custom 261 900 1,667 7,867 16,375 32,852 51,426Script 684 2,069 3,949 19,214 38,690 76,755 123,243
Requirements• Visual Studio – BIDS is not enough• Or Visual Basic / C# Express• Client Tools SDK
Starting Out• Target Framework 3.5 (Advanced compile options)• Sign the assembley• Add References(Program
file(x86)/<SqlServer>/100/sdk/Assemblies)o Microsoft.SqlServer.DTSPipelineWrapo Microsoft.SqlServer.DTSRuntimeWrapo Microsoft.SqlServer.ManagedDTSo Microsoft.SqlServer.PipeLine Host
Class Creation• Inherits PipelineComponent• Uses attribute DtsPipelineComponent
Post Build• Copy DLL to “C:\Program Files (x86)\Microsoft SQL
Server\100\DTS\PipelineComponents” • Register to Global Assembley Cache using
GACUTIL• Must Restart BIDS• For first use “Choose Items”,”SSIS Data Flow
Components” ,tick Component
MetaData• IDTSComponentMetaData100• PipelineComponent.ComponentMetaData• Describes the Component to the engine• Inputs, Outputs• Custom data held within IDTSCustomProperty100
o Most level s of object
MetaData• Inputs – IDTSInput100
o Exposed via InputCollection member in MetaDatao One instance for each attached inputo Contains virtual column collection
• Accessed with GetVirtualInput() member• View of the IDTSOutput100 of the Upstream component• IDTSVirtualInputColumn100
o Input Column Collection• Accessed with InputColumnCollection• Those that are used in the component• IDTSInputColumn100
o SetUsageType used to add the virtual column to the input column
MetaData• Outputs – IDTSOutput100
o Exposed via OutputCollection member in MetaDatao One class for each outputo output Column Collection
• Accessed with OutputColumnCollection• IDTSOutputColumn100
• Dispositions – Errorso Set IsErrorOut on IDTSOutput100
Icons• Size
• 16*16 For ToolBox• 32*32 For Design Surface• Order of “IconResources” is important
• Build action must be “Embedded Resource”
Errors and warnings• FireError
o At design or run time
Errors and warnings• FireWarning
Design Time Methods• Methods
o ProvideComponentProperties• Define initial metadata of component
o Validate• Tests the metadata is correct
o ReinitializeMetaData• Fix the metadata
Debug
Demo 2• Build a simple component
Run-Time Processing
Pre-Execute
PrimeOutput
ProcessInput
PostExecute
PreExecute• Setup the runtime objects• Interrogate the Metadata and buffer manager• Find the colindex(s) in buffers based on metadata
o BufferManager.FindColumnByLineageID(InputId,InputCol.LineageId)
PrimeOutput
Process Input• Loop on buffer.NextRow• If buffer.EndOfRowset is true set
outputBuffer.SetEndOfRowset()• MetaData functions are
not optimized for performance.
PipelineBuffer• Used for both input and output buffer• Get<DataType> and Set <DataType>
o SetString / GetStringo SetInt32 / GetInt32
• AddRowo Insert and move to new row
• SetEndOfRowseto After final row has been poplulated
Sync Or Async ?• Sync
o Add columns to existing data flowo SynchronousInputID of output = ID of input
• Asynco Create new data flow buffero SynchronousInputID =0
Demo 3• RunTime execution
User Interface
User Interface
User Interface• A Class that implements IDtsComponentUI• Registered to the component class with
UITypeName
• PublicKeyToken is found with GACUTIL
User Interface
User Interface• Demo 4
o User interface o UI Code Step Through
Conclusion• Like SSIS , large learning curve• Reusability • Potentially Faster ?• .Net skills are required